The architecture paving the way for Qwen4 arrives in an open-weight preview

Qwen3.8-Flash-Next previews the Qwen4 architecture with open weights, 6 billion active parameters, and a native 262K-token context window.

A model can contain hundreds of billions of parameters without using all of them for every token. Qwen3.8-Flash-Next pushes this principle further by combining a mixture-of-experts architecture, a large memory system based on word sequences, and sparse attention designed for very long contexts. The result is both a usable multimodal model and an early technical preview of the architecture being explored for Qwen4.

Qwen presents Qwen3.8-Flash-Next as an experimental open-weight release. It is not Qwen4, and the architecture of the future model may still change. Its purpose is to expose several components under development before the next generation reaches production.

The “Next” designation also matters. Qwen3.8-Flash-Next is the downloadable research model, while Qwen3.8-Flash refers to the production version intended for QwenCloud. The two share the same architectural foundation, but the hosted model may receive additional training, optimization, safety work, or infrastructure adjustments.

The model accepts text, images, and video as input, then produces text. It is therefore designed not only for conversational use, but also for programming agents, document analysis, visual reasoning, office tasks, and workflows combining several types of content.

Its main network contains 125 billion parameters, but only about 6 billion are activated for each token. This difference comes from its mixture-of-experts structure: instead of passing every token through the entire model, a routing mechanism selects a limited number of specialized components.

Qwen3.8-Flash-Next includes 512 routed experts. Ten are selected for each token, along with one shared expert that remains active. This structure increases total capacity without making every generation use all available parameters.

The headline figure of 125 billion parameters does not tell the whole story. The architecture also includes 51 billion parameters assigned to N-gram embeddings and approximately 4 billion parameters for multi-token prediction. Counting those components brings the overall total above 180 billion parameters, even though the active computation per token remains much smaller.

The N-gram system is one of the model’s most unusual additions. Traditional language models assign a representation to individual tokens. Qwen also creates dedicated representations for frequently encountered sequences of two or three tokens. These sequences can encode recurring expressions, fragments of code, names, or other patterns without requiring the main network to reconstruct them from scratch every time.

The model uses around 20 million slots for bigrams and another 20 million for trigrams. These tables are introduced near the beginning of the network, at its second layer. Because they are extremely large, Qwen stores them in host memory and retrieves the required entries before the accelerator needs them.

This design shifts part of the model’s capacity away from the accelerator. It adds substantial storage requirements but relatively little computation for each token, since only the embeddings corresponding to the current sequences need to be retrieved.

That distinction is important for local deployment. The reference weights published on Hugging Face occupy roughly 360 GB across 131 main shards. Open weights therefore do not mean the model will run comfortably on an ordinary consumer computer. Deploying it requires significant storage, memory, and serving infrastructure, even if its active computation is much lower than its total parameter count suggests.

Long-context processing is handled through a hybrid attention design. Three layers using Gated Delta Networks are followed by one layer using Qwen Sparse Attention. This pattern is repeated throughout the model.

Gated Delta Networks maintain a compressed recurrent state instead of repeatedly comparing each new token with every previous token. They are intended to keep memory and computation from rising too quickly as a prompt becomes longer.

Qwen Sparse Attention preserves more conventional access to the context but divides it into smaller blocks and selects only the most relevant ones. The model can therefore revisit distant information without applying dense attention to every token pair.

According to Qwen’s technical report, the sparse-attention module reaches up to 7.6 times the prefill speed and 4.9 times the decoding speed of the dense-attention kernel used for comparison at a one-million-token context length. These figures apply to the attention component under the reported test conditions, not necessarily to the complete model or an end-user application.

The architecture also introduces a gated residual system. Instead of relying on a single residual stream to carry information from one layer to the next, it maintains four branches whose contributions can be adjusted dynamically. The goal is to increase representational capacity and improve the circulation of information through the model without proportionally expanding the active network.

Training combines the Muon optimizer with AdamW. Muon is applied to suitable matrix parameters, while AdamW handles embeddings, normalization components, and other parameter types. Qwen says this arrangement made it possible to train without the batch-size warmup normally used to stabilize very large runs.

The reported comparison includes an important caveat. A conventional warmup configuration required 18.8% more optimizer steps in Qwen’s experiment. That result supports the chosen training method under the tested conditions, but it does not establish that Muon will produce the same savings for every architecture or training workload.

Qwen says the model required one-ninth of the training computation used for the Qwen3.7-Plus base model while outperforming it across the company’s evaluations. That ratio comes from combining roughly one-third as many active parameters with roughly one-third as many training tokens.

The comparison concerns estimated training operations.