On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability

Evaluation

• On fourteen pre-training benchmarks, Qwen3.8-Flash-Next leads the 397B-A17B predecessor on eight and trails on six by at most 2.6 points, with roughly 1/3 the activated parameters, 1/3 the training tokens, and 1/9 the training FLOPs.The model has 125B total parameters and 6B activated per token, plus 51B n-gram embedding parameters held off the accelerator. The predecessor is identified as Qwen Team, 2026.Source: §1 · p. 1
• Every candidate change is evaluated for loss and downstream benchmarks, training/prefill/decode cost, and effects on optimal hyperparameters and training stability.The paper treats loss, benchmarks, efficiency, and stability as one design problem while reporting each on its own terms.Source: §1 · p. 1

Architecture

• The architecture combines GDN with global attention, replaces full-attention layers with micro-block QSA during continued pre-training, widens residuals to four gated branches, and adds host-prefetched n-gram embeddings.GDN compresses the prefix into a fixed-size state at linear cost; QSA uses a compressed lightweight indexer, while the n-gram tables add capacity with negligible additional per-token FLOPs and latency. Figure 1 depicts three GDN layers alternating with one QSA layer per block of four.Source: §1 · p. 1, Figure 1 · p. 2

Evaluation

• Loss and downstream accuracy can diverge: enlarging the n-gram vocabulary lowers loss while accuracy saturates, and predicting residual read/write weights gives marginal loss reduction but a clear benchmark gain.Restricting blocks to the two highest-gated residual branches is nearly free in pre-training loss but degrades with further training; removing positional encoding from full-attention layers is indistinguishable during pre-training but affects later generation quality.Source: §1 · p. 2

Efficiency

• At a 1M context length, QSA is 7.6× faster than dense attention in prefill and 4.9× faster in decode at the kernel level.QSA compresses the key sequence by factor r, reducing indexer cost from O(n^2) to O(n^2/r); decode efficiency also relies on GDN’s fixed-size state, dropping GR’s Hres branch-mixing operator, and FP8 residual-state storage. FlashQLA achieves 2–3× forward and roughly 2× backward speedup over the Triton baseline on GPUs.Source: §1 · p. 2

Optimization

• The refitted scaling law predicts and verifies larger optimal batch size and learning rate, while batch-size warmup costs 18.8% more optimizer steps and performs no better than starting at the target.Muon applies to two-dimensional linear-map weights; embeddings, output head, MoE router, and GR low-rank projections remain on AdamW. Fused parameters are split before orthogonalization to avoid mixing singular directions across unrelated sub-blocks.Source: §1 · pp. 2–3

Training Stability

• At four times the optimal learning rate, the previous structure spikes frequently while the new recipe remains stable, and full-scale training proceeds without loss spikes or anomalous gradient-norm fluctuations.A single-variable pair isolating the GR gate identifies it as a key contributor to the stability margin; the full-scale run did not rely on explicit clipping methods such as qk-clip or SwiGLU-clip.Source: §1 · p. 3

2.1.1 GDN Hybrid Architecture

• One full-attention layer every four layers interleaves with three GDN layers, combining fixed-size content-dependent recurrent memory with direct token-level retrieval.The passage identifies periodic full attention as particularly important for long-context performance.Source: §2.1.1 · p. 4
• The GDN hybrid improves over full attention on eight of nine selected benchmarks and over the SWA hybrid on seven, achieving the highest overall average.It performs best on seven benchmarks, while SWA is marginally higher on MMLU and stronger on EvalPlus.Source: Table 1 · p. 5, §2.1.1 · p. 5

2.1.2 Qwen Sparse Attention

• QSA compresses context into micro-block representations for lightweight importance scoring and sparse attention, using four query heads, a 2048-token budget, and compression ratio 4.It replaces all full-attention layers in the backbone and MTP module; brief joint training after dense initialization lets the backbone recover to full-attention performance.Source: §2.1.2 · pp. 6, 7, 9
• QSA matches or outperforms full attention on seven of eight benchmarks and raises the average score from 75.9 to 76.8.The reported differences are small and show no systematic degradation in any task category.Source: Table 2 · p. 8, §2.1.2 · p. 8
• Beyond 512K, QSA improves RULER from 90.08 to 93.00; at 512K and 1M, it raises MRCR from 30.66 to 40.53 and 20.71 to 26.44, respectively.QSA is comparable to full attention at shorter lengths and performs better as context grows.Source: Table 3 · p. 8, §2.1.2 · p. 8
• QSA matches the full-attention baseline at 0.25 relative indexer latency, while IndexShare remains below baseline at 0.5.The ablation attributes the advantage to intra-layer compression in hybrid architectures, where limited inter-layer similarity can constrain cross-layer index sharing.Source: §2.1.2 · p. 8, Figure 5 · p. 9
• At 1M context, QSA achieves 7.6× prefill and 4.9× decode attention-module speedups over dense attention.QSA provides speedups from 64K context, with gains increasing as sequence length grows; decode evaluation includes three additional MTP steps.Source: Figure 6 · p. 9, §2.1.2 · p. 9

2.2 Residual

• Widening alone lowers training loss by roughly 0.01 in a 25B-A3B MoE trained on 400B tokens; it adds nr parameters per block, has negligible compute cost, and incurs extra memory traffic.Source: §2.2 · p. 10
• GR uses four independently normalized branches, an elementwise sigmoid read gate predicted from all branches, and data-dependent scalar writes; it drops Hres, reducing memory traffic and a potential source of instability.The ablations favor per-channel read granularity, per-branch scalar writes, and bounded positive sigmoid gates over tanh; GatedNorm also improves training stability.Source: §2.2 · pp. 10–12
• GR’s dynamic read and write add 1.98 benchmark points over static widening, while static widening adds 1.58 over the pre-norm baseline; their loss gap is only 0.002.The dynamic-versus-static benchmark gain is larger despite the smaller loss difference, illustrating why both loss and downstream benchmarks are evaluated.Source: Table 5 · p. 11, §2.2 · p. 11
• Full AttnRes matches GR’s loss at 1.762 in the 28-layer comparison, while Block AttnRes incurs losses of 0.008 at S = 2 and 0.011 at S = 4 relative to full AttnRes.GatedNorm lowers loss across the compared settings, including by 0.004–0.005 on AttnRes.Source: Table 6 · p. 12, §2.2 · p. 12
• GR allocates extra width to selected paths: one branch carries long-range information while the other three stay local, with attention layers receiving many of the long-range paths.Across five checkpoints, the long-range branch has a typical skip of 10.9 layers, versus 3.4–3.9 for the other branches; GR shifts path shares toward adjacent and long-range paths at the expense of mid-range paths.Source: §2.2 · pp. 12–14

2.3 N-gram Embedding

• N-gram embeddings retrieve vectors from tables using short n-grams ending at each token and add them to token representations, scaling capacity with negligible per-token FLOPs and enabling host-memory offloading.The experiments use multi-head hashing for lookup and contextual gating for injection into the residual stream.Source: §2.3 · p. 14

2.3.1 Placement of the N-gram Embedding Layers

• A single n-gram embedding layer at Layer 2 is sufficient: distributing a fixed parameter budget across layers gives no consistent benefit, and Layer 2 permits prefetching to overlap first-layer computation.No placement depth consistently dominates, and placement results are similar under full attention and GDN.Source: §2.3.1 · p. 14, Table 7 · p. 15

2.3.2 N-gram Vocabulary Size

• With a fixed total parameter budget, n-gram vocabulary scaling has a non-monotonic loss curve, lowest at 10× (25%), while downstream benchmarks show no clear improvement over the MoE-only baseline.The out-of-domain uncheatable PPL changes little across budgets, motivating a separate study that holds the MoE parameter budget fixed.Source: Table 8 · p. 15, §2.3.2 · p. 15
• With the MoE parameter budget fixed, scaling n-gram vocabulary from 20V to 200V monotonically decreases loss, but downstream gains can saturate or fluctuate; Chinese benchmarks improve consistently.The explored token-normalization, non-uniform allocation, and frequency-based partitioning strategies yield no consistent performance gains in this training recipe.Source: Table 9 · p. 15, §2.3.2 · p. 15

3.1 Optimizer

• Muon is used for matrix weights that act as linear maps, while AdamW handles embeddings, the output head, router, and GR low-rank projections; splitting fused projections before orthogonalization improves loss and benchmarks for qkv and GDN input projections.The optimizer uses eight Newton–Schulz steps, and splitting fc1 into gate and up halves leaves loss essentially unchanged while slightly improving benchmarks.Source: §3.1 · p. 16
• Canzona balances whole-parameter Newton–Schulz work across data-parallel ranks and reconstructs matrices across tensor-parallel ranks, while CUDA graphs reduce kernel-launch overhead after projection splitting.The pipeline preserves ZeRO-1 bucket geometry and the overlap of Megatron’s Reduce-Scatter with backward computation.Source: §3.1 · p. 16

3.2 Hyperparameter Scaling

• The new scaling law improves loss by 7.2 × 10−3 over the previous batch-size recipe at B = 25.2M; increasing to B = 37.7M incurs a 4.3 × 10−4 penalty.The batch-size prediction was tested on a 20-layer 10.8B-A0.89B MoE over 4T tokens; loss rises sharply below the predicted batch size and plateaus above it.Source: Figure 8 · p. 17, §3.2 · p. 17
• Batch-size warmup does not improve performance: both ramps are slightly worse than constant batch and require 18.8% more optimizer steps for the same token budget.Neither ramp improves stability; the constant-batch baseline ultimately surpasses the warmup runs as learning rate decays and the model approaches convergence.Source: Figure 8 · p. 17, §3.2 · p. 18
• The predicted learning rate sits in a flat near-optimal basin: the previous recipe ends 7.8 × 10−3 higher, while four nearby settings finish within 7 × 10−4 and the prediction has the highest average benchmark accuracy.The predicted setting is best or tied-best on most evaluated tasks; slight increases in learning rate or batch size have only marginal, statistically insignificant benchmark effects.Source: Figure 9 · p. 18, §3.2 · pp. 18–19, Table 10 · p. 19

3.3 Stability Stress Test

• At 4× the optimal learning rate, AdamW spikes 183 times per 10k steps and crosses the clipping threshold 213 times, while both Muon runs avoid clipping and Muon with GR has zero loss spikes.At 2×, AdamW spikes 4.3 times per 10k steps, versus 0.2 for either Muon configuration.Source: Figure 10 · p. 19, §3.3 · p. 20
• At 276B tokens, GR lowers loss by 0.026 and the full Flash-Next recipe by a further 0.032, for a 0.058 gain over the Muon baseline.The full recipe also has lower gradient norms and steadier gradients than Muon alone; adding GR reduces residual maximum activations throughout the network.Source: Figure 13 · p. 21, §3.3 · p. 22

4 Evaluation

• The evaluation covers 14 benchmarks across general tasks, math and STEM, coding, and multilingual understanding, using specified few-shot and chain-of-thought protocols.Coding measures include EvalPlus, MultiPL-E, and SWEBench-Pretrain.Source: §4 · p. 22
• 8 of 14 benchmarks: Qwen3.8-Flash-Next-Base outperforms Qwen3.7-Plus-Base and beats Qwen3.8-27B-Base on all 14, using 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.It remains competitive with Qwen3.7-Plus-Base on the other benchmarks.Source: §4 · p. 22
• Architectural improvements and fewer activated parameters give Qwen3.8-Flash-Next-Base significantly lower inference cost and a substantially better performance-efficiency trade-off in training and inference.Source: §4 · p. 22

• Qwen3.8-Flash-Next retains the previous-generation 397B-A17B flagship’s quality while activating a third of the parameters, using a third of the training tokens, and consuming roughly a ninth of the FLOPs.Source: §5 · p. 22
• Gated Residual (GR) rescaling markedly improves training stability, shifting optimal learning rate and batch size upward and improving throughput and convergence.The conclusion argues that architecture, efficiency, and optimization form one coupled system.Source: §5 · p. 22
• Phase-by-phase cost accounting guided sparse-attention design toward the indexer and concentrated GR’s expressiveness on the read.The conclusion warns that removing an axis from the design loop could permit sparse writes that degrade after post-training, positional encoding that appears dispensable during pre-training, or batch-size warmup that adds optimizer steps without benefit.Source: §5 · p. 22
• SWEBench-Pretrain scores are based on sequence similarity between golden patches and patches generated by base models from problem descriptions and relevant code files.Source: §5 · p. 22
• The validation protocol tests claims at tractable scale, using stress tests at multiples of the optimal learning rate and verifying scaling-law predictions in their most sensitive regimes.Joint evaluation caught cases where pre-training metrics agreed but late-stage evaluation diverged; the conclusion identifies evaluation throughput as the tightest bottleneck and calls for a cheaper mid-scale probe predictive of post-training ordering.Source: §5 · p. 23