Source-linked AI summary
On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability
Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, Jianwei Zhang, Jing Xu, Junjie Wang, Langshi Chen, Liangyu Wang, Linlang Jiang, Man Yuan, Minmin Sun, Peng Jin, Siqi Zhang, Siyu Wang, Xingzhang Ren, Yakai Wang, Yi Zhang, Yiming Dong, Yizhong Cao, Yubo Ma, Yunfei Mao, Bo Zheng, Dayiheng Liu
TL;DR
The paper asks how to preserve flagship model quality while reducing compute and serving costs without sacrificing training stability. It evaluates a sparse architecture, hybrid token mixing, gated residuals, off-accelerator n-gram capacity, and Muon across loss, benchmarks, efficiency, hyperparameters, and stress tests. The resulting recipe matches the previous generation’s quality while activating one-third as many parameters, using one-third as many training tokens, consuming roughly one-ninth the training FLOPs, and improving stability.
Problem
The paper addresses how to retain the previous generation’s flagship quality at a fraction of its compute budget while accounting for efficiency and training stability.
Method
The paper evaluates a sparse mixture-of-experts architecture with GDN–attention hybrid token mixing, gated residuals, host-prefetched n-gram embeddings, and Muon across loss, benchmarks, cost, hyperparameters, and stability.
Results
The model retains the 397B-A17B flagship’s quality while activating a third of the parameters, training on a third of the tokens, and consuming roughly a ninth of the FLOPs.
Takeaways & Limitations
Jointly optimizing loss, benchmarks, efficiency, and stability yields a recipe that is more efficient, capable, and stable within the reported evaluation.
Takeaways & Limitations
Alternative strategies for improving n-gram parameter efficiency produced no consistent performance gains in the reported training recipe.
Abstract
from arXiv · showhide
We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.
1 Introduction
Qwen3.8-Flash-Next combines sparse architecture, hybrid token mixing, gated residuals, off-accelerator n-gram capacity, and Muon optimization to jointly target capability, efficiency, and stability. It matches the previous flagship’s quality on fourteen benchmarks while using substantially fewer activated parameters, tokens, and FLOPs.
- Model and evaluation: Qwen3.8-Flash-Next leads the 397B-A17B predecessor on eight of fourteen benchmarks and trails it on the other six by at most 2.6 points.The model has 125B total parameters, 6B activated per token, and 51B additional n-gram embedding parameters held off the accelerator.
- Evaluation framework: The design evaluates loss and downstream benchmarks alongside training, prefill, and decode cost, hyperparameter optima, and training stability.The paper treats disagreements among these axes as design signals rather than relying on loss alone.
- Architecture: Token mixing combines Gated DeltaNet with global attention, using one full-attention layer per four and Qwen Sparse Attention during continued pretraining.QSA scores context at micro-block granularity with a compressed lightweight indexer, while GDN compresses prefixes into fixed-size states at linear cost.
- Architecture: The residual stream widens to four branches and uses an elementwise read gate, while an n-gram embedding layer adds host-prefetched capacity outside the backbone.These components are designed to add residual-path capacity, control its use, and scale parameters off the accelerator.
- Efficiency: Training uses FlashQLA for a 2–3× forward and roughly 2× backward speedup over the Triton baseline on GPUs.Muon adds shape-dependent orthogonalization and CUDA-graph engineering costs.
- Optimization and stability: Muon and the new architecture shift the optimal batch size and learning rate upward, make batch-size warmup unnecessary, and improve stability under stress.Starting directly at the target batch size costs fewer optimizer steps than warmup, while the new recipe remains stable at four times the optimal learning rate.
2 Model Architecture
The architecture combines hybrid GDN/global token mixing, QSA sparse attention, and a widened gated residual stream to balance capability, efficiency, and stability. Ablations show that these choices preserve or improve benchmark performance while reducing attention or residual-stream costs.
- GDN Hybrid Architecture: 8 of 9 selected benchmarks improve over the Transformer with the GDN hybrid, which also exceeds the SWA hybrid on 7 of 9.The GDN hybrid achieves the best result on seven benchmarks and the highest overall average, although SWA is marginally higher on MMLU and EvalPlus.
- Qwen Sparse Attention: QSA matches or outperforms full attention on 7 of 8 benchmarks while raising the average score from 75.9 to 76.8.QSA is designed to reduce long-sequence indexing overhead through compressed micro-block representations and selective context scoring.
- Qwen Sparse Attention: QSA improves long-context retrieval beyond 512K, increasing RULER from 90.08 to 93.00 and MRCR at 512K from 30.66 to 40.53.At 1M tokens, MRCR also rises from 20.71 to 26.44, while QSA remains comparable to full attention at shorter lengths.
- Residual: GR improves both loss and benchmarks, with dynamic residual read/write adding 1.98 average-accuracy points beyond static widening.Static widening already adds 1.58 points over the pre-norm baseline, while the benchmark gain from dynamic read/write exceeds its 0.002 loss improvement.
- Residual: GR removes Hres, reducing memory traffic and eliminating a potential instability source while retaining comparable performance to dynamic mHC.GatedNorm additionally improves stability, and the gate contributes to the model’s training behavior.
- Residual: One of four GR branches consistently carries long-range paths, with a typical skip of 10.9 layers versus 3.4–3.9 for the other branches.Across five GR checkpoints, the identity of the long-range branch varies because branches are exchangeable at initialization.
3 Optimization
The architecture and Muon optimizer are treated as a coupled optimization problem that shifts hyperparameter optima upward, eliminates batch-size warmup benefits, and improves stability under stress.
- Optimizer: Muon orthogonalizes momentum updates with Newton–Schulz iterations and is applied to two-dimensional weights that act as linear maps.Input embeddings, the output head, and the MoE router remain on AdamW because Muon destabilizes the router early in training.
- Hyperparameter Scaling: Batch-size warmup performs no better than a constant batch and requires 18.8% more optimizer steps for the same token budget.Neither warmup variant improves performance, and both remain slightly worse than the constant-batch baseline.
- Hyperparameter Scaling: The updated scaling law predicts substantially larger batch sizes and learning rates, with slower learning-rate decay as model size increases.The predictions were separately validated in regimes designed to expose batch-size and learning-rate effects.
- Hyperparameter Scaling: A batch size of 25.2M reaches loss 1.5702 versus 1.5774 under the previous recipe, improving loss by 7.2 × 10−3.Increasing batch size to 37.7M incurs only a nonsignificant 4.3 × 10−4 penalty, while loss rises sharply below the predicted optimum.
- Stability Stress Test: At 4× the optimal learning rate, AdamW spikes on 183 per 10k steps, whereas both Muon runs remain highly stable.Adding Gated Residual reduces gradient-norm spikes and activation outliers, helping explain the observed stability margin.
- Ablations: At 276B tokens, Gated Residual lowers loss by 0.026 and the full Flash-Next recipe adds 0.032, totaling a 0.058 gain over the Muon baseline.The loss improvement translates into significant benchmark gains at roughly a ninth of the training cost.
4 Evaluation
Qwen3.8-Flash-Next-Base is evaluated across 14 benchmarks spanning general knowledge, reasoning, mathematics, coding, scientific knowledge, and multilingual understanding. It outperforms the smaller Qwen3.8-27B-Base across all tasks and exceeds Qwen3.7-Plus-Base on eight while using substantially fewer activated parameters, tokens, and FLOPs.
- Benchmark Coverage: The evaluation covers 14 benchmarks across general knowledge, reasoning, mathematics, scientific knowledge, coding, and multilingual understanding.The benchmark suite includes general, math and STEM, coding, and multilingual task groups.
- Results: Qwen3.8-Flash-Next-Base consistently outperforms Qwen3.8-27B-Base across all 14 benchmarks.The gains span general knowledge, reasoning, mathematics, coding, and multilingual capabilities.
- Results: Qwen3.8-Flash-Next-Base outperforms the much larger Qwen3.7-Plus-Base on 8 of 14 benchmarks while remaining competitive on the others.These results use about one-third of the activated parameters and training tokens, corresponding to roughly one-ninth of the training FLOPs.
- Efficiency: The model delivers a substantially better performance-efficiency trade-off in both training and inference.The smaller number of activated parameters contributes to significantly lower inference cost alongside the training-efficiency gains.
5 Conclusion
The paper presents Qwen3.8-Flash-Next as the result of jointly optimizing architecture, efficiency, and training stability. The resulting model retains the previous flagship’s quality while using one-third of its activated parameters and training tokens and roughly one-ninth of its FLOPs.
- Conclusion: The resulting model retains the previous generation’s 397B-A17B flagship quality while activating a third of the parameters, training on a third of the tokens, and consuming roughly a ninth of the FLOPs.This conclusion summarizes the model’s reported quality-efficiency trade-off.
- Design Principle: The design treats architecture, efficiency, and optimization as one coupled system rather than independent objectives.The paper reports that removing an axis would admit shortcuts that later degrade performance or add unnecessary cost.
- Evaluation: SWEBench-Pretrain asks base models to generate diff patches from problem descriptions and relevant code files, scoring sequence similarity to golden patches.The benchmark construction is described as inspired by Xia et al. (2024).
- Validation: Validation targets production-training failure modes through stress tests at elevated learning rates and scaling-law tests in sensitive regimes.The evaluation budget is kept tractable while testing stability and extrapolation where each effect is most pronounced.
6 Authors
The paper lists core contributors and additional contributors involved in the work.
- Core Contributors: The core contributors are Zihan Qiu, Zekun Wang, Xiao Li, Yanpeng Li, Yang Xu, Yixuan Wang, Huaqing Zhang, Rui Men, Bo ZhengB, and Dayiheng LiuB.
- Contributors: Additional contributors include Bochao Mao, Chengruidong Zhang, Fan Zhou, Hao Luo, Haofeng Huang, Haoran Lian, Haoyan Huang, Hongqing Chen, and others listed in the paper.