PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen3 Technical Report

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, Zihan Qiu

arXiv:2505.09388v1cs.CL

TL;DR

Qwen3 addresses the need for language models that combine broad capability, efficient inference, adaptable reasoning effort, and multilingual coverage. It introduces an open-weight family with unified thinking and non-thinking modes, scalable architectures, and multi-stage training, achieving strong results across coding, mathematics, reasoning, and agent benchmarks. The report also identifies trade-offs in specialized reasoning performance and future work on longer contexts and agent-based reinforcement learning.

  • Problem

    Foundation models need to balance performance, inference efficiency, reasoning flexibility, multilingual capability, and development cost across different model sizes and applications.

  • Method

    Qwen3 combines dense and MoE architectures with unified thinking and non-thinking modes, adjustable thinking budgets, large-scale multilingual pre-training, multi-stage post-training, and teacher-model distillation for smaller models.

  • Results

    Qwen3 achieves strong performance across code generation, mathematics, reasoning, and agent benchmarks, with post-trained models competitive against leading proprietary and large MoE models.

  • Takeaways & Limitations

    Qwen3 offers a publicly accessible Apache 2.0 model family whose unified modes and thinking budgets support adaptable reasoning and inference use cases.

  • Takeaways & Limitations

    Performance in thinking mode decreases on challenging AIME’24 and LiveCodeBench tasks after broader general-task training, reflecting a trade-off for overall versatility.

Abstract

from arXiv · show

In this work, we present Qwen3, the latest version of the Qwen model family. Qwen3 comprises a series of large language models (LLMs) designed to advance performance, efficiency, and multilingual capabilities. The Qwen3 series includes models of both dense and Mixture-of-Expert (MoE) architectures, with parameter scales ranging from 0.6 to 235 billion. A key innovation in Qwen3 is the integration of thinking mode (for complex, multi-step reasoning) and non-thinking mode (for rapid, context-driven responses) into a unified framework. This eliminates the need to switch between different models--such as chat-optimized models (e.g., GPT-4o) and dedicated reasoning models (e.g., QwQ-32B)--and enables dynamic mode switching based on user queries or chat templates. Meanwhile, Qwen3 introduces a thinking budget mechanism, allowing users to allocate computational resources adaptively during inference, thereby balancing latency and performance based on task complexity. Moreover, by leveraging the knowledge from the flagship models, we significantly reduce the computational resources required to build smaller-scale models, while ensuring their highly competitive performance. Empirical evaluations demonstrate that Qwen3 achieves state-of-the-art results across diverse benchmarks, including tasks in code generation, mathematical reasoning, agent tasks, etc., competitive against larger MoE models and proprietary models. Compared to its predecessor Qwen2.5, Qwen3 expands multilingual support from 29 to 119 languages and dialects, enhancing global accessibility through improved cross-lingual understanding and generation capabilities. To facilitate reproducibility and community-driven research and development, all Qwen3 models are publicly accessible under Apache 2.0.

1 Introduction

Qwen3 is an open-weight family of dense and MoE LLMs that combines flexible reasoning modes, controllable thinking budgets, broad multilingual training, and strong benchmark performance. Its training and post-training pipeline targets both general capability and efficient development of smaller models.

  • Model family: Qwen3 provides dense and MoE models from 0.6 billion to 235 billion parameters, including a 235-billion-parameter flagship with 22 billion activated per token.The flagship MoE design is intended to combine high performance with efficient inference.
  • Capabilities: Thinking and non-thinking modes are unified in one model, while thinking budgets let users control reasoning effort according to task complexity.This avoids switching between separate chat and reasoning models and supports adaptive computation during inference.
  • Training: Qwen3 is pre-trained on approximately 36 trillion tokens covering up to 119 languages and dialects, using extracted and synthetic data for linguistic, mathematical, and coding content.The pre-training pipeline uses Qwen2.5-VL, Qwen2.5-Math, and Qwen2.5-Coder to expand and diversify training data.
  • Training: Post-training combines long chain-of-thought finetuning, mathematics and coding reinforcement learning, unified reasoning and non-reasoning data, and general-domain reinforcement learning.Smaller models additionally use strong-to-weak distillation from advanced teacher models.
  • Evaluation: Qwen3 models achieve strong results across coding, mathematics, and agent benchmarks, with the flagship scoring 85.7 on AIME’24, 81.5 on AIME’25, 70.7 on LiveCodeBench v5, 2,056 on CodeForces, and 70.8 on BFCL v3.Post-trained models perform competitively against leading proprietary and large MoE models.

2 Architecture

Qwen3 includes six dense and two MoE models, sharing a Transformer-based design with architecture-specific attention and expert-routing choices. The MoE variants use fine-grained experts and activate only a subset of parameters per token.

  • Model variants: The Qwen3 lineup contains six dense models and two MoE models, spanning 0.6B to 235B parameters.The flagship Qwen3-235B-A22B has 235B total parameters and 22B activated parameters.
  • Dense architecture: Qwen3 dense models retain GQA, SwiGLU, RoPE, and RMSNorm with pre-normalization, while removing QKV bias and adding QK-Norm for stable training.The architecture is broadly similar to Qwen2.5.
  • MoE architecture: Qwen3 MoE models use 128 total experts with 8 activated per token, fine-grained expert segmentation, no shared experts, and global-batch load-balancing loss.These choices are designed to encourage expert specialization.
  • Tokenization: The Qwen3 tokenizer uses byte-level byte-pair encoding with a vocabulary size of 151,669.This tokenizer is shared across Qwen3 models.

3 Pre-training

Qwen3 uses expanded multilingual data and a three-stage pretraining process spanning general knowledge, reasoning, and long-context modeling. Across dense and MoE configurations, its base models achieve strong benchmark performance with fewer parameters or activated parameters than several baselines.

  • Pre-training data: Qwen3 is trained on 36 trillion tokens spanning 119 languages and dialects, including coding, STEM, reasoning, books, multilingual, and synthetic data.The dataset contains twice as many pretraining tokens and three times more languages than Qwen2.5.
  • Pre-training data: The data pipeline expands corpus quality through PDF text recognition, model-based refinement, and multilingual annotations supporting filtering and dataset combination.The annotation system covers over 30 trillion tokens across educational value, fields, domains, and safety.
  • Evaluation results: Qwen3 MoE base models match dense-model performance with only 1/5 activated parameters and outperform Qwen2.5 MoE models with fewer than 1/2 activated parameters and fewer total parameters.The reported gains combine the Qwen3 MoE architecture, scaled training tokens, and advanced training strategies.
  • Evaluation results: Qwen3 dense models match Qwen2.5 models at higher parameter scales, while exceeding them especially on STEM, coding, and reasoning benchmarks.Qwen3-1.7B/4B/8B/14B/32B-Base are reported as comparable to Qwen2.5-3B/7B/14B/32B/72B-Base, respectively.
  • Evaluation results: Qwen3-235B-A22B-Base outperforms DeepSeek-V3-Base on 14 of 15 benchmarks using about 1/3 the total parameters and 2/3 the activated parameters.It also outperforms several open-source baselines in most tasks with significantly fewer total or activated parameters.
  • Evaluation results: Qwen3-32B-Base outperforms Qwen2.5-32B-Base and Gemma-3-27B on most benchmarks, and exceeds Qwen2.5-72B-Base on 10 of 15 benchmarks despite having less than half its parameters.It achieves 65.54 on MMLU-Pro and 39.78 on SuperGPQA, and has reported advantages on coding, mathematics, and reasoning benchmarks.
  • Evaluation results: Qwen3-14B-Base exceeds similar-sized baselines on all 15 benchmarks, while Qwen3-30B-A3B matches larger models with only 1/5 activated non-embedding parameters.Qwen3-30B-A3B also outperforms Qwen2.5-14B-Base on all tasks, with reported inference and training-cost advantages.

4 Post-training

Qwen3’s post-training pipeline unifies thinking and non-thinking modes, adds controllable thinking budgets, and uses strong-to-weak distillation to improve efficiency. Evaluations report strong performance across reasoning, coding, agent, alignment, and multilingual tasks, while documenting trade-offs in some specialized settings.

  • Post-training objectives: Qwen3’s post-training pipeline targets thinking control and strong-to-weak distillation.Thinking control integrates non-thinking and thinking modes with user-specified reasoning depth; distillation streamlines post-training for smaller models.
  • Strong-to-weak distillation: Distillation improves lightweight-model performance while avoiding an exhaustive four-stage training process for every small model.Teacher-logit distillation provides fine-grained control over reasoning and yields higher immediate Pass@1 scores.
  • Evaluation results: The flagship Qwen3-235B-A22B achieves state-of-the-art overall performance among open-source models in both thinking and non-thinking modes.It surpasses strong open-source baselines and remains highly competitive with leading closed-source models.
  • Evaluation results: Qwen3-32B and lightweight models outperform or match relevant baselines while using fewer parameters or activated parameters.Qwen3-32B surpasses QwQ-32B in most benchmarks, while Qwen3-30B-A3B and Qwen3-14B exceed Qwen2.5-32B-Instruct with fewer parameters.
  • Thinking budget: Increasing the thinking budget produces smooth performance improvements across mathematics, coding, and STEM benchmarks.Performance is expected to improve further when output length extends beyond 32K tokens, which the authors leave for future work.
  • Ablation findings: Training-stage ablations reveal a versatility trade-off: thinking-mode performance decreases on AIME’24 and LiveCodeBench after broader general-task training.The authors conjecture that broader training compromises specialized complex-problem capabilities and accept this trade-off for overall versatility.

5 Conclusion

Qwen3 combines thinking and non-thinking modes with controllable token budgets, extensive pre-training, and broad multilingual coverage. Comprehensive evaluations report strong performance across code, mathematics, reasoning, and agent tasks, while future work targets scaling, architecture, long contexts, and reinforcement learning.

  • Conclusion: Qwen3 combines thinking and non-thinking modes, allowing users to manage tokens allocated to complex thinking tasks.The unified design supports dynamic control of reasoning effort.
  • Conclusion: Qwen3 was pre-trained on 36 trillion tokens and supports text understanding and generation in 119 languages and dialects.The report links its multilingual scope to the extensive pre-training dataset.
  • Conclusion: Evaluations show strong Qwen3 performance across code generation, mathematics, reasoning, and agent benchmarks for pre-trained and post-trained models.The conclusion summarizes results across standard benchmark categories.
  • Future work: Future work will scale pre-training, improve architecture and training for compression and long contexts, and expand agent-based reinforcement learning.The stated goal is to build agents capable of complex tasks requiring inference-time scaling.

6 Authors

The report credits a large group of core contributors and additional contributors for the Qwen3 work.

  • Core Contributors: The report lists core contributors including An Yang, Anfeng Li, Baosong Yang, and Beichen Zhang.The complete core-contributor list appears in the report.
  • Contributors: The authorship section distinguishes the core-contributor list from the broader contributor list.The two passages use separate headings for these groups.
  • Contributors: The report separately lists additional contributors including Bei Chen, Biao Sun, Bin Luo, and Bin Zhang.The complete contributor list appears in the report.

A.1.1 Long-Context Ability

Qwen3 is evaluated on long-context processing with RULER, using YARN for length extrapolation and a fixed thinking budget. Non-thinking mode outperforms similarly sized Qwen2.5 models, whereas thinking mode slightly degrades on retrieval-oriented tasks.

  • Evaluation setup: Long-context evaluation uses the RULER benchmark, YARN scaling factor 4, and an 8192-token thinking budget.The budget is used in thinking mode to mitigate overly verbose reasoning on extremely long inputs.
  • Results: In non-thinking mode, Qwen3 outperforms similarly sized Qwen2.5 models on long-context processing tasks.This is the reported comparison for non-thinking mode.
  • Results: In thinking mode, Qwen3’s long-context performance slightly degrades on retrieval tasks that do not rely on reasoning.The authors hypothesize that thinking content may interfere with retrieval and identify improvement as future work.

A.1.2 Multilingual Ability

Qwen3 demonstrates competitive multilingual performance across evaluated benchmarks and languages. Belebele evaluation further assesses its natural-language understanding across 80 supported languages, with 42 unoptimized languages excluded.

  • Qwen3 models achieve competitive performance across evaluated multilingual benchmarks and languages.Reported languages include Spanish, French, Portuguese, Italian, Arabic, Japanese, Korean, Indonesian, Russian, Vietnamese, German, and Thai.
  • The multilingual benchmark results cover Spanish, French, Portuguese, Italian, Arabic, Japanese, and Korean.
  • The evaluation also covers Indonesian, Russian, Vietnamese, German, and Thai.
  • Belebele evaluates Qwen3 on natural-language understanding across 80 supported languages, excluding 42 unoptimized languages.The supported languages are organized by language family in Table 36.
  • Table 37 compares Qwen3's Belebele performance with other baseline models, highlighting the highest scores and underlining the second-best.
Loading 2505.09388v1…