Qwen3 Technical Report

1 Introduction

• 0.6 billion to 235 billion parameters span Qwen3’s open-weight dense and MoE models, supporting different downstream application needs.The flagship Qwen3-235B-A22B has 235 billion total parameters but activates 22 billion per token, supporting high performance and efficient inference.Source: §1 · p. 2
• Qwen3 unifies thinking and non-thinking modes within one model, while adjustable thinking budgets control reasoning effort and consistently improve performance when increased.Users can switch modes without changing models and allocate reasoning computation according to task complexity.Source: §1 · pp. 2–3
• 36 trillion pre-training tokens cover up to 119 languages and dialects, with PDF extraction and domain-specific synthetic generation expanding the training data.Fine-tuned Qwen2.5-VL (Bai et al., 2025) extracts PDF text; Qwen2.5-Math (Yang et al., 2024c) and Qwen2.5-Coder (Hui et al., 2024) generate mathematics and code data.Source: §1 · p. 2
• 30 trillion tokens approximately establish general knowledge before knowledge-intensive training strengthens STEM and coding reasoning, followed by long-context training.The final pre-training stage increases maximum context length from 4,096 to 32,768 tokens.Source: §1 · p. 2
• Qwen3’s post-training develops reasoning through long-CoT cold-start fine-tuning and mathematics-and-coding reinforcement learning, then unifies both operating modes and applies general-domain reinforcement learning.The mode-unification stage fine-tunes on a combined dataset containing examples with and without reasoning paths.Source: §1 · p. 2
• Strong-to-weak distillation improves smaller models through off-policy and on-policy knowledge transfer, significantly outperforming reinforcement learning in performance and training efficiency.Advanced larger models provide the teacher knowledge used for distillation.Source: §1 · p. 2
• 85.7 on AIME’24 is achieved by Qwen3-235B-A22B; Qwen3 base models achieve state-of-the-art performance, while post-trained models compete with leading proprietary and large MoE models.Other Qwen3-235B-A22B scores include 81.5 on AIME’25, 70.7 on LiveCodeBench v5 (Jain et al., 2024), 2,056 on CodeForces, and 70.8 on BFCL v3. The models particularly excel in coding, mathematics, and agent-related tasks.Source: §1 · pp. 2–3

2 Architecture

• 6 dense models and 2 MoE models comprise Qwen3, with flagship Qwen3-235B-A22B containing 235B total parameters and 22B activated parameters.The dense models are Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen3-32B; the other MoE model is Qwen3-30B-A3B. All use Qwen’s tokenizer (Bai et al., 2023), implementing byte-level byte-pair encoding (Brown et al., 2020; Wang et al., 2020; Sennrich et al., 2016) with a vocabulary size of 151,669.Source: §2 · p. 3
• Qwen3 removes QKV-bias used in Qwen2 (Yang et al., 2024a) and introduces QK-Norm (Dehghani et al., 2023) for stable training.Like Qwen2.5 (Yang et al., 2024b), dense models retain Grouped Query Attention (Ainslie et al., 2023), SwiGLU (Dauphin et al., 2017), Rotary Positional Embeddings, and RMSNorm (Jiang et al., 2023) with pre-normalization.Source: §2 · p. 3
• 128 total experts, with 8 activated per token and no shared experts, characterize Qwen3’s MoE models.They share the dense models’ fundamental architecture and follow Qwen2.5-MoE (Yang et al., 2024b) in using fine-grained expert segmentation (Dai et al., 2024). Global-batch load balancing loss (Qiu et al., 2025) encourages expert specialization; the report attributes substantial downstream performance improvements to these architectural and training innovations.Source: §2 · p. 3

3.1 Pre-training Data

• 36 trillion tokens spanning 119 languages and dialects form Qwen3’s pre-training corpus, including trillions of tokens from document extraction and synthetic generation.PDF text recognition followed by refinement adds trillions of high-quality tokens; synthetic textbooks, question-answering, instructions, and code snippets cover dozens of domains. Language coverage expands from 29 to 119.Source: §3.1 · pp. 3–4
• Over 30 trillion tokens receive multilingual annotations supporting instance-level data-mixture optimization through small-proxy-model ablations.Labels cover educational value, fields, domains, and safety, enabling fine-grained filtering and data combination.Source: §3.1 · p. 4

3.2 Pre-training Stage

• Over 30 trillion tokens establish general knowledge in S1 before about 5T reasoning-focused tokens in S2, both using 4,096-token sequences.S2 increases STEM, coding, reasoning, and synthetic data proportions while accelerating learning-rate decay. Scaling-law experiments predict learning-rate and batch-size strategies for each dense or MoE model using architecture, data, and training stage.Source: §3.2 · p. 4
• 32,768-token sequences support final-stage long-context pre-training on hundreds of billions of tokens, with additional techniques enabling fourfold inference sequence capacity.The corpus contains 75% text of length 16,384–32,768 tokens and 25% of length 4,096–16,384. ABF (Xiong et al., 2023) raises RoPE’s base frequency from 10,000 to 1,000,000; YARN (Peng et al., 2023) and Dual Chunk Attention (DCA, An et al., 2024) provide the inference extension.Source: §3.2 · p. 4

3.3 Pre-training Evaluation

• 14 of 15 benchmarks favor Qwen3-235B-A22B-Base over DeepSeek-V3-Base, using about 1/3 the total parameters and 2/3 the activated parameters.It also surpasses Qwen2.5-72B-Base on every benchmark with fewer than 1/3 of its activated parameters, and outperforms Llama-4-Maverick Base on most tasks.Source: §3.3 · p. 5, Table 3 · p. 6
• 10 of 15 benchmarks favor Qwen3-32B-Base over Qwen2.5-72B-Base despite using less than half its parameters, particularly in coding, mathematics, and reasoning.Qwen3-32B-Base scores 65.54 on MMLU-Pro and 39.78 on SuperGPQA. It beats Llama-4-Scout-Base on all 15 benchmarks with one-third the total parameters but twice the activated parameters.Source: Table 4 · p. 6, §3.3 · p. 9
• Using only 1/5 activated non-embedding parameters, Qwen3-30B-A3B significantly outperforms Qwen2.5-14B-Base on all tasks while achieving comparable performance to Qwen3-14B-Base and Qwen2.5-32B-Base, with advantages in inference and training costs.Qwen3-14B-Base outperforms Qwen2.5-14B-Base and Gemma-3-12B-Base on all 15 benchmarks. More broadly, Qwen3 MoE base models achieve similar performance to Qwen3 dense base models with 1/5 activated parameters and comparable performance to Qwen2.5 dense base models with 1/10 activated parameters.Source: §3.3 · pp. 5, 9
• Over half the benchmarks favor Qwen3-8B/4B/1.7B-Base over larger Qwen2.5-14B/7B/3B-Base models, respectively, especially in STEM and coding.The edge-side series, including Qwen3-0.6B-Base, maintains strong performance across nearly all benchmarks against similarly sized baselines.Source: Table 6 · p. 7, Table 7 · p. 8, Table 8 · p. 8

4 Post-training

• Qwen3’s flagship pipeline develops thinking abilities in two stages, then integrates non-thinking capabilities in two more, while distillation reduces lightweight-model computational costs and development effort.Users can choose whether the model reasons and control reasoning depth through a thinking-token budget.Source: §4 · p. 10

4.5 Strong-to-Weak Distillation

• 1/10 of four-stage training’s GPU hours suffices for distillation that improves lightweight students’ Pass@1 and Pass@64 while preserving reasoning control.Off-policy distillation first teaches reasoning and mode switching from teacher outputs in both modes. On-policy distillation then aligns logits on student-generated responses with Qwen3-32B or Qwen3-235B-A22B by minimizing KL divergence.Source: §4 · p. 10, §4.5 · p. 12

4.1 Long-CoT Cold Start

• Long-CoT cold start uses verified math, code, logic, and STEM problems, filtering queries and responses to establish reasoning patterns rather than maximize immediate performance.Query filtering removes unverifiable problems and those solvable without CoT, while balancing domains; response filtering removes incorrect, repetitive, guessed, inconsistent, stylistically problematic, or potentially validation-overlapping answers. A small selected dataset and limited training steps preserve flexibility for subsequent RL.Source: §4.1 · p. 10

4.2 Reasoning RL

• Qwen3-235B-A22B’s AIME’24 score increases from 70.1 to 85.1 over 170 Reasoning RL training steps, with consistent training reward and validation performance improvements without manual intervention on hyperparameters.GRPO (Shao et al., 2024) uses 3,995 query-verifier pairs not used during cold-start, learnable for the cold-start model, as challenging as possible, and covering broad sub-domains. Large batches, many rollouts per query, and off-policy training benefit training; model entropy is controlled to increase steadily or remain stable.Source: §4.2 · pp. 10–11

4.3 Thinking Mode Fusion

• Thinking Mode Fusion combines continual SFT and chat-template controls, enabling mode switching and emergent responses from incomplete reasoning under user-defined thinking budgets.SFT mixes rejection-sampled thinking responses with diverse non-thinking data, increasing translation tasks for low-resource languages; templates default to thinking, follow the last /think or /no think flag, and retain empty thinking blocks for non-thinking responses. At the budget threshold, an inserted stop-thinking instruction triggers a final answer from accumulated reasoning, without explicit budget-control training.Source: §4.3 · p. 11

4.4 General RL

• Over 20 tasks guide General RL toward instruction following, format adherence, preference alignment, agent ability, and specialized capabilities such as retrieval-augmented generation.Rewards combine rules, reference-based scoring by Qwen2.5-72B-Instruct, and a reference-free reward model trained from human preferences. Agent rollouts include complete multi-turn interactions with real execution feedback to improve long-horizon performance and stability.Source: §4.4 · p. 12

4.6 Post-training Evaluation

• 17/23 benchmarks favor Qwen3-235B-A22B (Thinking) over DeepSeek-R1, while 18/23 favor Qwen3-235B-A22B (Non-thinking) over GPT-4o-2024-11-20.The flagship leads open-source comparisons in both modes and is competitive with OpenAI-o1, Grok-3-Beta (Think), and Gemini2.5-Pro in thinking mode. Compared with DeepSeek-R1, it uses only 60% of the activated parameters and 35% of the total parameters.Source: Table 11 · p. 14, Table 12 · p. 14, §4.6 · p. 15
• Qwen3-32B surpasses QwQ-32B on most benchmarks and Qwen2.5-72B-Instruct in non-thinking mode; lightweight Qwen3 models outperform similarly sized or larger open-source models.Qwen3-32B performs comparably to OpenAI-o3-mini. Its non-thinking performance matches Qwen2.5-72B-Instruct on general tasks, with advantages in alignment, multilingual, and reasoning-related tasks.Source: §4.6 · p. 15, Table 13 · p. 16, Table 14 · p. 16
• Qwen3-8B, 4B, 1.7B, and 0.6B outperform larger baselines, including earlier Qwen2.5 models, in thinking and non-thinking modes.The authors attribute these lightweight models’ performance and reduced development costs to Strong-to-Weak Distillation.Source: Table 17 · p. 18, Table 18 · p. 18, Table 19 · p. 19
• Qwen3-30B-A3B and Qwen3-14B in non-thinking mode surpass baselines on most benchmarks and exceed Qwen2.5-32B-Instruct with fewer activated and total parameters.The reduced parameter requirements support more efficient and cost-effective performance.Source: §4.6 · p. 20

4.7 Discussion

• Increasing Qwen3-235B-A22B’s thinking budget produces smooth performance improvements across four benchmarks spanning mathematics, coding, and STEM.Further improvement beyond 32K output length is an expectation left for future exploration, not a demonstrated result.Source: Figure 2 · p. 20, §4.7 · p. 20
• Approximately 1/10 of direct reinforcement learning’s GPU hours enables on-policy distillation to achieve significantly better performance on math and code queries.Both methods start from the same off-policy-distilled 8B checkpoint. Teacher-logit distillation improves AIME’24 and AIME’25 pass@64 over that checkpoint, whereas reinforcement learning does not.Source: §4.7 · pp. 20–21, Table 21 · p. 21
• 98.9 is the ThinkFollow score after Stage 4 general RL, up from 88.7 after Stage 3 fusion, indicating more reliable mode switching.Stage 3 adds non-thinking mode and improves thinking-mode CounterFactQA by 10.9 points and LengthCtrl by 8.0 points, though switching occasionally fails. Stage 4 strengthens general, instruction-following, and agent capabilities in both modes.Source: Table 22 · p. 21, §4.7 · pp. 21–22
• Thinking Mode Fusion and General RL provide no significant Knowledge, STEM, Math, or Coding improvements and reduce thinking-mode performance on AIME’24 and LiveCodeBench.The authors conjecture that broader general-task training compromises specialized capabilities and explicitly accept this trade-off for greater versatility.Source: §4.7 · p. 22

• Qwen3 demonstrates strong performance on standard benchmarks for both pre-trained and post-trained models across code generation, mathematics, reasoning, and agent tasks.Source: §5 · p. 22
• Thinking and non-thinking modes let users dynamically manage the number of tokens used for complex thinking tasks.Source: §5 · p. 22
• 36 trillion pretraining tokens enable Qwen3 to understand and generate text in 119 languages and dialects.Source: §5 · p. 22
• Future work will scale pretraining with higher-quality, more diverse data and improve architecture and training methods for effective compression and extremely long contexts.Source: §5 · p. 22
• Planned reinforcement-learning compute increases emphasize agents learning from environmental feedback to tackle complex tasks requiring inference-time scaling.Source: §5 · p. 22

A.1.1 Long-Context Ability

• Qwen3 outperforms similarly sized Qwen2.5 models on long-context processing tasks in non-thinking mode.Table 23 reports the RULER benchmark evaluation.Source: Table 23 · p. 23, §A.1.1 · p. 23
• Long-context processing is evaluated using the RULER benchmark.Source: §A.1.1 · p. 23
• 4 is the scaling factor used with YARN (Peng et al., 2023) to enable length extrapolation.Source: §A.1.1 · p. 23
• 8192 tokens is the thinking-mode budget, chosen to mitigate overly verbose reasoning on extremely long inputs.Source: §A.1.1 · p. 23
• Thinking mode slightly degrades long-context performance.The authors identify improving thinking-mode long-context capability as future work.Source: §A.1.1 · p. 23
• The authors hypothesize that thinking content offers little benefit for retrieval tasks that do not require reasoning and may interfere with retrieval.Source: §A.1.1 · p. 23

A.1.2 Multilingual Ability

• Qwen3 models achieve competitive benchmark performance in Spanish, French, Portuguese, Italian, Arabic, Japanese, Korean, Indonesian, Russian, Vietnamese, German, and Thai.Source: §A.1.2 · p. 23
• 80 supported languages are evaluated on Belebele (Bandarkar et al., 2023), excluding 42 unoptimized languages.The evaluation tests natural language understanding, with the supported languages organized by language family.Source: §A.1.2 · p. 23, Table 36 · p. 30
• On Belebele (Bandarkar et al., 2023), Qwen3 achieves performance comparable to similarly sized Gemma models while significantly outperforming Qwen2.5.Source: §A.1.2 · p. 23, Table 37 · p. 30