PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen2.5 Technical Report

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu

arXiv:2412.15115v2cs.CL

TL;DR

LLM development continues to target language understanding, generation, and reasoning, while model options also need to suit resource-limited scenarios. Qwen2.5 combines expanded pre-training and post-training, and its 72B-Instruct model surpasses Llama-3.1-405B-Instruct on seven named benchmarks.

  • Problem

    The report situates its work amid continued LLM advances in language understanding, generation, and reasoning, alongside models suited to resource-limited scenarios.

    Source: §1 · p. 2

  • Method

    Qwen2.5 combines expanded pre-training data with 1 million post-training examples across supervised finetuning, direct preference optimization, and group relative policy optimization.

    Source: §1 · p. 2

  • Results

    Qwen2.5-72B-Instruct surpasses Llama-3.1-405B-Instruct on MMLU-redux, MATH, MBPP, MultiPL-E, LiveCodeBench, Arena-Hard, and MTBench; Qwen2.5-Plus wins 9 of 13 benchmarks.

    Source: §5.2.1 · p. 12

  • Takeaways & Limitations

    Qwen2.5's 3B, 14B, and 32B models offer more cost-effective options for resource-limited scenarios.

    Source: §1 · p. 2

    Current reward-model evaluation benchmarks do not accurately predict the performance of the RL models trained under their guidance.

    Source: §5.2.3 · p. 16

Abstract

from arXiv · show

In this report, we introduce Qwen2.5, a comprehensive series of large language models (LLMs) designed to meet diverse needs. Compared to previous iterations, Qwen 2.5 has been significantly improved during both the pre-training and post-training stages. In terms of pre-training, we have scaled the high-quality pre-training datasets from the previous 7 trillion tokens to 18 trillion tokens. This provides a strong foundation for common sense, expert knowledge, and reasoning capabilities. In terms of post-training, we implement intricate supervised finetuning with over 1 million samples, as well as multistage reinforcement learning. Post-training techniques enhance human preference, and notably improve long text generation, structural data analysis, and instruction following. To handle diverse and varied use cases effectively, we present Qwen2.5 LLM series in rich sizes. Open-weight offerings include base and instruction-tuned models, with quantized versions available. In addition, for hosted solutions, the proprietary models currently include two mixture-of-experts (MoE) variants: Qwen2.5-Turbo and Qwen2.5-Plus, both available from Alibaba Cloud Model Studio. Qwen2.5 has demonstrated top-tier performance on a wide range of benchmarks evaluating language understanding, reasoning, mathematics, coding, human preference alignment, etc. Specifically, the open-weight flagship Qwen2.5-72B-Instruct outperforms a number of open and proprietary models and demonstrates competitive performance to the state-of-the-art open-weight model, Llama-3-405B-Instruct, which is around 5 times larger. Qwen2.5-Turbo and Qwen2.5-Plus offer superior cost-effectiveness while performing competitively against GPT-4o-mini and GPT-4o respectively. Additionally, as the foundation, Qwen2.5 models have been instrumental in training specialized models such as Qwen2.5-Math, Qwen2.5-Coder, QwQ, and multimodal models.

1 Introduction

Qwen2.5 expands its open-weight and hosted model offerings while improving training data and practical capabilities. Its releases span multiple model sizes and formats, with updates to pre-training, post-training, generation length, structured input and output, and tool use.

1 Introduction

  • Qwen2.5-72B-Instruct competes with Llama-3-405B-Instruct, around 5 times larger; Qwen2.5-Turbo and Qwen2.5-Plus perform competitively with GPT-4o-mini and GPT-4o, respectively.The two hosted models are proprietary Mixture-of-Experts variants.Source: §1 · p. 2
  • Open-weight releases cover seven sizes—0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B—in pretrained and instruction-tuned forms, with quantized versions also available.The 3B, 14B, and 32B models are positioned as more cost-effective for resource-limited scenarios.Source: §1 · p. 2
  • Qwen2.5-Turbo and Qwen2.5-Plus offer a balance among accuracy, latency, and cost.Source: §1 · p. 2
  • Pre-training data grew from 7 trillion to 18 trillion tokens, focused on knowledge, coding, and mathematics; post-training uses 1 million examples across SFT, DPO, and GRPO.Pre-training is staged to allow transitions among different data mixtures.Source: §1 · p. 2
  • Generation length increased from 2K to 8K tokens, with better support for structured input and output and tool use; Turbo supports context lengths up to 1 million tokens.Source: §1 · p. 2

2 Architecture & Tokenizer

Qwen2.5 includes dense open-weight models and API MoE models, built on a Qwen2-derived Transformer architecture and a shared expanded tokenizer.

2 Architecture & Tokenizer

  • The series comprises dense open-weight models from 0.5B to 72B and the API MoE models Qwen2.5-Turbo and Qwen2.5-Plus.The dense sizes are 0.5B, 1.5B, 3B, 7B, 14B, 32B, and 72B.Source: §2 · p. 2
  • Dense models retain Qwen2’s Transformer decoder, using GQA, SwiGLU, RoPE, QKV bias, and pre-normalized RMSNorm (Vaswani et al., 2017; Radford et al., 2018; Yang et al., 2024a; Ainslie et al., 2023; Dauphin et al., 2017; Su et al., 2024; Su, 2023; Jiang et al., 2023b).The components support efficient KV-cache use, nonlinear activation, positional encoding, and stable training.Source: §2 · pp. 2–3
  • MoE models replace dense FFN layers with multiple experts and top-K token routing, using fine-grained expert segmentation and shared-expert routing; the report says these innovations improve downstream performance (Yang et al., 2024a; Dai et al., 2024; Rajbhandari et al., 2022).This design extends the dense model architectures.Source: §2 · p. 3
  • The tokenizer uses BBPE with 151,643 regular tokens and expands control tokens from 3 to 22, including two tool-functionality tokens, to unify vocabulary across Qwen2.5 models (Bai et al., 2023; Brown et al., 2020; Wang et al., 2020; Sennrich et al., 2016).The report says the shared vocabulary enhances consistency and reduces potential compatibility issues.Source: §2 · p. 3

3 Pre-training

Qwen2.5 pre-training combines a larger, higher-quality dataset with scaling-law-guided hyperparameter selection and dedicated long-context training. The section describes data curation, training-parameter optimization, and methods for extending context capacity.

3 Pre-training

  • The pre-training process combines curated, strategically mixed data, hyperparameter optimization across model scales, and specialized long-context training.Source: §3 · p. 3

3.1 Pre-training Data

  • Qwen2-Instruct models support multilingual sample filtering and domain balancing, retaining more high-quality data while down-sampling overrepresented domains and up-sampling high-value ones.Source: §3.1 · pp. 3–4
  • Pre-training incorporates Qwen2.5-Math (Yang et al., 2024b) and Qwen2.5-Coder (Hui et al., 2024) data, alongside synthetic data filtered with reward models.Synthetic data is generated using Qwen2-72B-Instruct (Yang et al., 2024a) and Qwen2-Math-72B-Instruct, then filtered using a general reward model and Qwen2-Math-RM-72B.Source: §3.1 · p. 3
  • The pre-training dataset grew from 7 trillion tokens for Qwen2 to 18 trillion tokens for Qwen2.5.Source: §3.1 · p. 4

3.2 Scaling Law for Hyper-parameters

  • Scaling laws guide batch size B and learning rate µ choices across dense and MoE architectures, and predict loss and MoE performance relative to dense models.Experiments cover dense models from 44M to 14B parameters and MoE models from 44M to 1B activated parameters, trained on 0.8B to 600B tokens; tuning can achieve performance parity with Qwen2.5-72B and Qwen2.5-14B dense variants.Source: §3.2 · p. 4

3.3 Long-context Pre-training

  • Most variants extend context from 4,096 to 32,768 tokens in final pre-training, while Qwen2.5-Turbo progressively trains at lengths up to 262,144 tokens.Turbo uses four stages—32,768, 65,536, 131,072, and 262,144 tokens—with 40% of training sequences at each stage’s maximum length and 60% shorter sequences.Source: §3.3 · p. 4
  • YARN (Peng et al., 2023) and Dual Chunk Attention (DCA, An et al., 2024) enable up to 1 million tokens for Qwen2.5-Turbo and 131,072 for other models while reducing long-sequence perplexity.The methods increase sequence-length capacity four-fold and preserve strong performance on shorter sequences.Source: §3.3 · p. 4

4 Post-training

Qwen2.5 expands supervised fine-tuning with millions of examples and adds offline and online reinforcement learning. For Qwen2.5-Turbo, long-context fine-tuning combines short and long SFT examples while RL remains focused on short instructions.

4 Post-training

  • Qwen2.5’s post-training adds high-quality SFT examples and divides RL into offline and online stages with distinct learning targets.Offline RL focuses on reasoning, factuality, and instruction following; online RL uses reward-model judgments of truthfulness, helpfulness, conciseness, relevance, harmlessness, and debiasing.Source: §4 · pp. 4–5

4.1 Supervised Fine-tuning

  • SFT supports long-sequence generation, math reasoning, and coding through long-response data, Qwen2.5-Math chain-of-thought examples, and multilingual coding instruction data.Long-generation training targets outputs up to 8,192 tokens; coding data spans nearly 40 programming languages and is checked with sandboxing and automated tests.Source: §4.1 · p. 5
  • Instruction-following data is curated with execution-feedback rejection sampling, while diverse system prompts improve robustness and reduce performance variance.LLMs generate instructions, verification code, and unit tests; evaluations with different system prompts show good performance and reduced variance.Source: §4.1 · pp. 5–6
  • SFT also covers structured-data tasks, 70,000 logical-reasoning queries, and cross-lingual transfer using translated instructions and semantically aligned responses.Structured-data responses incorporate reasoning chains; logical-reasoning data is filtered for incorrect answers and flawed reasoning.Source: §4.1 · pp. 5–6

4.2 Offline Reinforcement Learning

  • Offline RL uses quality-checked SFT-model responses as positive or negative DPO examples for objective tasks, with human and automated review of roughly 150,000 training pairs.The pipeline targets domains such as mathematics, coding, instruction following, and logical reasoning, where reward models can struggle to evaluate answers.Source: §4.2 · p. 6

4.3 Online Reinforcement Learning

  • Online RL trains with GRPO, prioritizing queries whose reward-model response scores have higher variance and sampling eight responses per query.Reward-model preference pairs combine human and automated labels, drawing on public and proprietary queries and responses from Qwen checkpoints.Source: §4.3 · p. 7

4.4 Long Context Fine-tuning

  • Qwen2.5-Turbo’s two-stage SFT adds instructions up to 262,144 tokens after short-instruction training, improving long-context instruction following while preserving short-task performance.RL uses short instructions only, due to cost and the scarcity of suitable long-context reward models; the authors report that this still improves long-context preference alignment.Source: §4.4 · p. 7

5 Evaluation

The evaluation combines open benchmarks with skill-oriented in-house datasets and is designed to rely primarily on automatic assessment. Instruction-tuned models are also assessed for human preference, multilingual performance, and long-context capability.

5.1 Base Models

  • Qwen2.5-72B achieves results comparable to Llama3-405B with one-fifth the parameters, and Qwen2.5-Plus scores 64.0 on MMLU-Pro, 5.9 points above Qwen2.5-72B.Qwen2.5-72B significantly outperforms peers in its category and improves on Qwen2-72B across nearly all benchmarks, particularly in general tasks, mathematics, and coding.Source: §5.1 · p. 8, Table 2 · p. 8
  • Qwen2.5-14B scores 79.7 on MMLU and 78.2 on BBH, while Qwen2.5-32B scores 57.7 on MATH and 84.5 on MBPP.Qwen2.5-Turbo has significantly lower training and inference costs than Qwen2.5-14B while achieving comparable results, including a higher MMLU-Pro score than Qwen2.5-32B.Source: Table 3 · p. 9, §5.1 · p. 10
  • Qwen2.5-7B scores 74.2 on MMLU, 49.8 on MATH, and 57.9 on HumanEval, surpassing predecessors and counterparts despite having 6.5B non-embedding parameters.Qwen2.5-0.5B, 1.5B, and 3B maintain strong performance across nearly all benchmarks, and the 0.5B model beats Gemma2-2.6B on various math and coding tasks.Source: Table 4 · p. 9, §5.1 · p. 10, Table 5 · p. 10

5.2.1 Open Benchmark Evaluation

  • Qwen2.5-72B-Instruct surpasses the larger Llama-3.1-405B-Instruct on several benchmarks, while Qwen2.5-Plus beats it on 9 of 13 benchmarks.The reported benchmarks include MMLU-redux, MATH, MBPP, MultiPL-E, LiveCodeBench, Arena-Hard, and MTBench.Source: Table 6 · p. 11, §5.2.1 · p. 12
  • Qwen2.5-Turbo outperforms Qwen2.5-14B-Instruct on eight of ten benchmarks despite significantly lower training and inference costs, while Qwen2.5-14B-Instruct rivals GPT-4o-mini.Qwen2.5-32B-Instruct performs better across most tasks than similar-sized models, and Qwen2.5-7B-Instruct beats competitors on all tasks except IFEval, scoring 75.5 on MATH and 84.8 on HumanEval.Source: Table 7 · p. 11, §5.2.1 · pp. 12–13, Table 8 · p. 12

5.2.2 In-house Automatic Evaluation

  • Qwen2.5-72B matches or exceeds Llama-3.1-405B on all in-house metrics except instruction following, while Qwen2.5-Plus improves Chinese instruction following.Qwen2.5-72B also narrows the gap with GPT-4 and Claude3.5-sonnet; Qwen2.5-Plus further improves its advantages in other areas.Source: Table 11 · p. 13, §5.2.2 · p. 14, Table 12 · p. 14

5.2.3 Reward Model

  • Qwen2.5-RM-72B leads on PPE and Human-Preference-Chinese, but current reward-model benchmark scores do not reliably predict the performance of RL models trained with those rewards.Llama-3.1-Nemotron-70B-Reward leads on Reward Bench, Athene-RM-70B leads on RMB, and the authors warn that over-optimizing a single benchmark may degrade performance on others.Source: §5.2.3 · pp. 15–16, Table 15 · p. 16

5.2.4 Long Context Capabilities

  • Qwen2.5-Turbo achieves 100% accuracy on 1M-token passkey retrieval, while Qwen2.5-72B-Instruct performs strongest across context lengths on three long-context datasets.With sparse attention based on Minference, 1M-token sequences reduce attention computation 12.5 times and achieve 3.2–4.3 times faster time to first token across hardware configurations; the models use DCA + YARN for length extrapolation.Source: §5.2.4 · p. 16, Figure 2 · p. 17, Table 16 · p. 17

6 Conclusion

Qwen2.5 combines enhanced pre-training on 18 trillion tokens with supervised fine-tuning and multi-stage reinforcement learning, and the report outlines future work on data, multimodality, and reasoning.

6 Conclusion

  • Six times smaller, Qwen2.5-72B-Instruct matches the performance of Llama-3-405B-Instruct.The report also describes Qwen2.5 as a foundation for specialized models and domain-specific applications.Source: §6 · p. 18
  • Qwen2.5’s enhanced training boosts human preference alignment, long text generation, structural data analysis, and instruction following.The model series includes open-weight configurations from 0.5B to 72B parameters and proprietary cost-effective MoE variants Qwen2.5-Turbo and Qwen2.5-Plus.Source: §6 · p. 18
  • Future work will refine base and instruction-tuned models with broader, more diverse, higher-quality data, develop multimodal models, and scale inference compute to improve reasoning.The report aims to integrate text, visual, and auditory modalities in a unified framework for end-to-end information processing.Source: §6 · p. 18

Related papers

Loading 2412.15115v2…