PaperlayerView exact revision on arXiv

Source-linked AI summary

Qwen2 Technical Report

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, Zhihao Fan

arXiv:2407.10671v4cs.CLcs.AI

TL;DR

Qwen2 addresses the need for capable, broadly usable open-weight language models by releasing dense and Mixture-of-Experts foundational and instruction-tuned variants. It combines large-scale multilingual pre-training with supervised and preference-based post-training, and reports stronger performance than prior open-weight baselines across many capabilities. The resulting models span 0.5B to 72B parameters, support approximately 30 languages, and are openly released for research and applications.

  • Problem

    Open-weight language models need broad capability across language understanding, generation, multilingualism, coding, mathematics, and reasoning while remaining accessible for research and applications.

  • Method

    Qwen2 provides dense and Mixture-of-Experts base and instruction-tuned Transformer models trained on over 7 trillion tokens and aligned using supervised fine-tuning and preference optimization.

  • Results

    Qwen2 outperforms previous open-weight models and is competitive with proprietary models across language, coding, mathematics, reasoning, generation, and multilingual benchmarks.

  • Takeaways & Limitations

    The openly accessible Qwen2 weights support community research and applications across diverse model sizes and approximately 30 languages.

Abstract

from arXiv · show

This report introduces the Qwen2 series, the latest addition to our large language models and large multimodal models. We release a comprehensive suite of foundational and instruction-tuned language models, encompassing a parameter range from 0.5 to 72 billion, featuring dense models and a Mixture-of-Experts model. Qwen2 surpasses most prior open-weight models, including its predecessor Qwen1.5, and exhibits competitive performance relative to proprietary models across diverse benchmarks on language understanding, generation, multilingual proficiency, coding, mathematics, and reasoning. The flagship model, Qwen2-72B, showcases remarkable performance: 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH as a base language model. The instruction-tuned variant, Qwen2-72B-Instruct, attains 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench. Moreover, Qwen2 demonstrates robust multilingual capabilities, proficient in approximately 30 languages, spanning English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, Vietnamese, and more, underscoring its versatility and global reach. To foster community innovation and accessibility, we have made the Qwen2 model weights openly available on Hugging Face and ModelScope, and the supplementary materials including example code on GitHub. These platforms also include resources for quantization, fine-tuning, and deployment, facilitating a wide range of applications and research endeavors.

1 INTRODUCTION

Qwen2 is a family of foundational and instruction-tuned Transformer language models spanning multiple dense and Mixture-of-Experts configurations. It uses expanded multilingual, code, and mathematics data and achieves strong results across language, coding, mathematics, and reasoning benchmarks.

  • Model family: Qwen2 comprises four dense models from 0.5B to 72B parameters and one 57B Mixture-of-Experts model with 14B parameters active per token.The series includes foundational base models and instruction-tuned models for chat and agent applications.
  • Training: Over 7 trillion tokens of multilingual, code, and mathematics data were used for pre-training, while supervised fine-tuning and direct preference optimization aligned the models with human preferences.The broader code and mathematics content was intended to improve reasoning abilities.
  • Evaluation: Qwen2-72B-Instruct scores 9.1 on MT-Bench, 48.1 on Arena-Hard, and 35.7 on LiveCodeBench.These results concern the instruction-tuned variant.
  • Evaluation: Qwen2-72B achieves 84.2 on MMLU, 37.9 on GPQA, 64.6 on HumanEval, 89.5 on GSM8K, and 82.4 on BBH.These results concern the base language model.

2 TOKENIZER & MODEL

Qwen2 combines a shared multilingual tokenizer with Transformer dense and Mixture-of-Experts architectures. Its design includes grouped-query attention for inference efficiency and long-context mechanisms that segment sequences and rescale attention weights.

  • Tokenizer: All Qwen2 models share a byte-level byte-pair tokenizer with 151,643 regular tokens and 3 control tokens.The tokenizer’s compression efficiency supports multilingual capabilities.
  • Architecture: Qwen2 uses causal-mask Transformer models in four dense scales and one Mixture-of-Experts configuration.The MoE architecture replaces the original feed-forward network with multiple expert FFNs selected by a gated network.
  • Dense models: Grouped Query Attention replaces conventional multi-head attention to optimize KV-cache usage and enhance inference throughput.The model retains other components including SwiGLU, RoPE, QKV bias, RMSNorm, and pre-normalization.
  • Long context: Dual Chunk Attention segments long sequences into manageable chunks, while YARN rescales attention weights for length extrapolation.DCA matches original attention within a chunk and captures relative positional information across chunks.
  • Configurations: Qwen2 models have lower KV size per token than Qwen1.5 models, reducing memory footprint for long-context inference.This property is reported across the Qwen2 model configurations.

3 PRE-TRAINING

Qwen2’s pre-training uses a larger, higher-quality multilingual dataset with expanded code and mathematics content, while extending context handling to much longer sequences. The authors selected 7 trillion tokens after a larger dataset failed to improve performance significantly.

  • Data composition: The expanded dataset contains substantially more high-quality code, mathematics, and multilingual data and supports approximately 30 languages.Examples include English, Chinese, Spanish, French, German, Arabic, Russian, Korean, Japanese, Thai, and Vietnamese.
  • Data scale: The pre-training corpus was expanded from 3 trillion tokens in Qwen1.5 to 7 trillion tokens in Qwen2.The dataset also increased in scale, quality, and diversity across several domains.
  • Data scale: A 12 trillion-token dataset did not produce significant performance improvement over the 7 trillion-token model, so the authors selected the higher-quality 7 trillion-token dataset for larger models.The decision also considered training costs.
  • Training allocation: All dense models except Qwen2-0.5B used the dataset of over 7 trillion tokens, while Qwen2-0.5B used the 12 trillion-token dataset.The MoE model received an additional 4.5 trillion tokens of pre-training.
  • Long context: Qwen2 expanded context length from 4,096 to 32,768 tokens during the concluding phase of pre-training and introduced more high-quality lengthy data.The RoPE base frequency was changed from 10,000 to 1,000,000 for long-context performance.
  • Long context: YARN and Dual Chunk Attention enable sequences up to 131,072 tokens while maintaining high performance with minimal preliminary perplexity degradation.These mechanisms target length extrapolation and long-context processing.

4 POST-TRAINING

Qwen2 post-training combines supervised demonstrations, preference optimization, human annotation, and automated synthesis across diverse skills and safety domains. The pipeline uses task-specific verification and repurposing strategies to scale data construction beyond extensive human supervision.

  • Post-training goals: Post-training targets coding, mathematics, logical reasoning, instruction following, multilingual comprehension, and alignment with human values.The process emphasizes scalable alignment with minimal human annotation.
  • Training data: Demonstration data pairs instructions with satisfactory responses, while preference data contains preferred and alternative responses for optimization.The dataset notation distinguishes demonstration data D from preference data P.
  • Data construction: Training-data construction first uses collaborative annotation and then automated synthesis across code, mathematics, instruction following, creation, role-playing, and safety.Human annotators produce target and positive or negative responses before automated alignment strategies expand coverage.
  • Instruction construction: Instructions are extracted into an ontology, selected for diversity and complexity, and evolved by adding constraints or requirements.This produces representative instructions spanning varied difficulty levels.
  • Human annotation: Human annotators rank multiple model-generated responses to yield demonstration and preference data.Responses are generated using diverse strategies and models of different scales.
  • Automated data synthesis: Automated alignment addresses the difficulty of producing expert, careful, or patient annotations at large scale.The authors identify large-scale annotation quality as a significant challenge.
  • Task-specific synthesis: Rejection sampling preserves accurate and reasonable reasoning paths for mathematical tasks, while execution feedback evaluates code by compiling and running generated test cases.Correct and incorrect paths are contrasted to form preference data.
  • Task-specific synthesis: Data repurposing uses public-domain literary works and model-generated instructions to create demonstration data for writing and role-play.Constitutional feedback separately uses predefined principles to generate aligned and deviated responses for safety data.

5 EVALUATION

Qwen2 is evaluated through a comprehensive protocol spanning general knowledge, language comprehension, generation, coding, mathematics, reasoning, and other competencies.

  • The evaluation protocol examines general knowledge understanding, language comprehension, generation, coding, mathematics, reasoning, and additional areas of expertise.

5.1 BASE LANGUAGE MODELS

Qwen2 base models are evaluated across knowledge, language, coding, mathematics, reasoning, and multilingual benchmarks using few-shot or zero-shot prompting. Across model sizes, Qwen2 generally outperforms baselines, with the largest gains in coding, mathematics, and Chinese-language understanding, while Qwen2-72B achieves the strongest overall performance.

  • Evaluation protocol: Few-shot or zero-shot benchmark prompting evaluates knowledge, natural language understanding, coding, mathematics, scientific knowledge, reasoning, and multilingual capabilities.The benchmark suite includes MMLU, GPQA, BBH, HumanEval, GSM8K, MATH, C-Eval, CMMLU, and multilingual datasets.
  • Qwen2-72B: Qwen2-72B improves over Llama-3-70B by 4.7 points on MMLU and 2.8 points on MMLU-Pro, with further gains of 1.6 on GPQA and 9.8 on Theorem QA.
  • Qwen2-72B: Qwen2-72B exceeds Qwen1.5-72B by 18.3 percentage points on HumanEval and 10.0 percentage points on MBPP.
  • Qwen2-57B-A14B: Qwen2-57B-A14B activates 14 billion parameters per forward pass while maintaining the performance level of a 30 billion parameter dense model.It performs comparably to Yi-1.5-34B in natural language understanding and outperforms baseline models in coding and mathematics.
  • Qwen2-7B: Qwen2-7B demonstrates superior performance across most datasets, particularly in coding, mathematics, Chinese-language tasks, multilingual understanding, and exams.
  • Qwen2-1.5B and Qwen2-0.5B: Qwen2-1.5B outperforms Phi-2 in language understanding, while both smaller Qwen2 models outperform competitors in mathematics and Chinese-language understanding.Phi-2 generally performs best in general reasoning, whereas Qwen2-1.5B performs best on TruthfulQA.
  • Overall results: Across model sizes, Qwen2 generally outperforms baselines, and Qwen2-72B achieves the highest performance among Qwen2 models.

5.2 INSTRUCTION-TUNED MODEL

Qwen2 instruction-tuned models are evaluated across foundational skills, human preference, long-context processing, multilingual performance, safety, and contamination controls. Results show broad strengths across model sizes and tasks, alongside a remaining instruction-following weakness for the 7B model and safety headroom.

  • Evaluation protocol: The evaluation protocol combines automatic and human assessments of language understanding, coding, mathematics, reasoning, instruction following, long context, multilingual safety, and red teaming.
  • OPEN BENCHMARK EVALUATION: Qwen2-72B-Instruct outperforms its compared instruction-tuned peers in language understanding, coding, and mathematics, except on GPQA and MBPP, while showing advantages in alignment and instruction following.
  • OPEN BENCHMARK EVALUATION: Qwen2-57B-A14B-Instruct surpasses Qwen1.5-32B-Chat in almost all benchmarks and outperforms Yi-1.5-34B-Chat in most evaluations except mathematics.
  • OPEN BENCHMARK EVALUATION: Qwen2-7B-Instruct improves over Qwen1.5-7B-Chat and is competitive with Llama-3-8B-Instruct, especially in coding, but greatly trails the competitor in instruction following.
  • OPEN BENCHMARK EVALUATION: Qwen2 smaller models outperform their Qwen1.5 counterparts across core capabilities and instruction following, supporting data scaling as effective even for sub-billion-parameter models.
  • LONG CONTEXT CAPABILITIES: YARN and DCA improve Qwen2 long-context performance on NeedleBench and LV-Eval, with Qwen2-7B-Instruct surpassing or matching ChatGLM4-9B-1M in reported comparisons.
  • MULTILINGUAL EVALUATION: Qwen2-72B-Instruct is competitive with GPT-4-Turbo and slightly behind Claude-3-Opus on average multilingual evaluations, while significantly outperforming GPT-3.5-Turbo.
  • SAFETY & RESPONSIBILITY: Qwen2-72B-Instruct generates fewer harmful responses than GPT-4 and Mixtral-8x22B-Instruct, but the authors identify substantial room for improvement, particularly on pornography.

6 CONCLUSION

Qwen2 is a versatile family of dense and Mixture-of-Experts foundational and instruction-tuned models spanning 0.5 to 72 billion parameters. It outperforms prior open-weight models and competes with proprietary systems across diverse capabilities, while its open weights support community research and applications.

  • Qwen2 spans 0.5 to 72 billion parameters and includes dense and Mixture-of-Experts foundational and instruction-tuned models.
  • Qwen2 outperforms previous open-weight models and competes with proprietary models across language, multilingual, coding, mathematics, reasoning, long-context, and safety evaluations.
  • Openly accessible Qwen2 weights enable researchers and developers to use the models in varied applications and research projects.
Loading 2407.10671v4…