Source-linked AI summary
Qwen2.5-Coder Technical Report
Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, Yunlong Feng, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, Junyang Lin
TL;DR
Code language models need strong coding performance across tasks without losing broader capabilities. Qwen2.5-Coder addresses this gap with six Qwen2.5-based models trained and post-trained on curated, mixed code data, achieving state-of-the-art results across code evaluations. The series also explores scaling, while continued work is needed on data and model size and reasoning capabilities.
Problem
Code-specific language models require strong coding performance while preserving general and mathematical abilities across varied model sizes and tasks.
Method
Qwen2.5-Coder builds on Qwen2.5 and uses curated code, synthetic, mathematical, and general data with balanced mixing and coding-focused instruction tuning.
Results
Qwen2.5-Coder achieves state-of-the-art performance across more than ten code-focused benchmarks, with performance scaling positively with model size.
Takeaways & Limitations
The six-size Qwen2.5-Coder release supports code intelligence research and real-world developer applications through permissive licensing.
Takeaways & Limitations
Future work will examine scaling with data and model size and further improve reasoning capabilities.
Abstract
from arXiv · showhide
In this report, we introduce the Qwen2.5-Coder series, a significant upgrade from its predecessor, CodeQwen1.5. This series includes six models: Qwen2.5-Coder-(0.5B/1.5B/3B/7B/14B/32B). As a code-specific model, Qwen2.5-Coder is built upon the Qwen2.5 architecture and continues pretrained on a vast corpus of over 5.5 trillion tokens. Through meticulous data cleaning, scalable synthetic data generation, and balanced data mixing, Qwen2.5-Coder demonstrates impressive code generation capabilities while retaining general and math skills. These models have been evaluated on a wide range of code-related tasks, achieving state-of-the-art (SOTA) performance across more than 10 benchmarks, including code generation, completion, reasoning, and repair, consistently outperforming larger models of the same model size. We believe that the release of the Qwen2.5-Coder series will advance research in code intelligence and, with its permissive licensing, support wider adoption by developers in real-world applications.
1 Introduction
Qwen2.5-Coder is a six-size code-model series built on Qwen2.5 and designed for strong coding performance while retaining general and mathematical abilities. It uses large-scale curated data and achieves strong results across diverse code-focused evaluations.
- Qwen2.5-Coder extends CodeQwen1.5 with six model sizes from 0.5B to 32B parameters, targeting top-tier coding performance.
- The models use over 5.5 trillion coding-related tokens from public repositories and web-crawled code-text data, filtered for quality.
- The training mixture combines code, mathematics, and general text to balance coding expertise with broader language understanding.
- Qwen2.5-Coder achieves state-of-the-art performance across more than ten code-focused benchmarks while maintaining general and mathematical reasoning abilities.
- The series is released to support code intelligence research and real-world adoption through permissive licensing.
2 Model Architecture
Qwen2.5-Coder derives its architecture and vocabulary from Qwen2.5 while adapting tokenization for code and scaling across six model sizes. The configurations vary in capacity-related dimensions as model size increases.
- Qwen2.5-Coder uses the Qwen2.5 architecture across 0.5B, 1.5B, 3B, 7B, 14B, and 32B parameter sizes.
- Most architectural dimensions increase with model size, although the 1.5B model has a larger intermediate size and the 3B model has more layers.
- The 7B and 32B models differ in hidden size, query and key-value heads, and intermediate size, reflecting their different capacities.
- All listed models use a vocabulary of 151,646 tokens, while smaller models tie embeddings and larger models do not.
- Qwen2.5-Coder inherits Qwen2.5’s vocabulary and adds special tokens for code-processing functions such as Fill-in-the-Middle.
3 Pre-training
Qwen2.5-Coder’s pre-training combines cleaned code, text-code, synthetic, mathematical, and general-text data with a selected mixture and a staged file-to-repository training pipeline. The process emphasizes data quality, long context, and code completion objectives.
- 3.1 Data Composition: Qwen2.5-Coder-Data combines source code, text-code grounding, synthetic, mathematics, and general-text data from diverse sources.
- 3.1.1 Data Composition: 4-stage filtering increased Qwen2.5-Coder-1.5B’s average HumanEval and MBPP scores from 41.6% to 46.8% versus baseline.
- 3.1 Data Composition: Synthetic code generated by CodeQwen1.5 is validated with an executor so that only executable code is retained.
- 3.1.2 Data Mixture: The final mixture uses 70% Code, 20% Text, and 10% Math, and the resulting training dataset contains 5.2 trillion tokens.
- 3.2 Training Policy: Training proceeds through file-level pretraining, repository-level pretraining, and instruction tuning.
- 3.2 Training Policy: File-level pretraining uses next-token prediction and Fill-in-the-Middle over 8,192-token sequences, while repository-level pretraining extends context to 32,768 tokens and supports sequences up to 131,072 tokens.
4 Post-training
Qwen2.5-Coder’s post-training constructs and filters coding instructions, verifies code across languages, and applies coarse-to-fine tuning and preference optimization. These stages target instruction quality, code correctness, long-context completion, and alignment.
- 4.1 A Recipe for Instruction Data: Instruction data is synthesized from code snippets, multilingual language-specific agents, and checklist-based scoring of quality dimensions.
- 4.1 A Recipe for Instruction Data: A multilingual sandbox performs static checking, generates unit tests, and evaluates self-contained code snippets across programming languages.
- 4.2 Fine-tuning: Coarse-to-fine fine-tuning begins with tens of millions of diverse samples and then uses millions of higher-quality samples with rejection sampling and supervised fine-tuning.
- 4.2 Fine-tuning: Mixed tuning uses Fill-in-the-Middle instruction pairs and tree-sitter to extract basic logic blocks as infilling targets.
- 4.3 Direct Preference Optimization: Offline direct preference optimization combines code-execution feedback from a multilingual sandbox with LLM-based judgment feedback.
5 Decontamination
Qwen2.5-Coder underwent decontamination across pre-training and post-training data to reduce test-set leakage. The process removed benchmark data and filtered training samples using 10-gram overlap.
- HumanEval, MBPP, GSM8K, and MATH were removed from the training data.
- A 10-gram word-level overlap filter removed training data overlapping with test data.
6 Evaluation on Base Models
The base models were evaluated across code generation, completion, reasoning, mathematics, general language understanding, and long-context modeling. Qwen2.5-Coder achieved strong results across these evaluations, including state-of-the-art performance and competitiveness with larger models.
- Evaluation scope: The evaluation covered six aspects: code generation, completion, reasoning, mathematical reasoning, natural language understanding, and long-context modeling.Evaluation code was publicly released for reproducibility.
- Code generation: Qwen2.5-Coder achieved state-of-the-art code-generation results among same-size open-source models and surpassed larger models.The 7B model outperformed DS-Coder-33B across all five reported metrics.
- Code generation: HumanEval and MBPP measured basic code generation, while BigCodeBench-Complete assessed tool use, complex instruction following, and out-of-distribution generalization.BigCodeBench-Complete contains full and hard subsets.
- Code generation: Qwen2.5-Coder maintained strong BigCodeBench-Complete performance, indicating generalization beyond HumanEval and MBPP.
- Multi-programming language: Qwen2.5-Coder achieved state-of-the-art MultiPL-E results and scored over 60% in five of eight evaluated languages.The results were described as balanced across languages.
- Code completion: On HumanEval-FIM, Qwen2.5-Coder-1.5B improved average performance by 3.7%, while the 7B model matched DS-Coder-33B-Base.The benchmark used Exact Match for single-line infilling across Python, Java, and JavaScript.
- Code completion: Qwen2.5-Coder-32B achieved a 3.7% improvement on CrossCodeEval and state-of-the-art RepoEval results with average gains of 7.9% EM and 4.2% ES.The series also matched models above 20B parameters at smaller sizes on these repository and cross-file tasks.
- Code completion: Qwen2.5-Coder-32B achieved state-of-the-art CrossCodeLongEval performance, although all models had low Exact Match on function completion tasks.The authors attributed the low scores likely to the difficulty of precisely matching multi-line code.
7 Evaluation on Instruct Models
The instruct models were evaluated across coding, reasoning, editing, mathematical, and general-language tasks, with strong results across code generation, reasoning, multilingual programming, and editing.
- Code Generation: Qwen2.5-Coder-7B-Instruct achieved 41.0% on BigCodeBench-Instruct full and 18.2% on its hard subset, outperforming similarly sized instruct models.
- Code Generation: Qwen2.5-Coder-32B-Instruct reached 49.6% on BigCodeBench-Instruct full and 27.0% on its hard subset, surpassing several closed-source APIs.
- Code Generation: 37.6% Pass@1 on LiveCodeBench was achieved by Qwen2.5-Coder-7B-Instruct, outperforming similarly sized and several larger models.
- Code Reasoning: 65.8% Input-CoT and 65.9% Output-CoT were achieved by Qwen2.5-Coder-7B-Instruct on CRUXEval, exceeding DS-Coder-V2-Lite-Instruct by 12.8% and 13.0%.
- Code Reasoning: 75.2% Input-CoT and 83.4% Output-CoT were achieved by Qwen2.5-Coder-32B-Instruct on CRUXEval, outperforming other open-source code models.
- Code Editing: 60.9% Pass@1 and 73.7% Pass@2 were achieved by Qwen2.5-Coder-32B-Instruct on Aider code-editing tasks.
8 Discussion: Scaling is All You Need
Across MBPP-3shot for base models and recent LiveCodeBench questions for instruction models, performance increased with model size, and Qwen2.5-Coder achieved state-of-the-art results at every size.
- Qwen2.5-Coder achieved state-of-the-art performance across all evaluated model sizes on MBPP-3shot and LiveCodeBench.MBPP-3shot was selected for base models, while LiveCodeBench questions from 2024.07–2024.11 were used for instruction models to avoid test contamination.
- Model performance showed a positive correlation with model size on MBPP-3shot and LiveCodeBench.
9 Conclusion
Qwen2.5-Coder extends Qwen2.5 through large-scale pretraining, curated code data, and instruction tuning across six model sizes. The authors identify further scaling and reasoning improvements as future directions.
- Qwen2.5-Coder comprises six Qwen2.5-based models ranging from 0.5B to 32B parameters, trained with curated public-code and web-derived data.
- Future work will examine the effects of scaling code-LLM data and model size while improving reasoning capabilities.