Source-linked AI summary
DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence
DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang, Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao, Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, Wenfeng Liang
TL;DR
Open-source code models still lagged leading closed-source systems on code intelligence tasks. DeepSeek-Coder-V2 continues pre-training DeepSeek-V2 on six trillion additional tokens with expanded code, math, and language data, and achieves comparable code- and math-task performance while extending language and context coverage. Its main limitation is a significant instruction-following gap in complex scenarios such as SWEbench.
Problem
Open-source code models had not closed the performance gap with state-of-the-art closed-source models in code intelligence.
Method
DeepSeek-Coder-V2 is an open-source MoE model continued-pre-trained from DeepSeek-V2 with six trillion additional tokens and expanded code, math, and natural-language data.
Results
DeepSeek-Coder-V2 achieves performance comparable to GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro on code- and math-specific tasks, including 90.2% on HumanEval and 75.7% on MATH.
Takeaways & Limitations
The model expands programming-language support from 86 to 338 and maximum context length from 16K to 128K tokens while retaining comparable general-language performance.
Takeaways & Limitations
Despite strong benchmark performance, DeepSeek-Coder-V2 has a significant instruction-following gap versus current state-of-the-art models, causing poor performance in complex tasks such as SWEbench.
Abstract
from arXiv · showhide
We present DeepSeek-Coder-V2, an open-source Mixture-of-Experts (MoE) code language model that achieves performance comparable to GPT4-Turbo in code-specific tasks. Specifically, DeepSeek-Coder-V2 is further pre-trained from an intermediate checkpoint of DeepSeek-V2 with additional 6 trillion tokens. Through this continued pre-training, DeepSeek-Coder-V2 substantially enhances the coding and mathematical reasoning capabilities of DeepSeek-V2, while maintaining comparable performance in general language tasks. Compared to DeepSeek-Coder-33B, DeepSeek-Coder-V2 demonstrates significant advancements in various aspects of code-related tasks, as well as reasoning and general capabilities. Additionally, DeepSeek-Coder-V2 expands its support for programming languages from 86 to 338, while extending the context length from 16K to 128K. In standard benchmark evaluations, DeepSeek-Coder-V2 achieves superior performance compared to closed-source models such as GPT4-Turbo, Claude 3 Opus, and Gemini 1.5 Pro in coding and math benchmarks.
1. Introduction
DeepSeek-Coder-V2 addresses the remaining gap between open-source and leading closed-source code models through continued pre-training and expanded model support. It improves coding and mathematical reasoning while maintaining comparable general-language performance.
- Open-source code models had approached but not matched leading closed-source systems, motivating DeepSeek-Coder-V2.
- DeepSeek-Coder-V2 uses 60% source code, 10% math, and 30% natural-language data, expanding programming-language coverage from 86 to 338.The source-code corpus contains 1,170B code-related tokens.
- The models extend context length from 16K to 128K tokens and enhance coding and mathematical reasoning while maintaining comparable general-language performance.
- DeepSeek-Coder-V2 is built as 16B and 236B MoE models with 2.4B and 21B activation parameters and is released under a permissive license.
- 90.2% on HumanEval and 76.2% on MBPP demonstrate code-generation performance comparable to leading closed-source models; MATH reaches 75.7% versus GPT-4o’s 76.6%.
2. Data Collection
The data pipeline combines filtered public code repositories with web text, producing a broader multilingual corpus and validating its quality through ablation experiments.
- The pre-training mixture contains 60% source code, 10% math, and 30% natural language, with code and math data receiving focused collection and filtering.
- GitHub repositories are filtered and near-deduplicated to remove low-quality and duplicated code before corpus construction.
- 821B code tokens spanning 338 programming languages remain after applying language-specific filtering rules.
- Common Crawl collection seeds retrieval with coding forums, library documentation, and mathematics websites, then recalls related pages using fastText.
- With 2T training tokens, the 1B model reaches 37.2% HumanEval and 54.0% MBPP, improving over the original corpus baseline.
3. Training Policy
Training combines next-token prediction, optional fill-in-the-middle learning, continued pre-training, alignment, and long-context extension. The resulting system performs well through 128K-token contexts and benefits from reward-model-guided reinforcement learning.
- Training objectives: The 16B model uses next-token prediction and fill-in-the-middle training, while the 236B model uses only next-token prediction.The 16B model applies the Prefix-Suffix-Middle format at a 0.5 FIM rate.
- Architecture and optimization: Training instability and gradient spikes attributed to exponential normalization are addressed by reverting to conventional normalization.
- Continued pre-training: The models continue pre-training from a DeepSeek-V2 intermediate checkpoint, reaching 10.2T total pre-training tokens.
- Long context extension: Yarn extends context to 128K tokens through two additional 1000-step stages using sequence lengths of 32K and 128K.
- Long context extension: DeepSeek-Coder-V2 performs well across all NIAH context-window lengths up to 128K.
- Reinforcement learning: GRPO reinforcement learning uses compiler feedback, test cases, and reward models; reward-model signals outperform raw compiler signals on in-house Leetcode tests.
4. Experimental Results
DeepSeek-Coder-V2 is evaluated across coding, mathematics, and general-language tasks against leading models. It achieves strong coding results, including 75.3% average performance on HumanEval and MBPP+, and 43.4% on LiveCodeBench and USACO.
- The evaluation covers coding, mathematics, and general natural-language tasks against previous state-of-the-art large language models.
- HumanEval uses 164 Python tasks verified by test cases, while MBPP evaluation uses the MBPP-Plus version and multilingual extensions of HumanEval.The extension covers C++, Java, PHP, TypeScript, C#, Bash, and JavaScript.
- 75.3% average score places DeepSeek-Coder-V2-Instruct second on HumanEval and MBPP+, behind GPT-4o at 76.4%.It records the highest scores in Java and PHP and strong results across Python, C++, C#, TypeScript, and JavaScript.
- DeepSeek-Coder-V2-Lite-Instruct scores 65.6% versus 61.9% for the larger 33B model.The comparison highlights competitive performance from the 16B model despite its smaller size.
- 43.4% on LiveCodeBench and USACO ties DeepSeek-Coder-V2-Instruct with GPT-4o among large models and ranks it second overall.GPT-4-Turbo-0409 leads with 45.7%.
4.2. Code Completion
Repository-level and fill-in-the-middle evaluations assess code completion across languages and context settings. DeepSeek-Coder-V2-Lite-Base achieves comparable repository completion with few active parameters and leads the reported FIM results.
- Repository-Level Code Completion Evaluation: RepoBench v1.1 uses December 2023 data from Python and Java repositories to avoid overlap with pre-training data.The evaluation targets open-source models below 35B parameters.
- Repository-Level Code Completion Evaluation: The repository-level evaluation tests 2k, 4k, 8k, 12k, and 16k context lengths in cross-file-first, cross-file-random, and in-file settings.Models use greedy search, generate at most 64 new tokens, and receive prompts truncated to 15,800 tokens.
- Repository-Level Code Completion Evaluation: 2.4B active parameters enable DeepSeek-Coder-V2-Lite-Base to match DeepSeek-Coder-Base 33B in Python and DeepSeek-Coder-Base 7B in Java.Its performance is lower than CodeStral, which has ten times more active parameters, but the authors associate its smaller active count with faster completion.
- Fill-in-the-Middle Code Completion: A 0.5 Fill-In-the-Middle rate trains DeepSeek-Coder-V2-Lite to complete blanks using both preceding and following code context.This is particularly advantageous for code completion tools.
- Fill-in-the-Middle Code Completion: 86.4% mean FIM accuracy leads the reported results, with 80.0% in Python, 89.1% in Java, and 87.2% in JavaScript.DeepSeek-Coder-V2-Lite-Base uses 2.4B active parameters and achieves comparable performance with larger models.
- Fill-in-the-Middle Code Completion: The FIM comparison covers Python, Java, and JavaScript and uses line exact match accuracy as its main metric.
4.3. Code Fixing
DeepSeek-Coder-V2 is evaluated on software-repair benchmarks covering real-world bugs, repository-level issues, and Python editing tasks. DeepSeek-Coder-V2-Instruct achieves the strongest reported open-source repair performance, including 73.7% on Aider.
- Benchmark setup: 238 Defects4J bugs requiring changes to only one method were selected to limit the benchmark’s long-context repair demands.Defects4J contains real-world bugs from open-source projects, with test suites for validating repairs.
- Benchmark setup: SWE-bench tests whether a model can generate patches resolving real-world GitHub issues given the associated codebase.The benchmark is designed to measure practical software-issue resolution.
- Benchmark setup: Aider evaluates Python-file modification across 133 coding tasks while checking consistency with prompt specifications.
- Results: Table 7 compares model performance on Defects4J, SWE-Bench, and Aider, while excluding Llama3-Instruct from SWE-Bench because its context length is 8K.
- Results: 73.7% on Aider was the highest score among listed models, making DeepSeek-Coder-V2-Instruct the top open-source model on that repair benchmark.DeepSeek-Coder-Instruct reached 21% on Defects4J and 12.7% on SWE-Bench among open-source models.
4.4. Code Understanding and Reasoning
CRUXEval measures code reasoning in both forward and reverse directions using Python functions paired with input-output examples. DeepSeek-Coder-V2-Instruct leads open-source models with 70.0% on input prediction and 75.1% on output-conditioned input prediction.
- Benchmark and tasks: CRUXEval contains 800 Python functions paired with input-output examples and defines forward and reverse reasoning tasks.CRUXEval-I predicts outputs from inputs, whereas CRUXEval-O predicts inputs from known outputs.
- Results: 70.0% on CruxEval-I-COT and 75.1% on CruxEval-O-COT were achieved by DeepSeek-Coder-V2-Instruct, the strongest open-source results reported.
- Results: Table 8 reports model performance on the CruxEval benchmark.
4.5. Mathematical Reasoning
DeepSeek-Coder-V2 is assessed on grade-school, competition-level, and advanced mathematical reasoning benchmarks. It reaches 75.7% on MATH and 53.7% on Math Odyssey, comparable to GPT-4o, and solves more AIME 2024 problems than other models.
- Benchmarks: GSM8K, MATH, AIME 2024, and Math Odyssey evaluate DeepSeek-Coder-V2 across grade-school and competition-level mathematical reasoning.
- Results: 75.7% on MATH and 53.7% on Math Odyssey were achieved with greedy decoding, comparable to GPT-4o.
- Results: DeepSeek-Coder-V2 solves more AIME 2024 problems than the other compared models.The table caption notes that DeepSeek-Coder-V2-Instruct can achieve 5/30 on AIME 2024 with maj@64.
- Evaluation protocol: The four mathematical benchmarks use zero-shot chain-of-thought prompting, with instructions to reason step by step and format the final answer within boxed delimiters.
4.6. General Natural Language
The paper compares DeepSeek-Coder-V2 with DeepSeek-V2 on English and Chinese general-language benchmarks. Coder-V2 is stronger on reasoning-heavy evaluations, while DeepSeek-V2 Chat remains slightly better on several general-purpose alignment benchmarks.
- Evaluation scope: The general-language comparison covers English and Chinese benchmarks including BBH, MMLU, ARC, TriviaQA, NaturalQuestions, and AGIEval.
- 236B models: Table 10 compares DeepSeek-Coder-V2 Instruct with DeepSeek-V2 Chat.
- 16B models: DeepSeek-Coder-V2-Lite-Instruct outperforms DeepSeek-V2-Lite-Chat on BBH and Arena-Hard among 16B models.
- 16B models: DeepSeek-Coder-V2-Lite-Instruct trails on TriviaQA, which the passage attributes to less web data during pre-training.
- 236B models: DeepSeek-Coder-V2-Instruct is stronger on reasoning benchmarks, particularly Arena-Hard, among 236B models.DeepSeek-V2 Chat performs slightly better on MT-bench, AlpacaEval 2.0, and AlignBench, attributed to its general-purpose alignment stage.
5. Conclusion
DeepSeek-Coder-V2 is continually pre-trained from DeepSeek-V2 with 6 trillion tokens, improving coding and mathematical reasoning while retaining comparable general-language performance. It reaches comparable performance to leading closed-source models on code and math tasks, but still has a significant instruction-following gap.
- DeepSeek-Coder-V2 is continually pre-trained from DeepSeek-V2 with 6 trillion tokens from a high-quality, multi-source corpus.
- The model significantly enhances coding and mathematical reasoning while maintaining comparable general language performance to DeepSeek-V2.
- DeepSeek-Coder-V2 achieves performance comparable to GPT-4 Turbo, Claude 3 Opus, and Gemini 1.5 Pro in code- and math-specific tasks.
- A significant instruction-following gap remains relative to current state-of-the-art models such as GPT-4 Turbo, causing poor performance in complex tasks such as SWEbench.
A. Supported Programming Languages
The supported programming-language set includes a broad range of languages and related programming or configuration formats, including ABAP, C, C#, C++, and Clojure.
- The supported set spans diverse languages and programming-related formats, including ABAP, C, C#, C++, and Clojure.