Source-linked AI summary
DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, Wenfeng Liang
TL;DR
Open-source code models lag behind powerful closed-source systems, limiting broad research and development. DeepSeek-Coder introduces models trained on project-level code with Fill-In-the-Middle training and a 16K context, and its evaluations report leading open-source performance alongside GPT-3.5 Turbo comparisons. The paper also notes possible data contamination in some evaluation data.
Problem
The performance gap between open-source and closed-source code models, whose proprietary nature limits accessibility, motivates more capable open-source alternatives.
Method
DeepSeek-Coder trains 1.3B–33B open-source models from scratch on project-level code using next-token prediction, Fill-In-the-Middle training, and a 16K context window.
Results
DeepSeek-Coder-Base 33B surpasses existing open-source code models across standard tests, while DeepSeek-Coder-Instruct 33B outperforms GPT-3.5 Turbo across a range of coding tasks.
Takeaways & Limitations
The series provides openly shared code-focused models spanning multiple scales, with results reported across public code benchmarks and coding-related tasks.
Takeaways & Limitations
The authors caution that data contamination cannot be entirely ruled out when evaluating models with the released LeetCode data.
Abstract
from arXiv · showhide
The rapid development of large language models has revolutionized code intelligence in software development. However, the predominance of closed-source models has restricted extensive research and development. To address this, we introduce the DeepSeek-Coder series, a range of open-source code models with sizes from 1.3B to 33B, trained from scratch on 2 trillion tokens. These models are pre-trained on a high-quality project-level code corpus and employ a fill-in-the-blank task with a 16K window to enhance code generation and infilling. Our extensive evaluations demonstrate that DeepSeek-Coder not only achieves state-of-the-art performance among open-source code models across multiple benchmarks but also surpasses existing closed-source models like Codex and GPT-3.5. Furthermore, DeepSeek-Coder models are under a permissive license that allows for both research and unrestricted commercial use.
1. Introduction
DeepSeek-Coder addresses the gap between accessible open-source and powerful closed-source code models with a broad open-source model series and project-level training. Across public code benchmarks, its largest models outperform open-source alternatives and, after instruction tuning, GPT-3.5 Turbo on most evaluated tasks.
- Motivation and approach: The DeepSeek-Coder series spans 1.3B to 33B parameters and includes base and instruction-tuned models trained from scratch on 2 trillion tokens from 87 programming languages.The training data is organized at repository level, and pretraining combines next-token prediction with Fill-In-the-Middle using a 16K context length.
- Evaluation results: DeepSeek-Coder-Base 33B consistently delivers superior performance among open-source models across the evaluated public code benchmarks.The experiments cover a variety of public code-related benchmarks.
- Evaluation results: DeepSeek-Coder-Instruct 33B surpasses OpenAI GPT-3.5 Turbo on the majority of evaluation benchmarks, narrowing the performance gap with GPT-4.The paper attributes this result to instruction fine-tuning using instructional data.
- Evaluation results: DeepSeek-Coder-Base 7B performs competitively against models five times larger, including CodeLlama-33B.This comparison is reported despite the smaller model’s lower parameter count.
- Contributions: The contributions include repository-level data construction, analysis of FIM training strategies, and extensive evaluation across code-related tasks.The paper reports that repository-level construction significantly boosts cross-file code generation.
2. Data Collection
The dataset is built from filtered public repository code and supplementary language data, while preserving project structure through dependency ordering and repository-level deduplication. Quality screening further removes low-quality or potentially unsuitable material before training-data compilation.
- Data composition: The training dataset contains 87% source code, 10% English code-related natural language, and 3% code-unrelated Chinese natural language.The English material comes from GitHub Markdown and StackExchange, while the Chinese material supports Chinese-language understanding.
- Collection and filtering: Public GitHub repositories created before February 2023 are restricted to 87 programming languages and preliminarily filtered using code-quality rules.The filtering reduces the dataset to 32.8% of its original size.
- Collection and filtering: The filtering rules remove files with extreme line lengths, insufficient alphabetic content, certain XML patterns, or inadequate visible text in HTML.The HTML rule retains files whose visible text is at least 20% of the code and no less than 100 characters.
- Repository structure: Dependency analysis orders files so that each file’s prerequisite context appears earlier in the input sequence.The procedure parses file dependencies and uses a modified topological sort that can handle cycles within dependency subgraphs.
- Repository structure: Near-deduplication is applied to concatenated repository code rather than individual files to preserve repository structure.The approach treats each repository-level concatenation as a single sample during deduplication.
- Quality screening: Compiler checks, a quality model, and heuristic rules further remove syntax errors, poor readability, and low modularity.The cleaned-data summary reports 798 GB across 603 million files, organized by language in Table 1.
3. Training Policy
DeepSeek-Coder combines next-token prediction with Fill-In-the-Middle training to support both sequential generation and code infilling. Ablations select a 50% PSM policy because full FIM maximizes infilling performance but weakens ordinary code completion.
- Next Token Prediction: Next-token prediction trains the model to predict each subsequent token from a fixed-length sequence formed by concatenating files.It is the model’s first stated training objective.
- Fill-in-the-Middle: Fill-In-the-Middle randomly divides text into prefix, suffix, and middle segments, then trains the model to reconstruct the middle using PSM or SPM ordering.PSM orders Prefix-Suffix-Middle, whereas SPM orders Suffix-Prefix-Middle.
- Ablation study: The FIM ablation evaluates DeepSeek-Coder-Base 1.3B on the Python HumanEval-FIM benchmark across multiple PSM, FIM-rate, and MSP configurations.HumanEval-FIM masks one line from a HumanEval solution and tests prediction of the missing line.
- Ablation study: 100% FIM produces the best HumanEval-FIM performance but the weakest code-completion capability, revealing a trade-off between infilling and completion.The comparison is made across the tested training configurations.
- Ablation study: The selected training policy is a 50% PSM rate, which outperforms MSP while balancing FIM efficiency and code-completion proficiency.The implementation applies FIM at the document level before packing, using a 0.5 rate in PSM mode.
3.2. Tokenizer
The tokenizer uses byte-pair encoding with a 32,000-token vocabulary, while the model family spans multiple parameter scales.
- Tokenizer: 32,000-token vocabulary is used for the Byte Pair Encoding tokenizer.The tokenizer is trained with the HuggingFace Tokenizer library on a subset of the training corpus.
- Model scales: The model family includes 1.3B, 6.7B, and 33B-parameter models built on a shared decoder-only Transformer framework.The architecture uses RoPE; the 33B model additionally integrates GQA with group size 8 and FlashAttention v2.
3.4. Optimization
Training uses AdamW with scaling-law-based batch sizes and learning rates, implemented in an efficiency-oriented framework with multiple parallelism strategies.
- Optimization: AdamW is used with β1=0.9 and β2=0.95, while batch sizes and learning rates follow DeepSeek LLM scaling laws.The learning-rate schedule uses three stages with 2000 warm-up steps, but the supplied passages truncate the remaining schedule details.
- Training environment: HAI-LLM combines tensor parallelism, ZeRO data parallelism, and PipeDream pipeline parallelism to optimize training efficiency.Experiments use NVIDIA A100 and H800 GPU clusters connected through intra-node and inter-node high-speed links.
3.6. Long Context
The paper extends context handling through RoPE rescaling and develops an instruction-tuned model that supports complete solutions in multi-turn coding dialogues.
- 3.6. Long Context: RoPE scaling increases the scaling factor from 1 to 4 and the base frequency from 10000 to 100000 for extended contexts.The model receives an additional 1000 training steps after this reconfiguration.
- 3.7. Instruction Tuning: DeepSeek-Coder-Instruct is produced by instruction-based fine-tuning of DeepSeek-Coder-Base on high-quality human instructions.Training uses Alpaca-format data, an <|EOT|> delimiter, a cosine schedule, 100 warm-up steps, a 1e-5 initial learning rate, and 2B total tokens.
- 3.7. Instruction Tuning: A multi-turn snake-game example shows the model generating a runnable game and then adding scoring functionality with an explanation.The example illustrates complete solutions in multi-turn dialogue settings.
4. Experimental Results
DeepSeek-Coder is evaluated across code generation, FIM completion, cross-file completion, and program-based math reasoning, with strong results across several benchmarks. The results also identify data contamination as an evaluation caveat and show that chain-of-thought prompting can improve performance on challenging subsets.
- Experimental scope: The evaluation covers code generation, FIM code completion, cross-file code completion, and program-based math reasoning against prior state-of-the-art models.The supplied benchmark descriptions include multilingual HumanEval and MBPP, DS-1000, LeetCode Contest, and CrossCodeEval settings.
- Code Generation: 50.3% HumanEval accuracy and 66.0% MBPP accuracy are reported for DeepSeek-Coder-Base, with 9% and 11% improvements over CodeLlama-Base 34B.DeepSeek-Coder-Base 6.7B also surpasses CodeLlama-Base 34B, while instruction fine-tuning surpasses GPT-3.5-Turbo on HumanEval.
- Code Generation: DeepSeek-Coder achieves relatively high accuracy across all DS-1000 libraries, including practical data-science workflows.The benchmark evaluates executable code across seven libraries and reports pass@1 results by library and overall.
- Code Generation: 19.4% and 27.8% Pass@1 are achieved by DeepSeek-Coder-Instruct 6.7B and 33B on the LeetCode Contest benchmark.The 33B instructed model is the only open-source model reported to outperform GPT-3.5-Turbo on this task, while remaining below GPT-4-Turbo.
- Analysis: Chain-of-thought prompting improves DeepSeek-Coder-Instruct performance, particularly on more challenging task subsets.The prompt asks the model to write a step-by-step outline before producing code.
- Limitations: Data contamination cannot be entirely ruled out despite efforts to collect recent code questions for testing.The authors specifically caution about contamination when evaluating models with the released LeetCode data.
- Fill-in-the-Middle Code Completion: A 0.5 FIM training rate supports code completion from both prefix and suffix context, and the 1.3B model outperforms larger StarCoder and CodeLlama counterparts in the reported FIM benchmarks.The authors associate this result with the quality of the pre-trained data and report performance increasing with model size.
- Cross-file Code Completion: DeepSeek-Coder consistently outperforms other models in cross-file completion across multiple languages under the reported exact-match and edit-similarity evaluation.The evaluation uses 2048-token sequences, 50-token outputs, and up to 512 tokens of retrieved cross-file context.
5. Continue Pre-Training From General LLM
DeepSeek-Coder-v1.5 7B is further pretrained from DeepSeek-LLM-7B Base and evaluated against DeepSeek-Coder 6.7B across programming, math reasoning, and natural-language tasks. It improves most non-coding tasks while slightly reducing coding performance.
- 5. Continue Pre-Training From General LLM: DeepSeek-Coder-v1.5 7B is created by additional pre-training from DeepSeek-LLM-7B Base using the data sources listed in Table 9.Unlike DeepSeek-Coder, this version uses next-token prediction with a 4K context length.
- 5. Continue Pre-Training From General LLM: The comparison with DeepSeek-Coder 6.7B reruns all benchmarks through the authors’ evaluation pipeline for a fair comparison.The evaluation covers programming, math reasoning, and natural-language categories.
- 5. Continue Pre-Training From General LLM: DeepSeek-Coder-Base-v1.5 improves across most tasks despite a slight decrease in coding performance.Its strongest reported gains occur across all benchmarks in the Math Reasoning and Natural Language categories.
6. Conclusion
The conclusion presents DeepSeek-Coder as a family of code-focused models trained with project-level data and a fill-in-the-blank objective. It reports strong open-source benchmark performance, GPT-3.5 Turbo outperformance by Instruct 33B, and improved natural-language comprehension in v1.5.
- 6. Conclusion: DeepSeek-Coder spans 1.3B, 6.7B, and 33B parameters and uses project-level code data, fill-in-the-blank pre-training, and a 16,384-token context window.The Base 33B model surpasses existing open-source code models across standard tests, while Base 6.7B performs on par with 34B CodeLlama.
- 6. Conclusion: DeepSeek-Coder-Instruct 33B outperforms OpenAI GPT-3.5 Turbo across a range of coding-related tasks after instruction fine-tuning.The fine-tuning uses high-quality instructional data to augment the Base models’ zero-shot instruction capabilities.
- 6. Conclusion: DeepSeek-Coder-v1.5 maintains high-level coding performance while improving natural-language comprehension after additional pretraining from the DeepSeek-LLM 7B checkpoint.The conclusion links coding-task interpretation and execution to understanding human instructions expressed in natural language.
A. Cases of Chatting with DeepSeek-Coder-Instruct
The case studies examine multi-turn interactions in database construction, data analysis, and LeetCode problem solving. The database example reports comprehensive, bug-free code with explanatory details, while the supplied passages mainly describe the interaction setup.
- A. Cases of Chatting with DeepSeek-Coder-Instruct: The case studies cover a multi-turn conversation about building a student database and analyzing age distributions, plus a LeetCode problem-solving interaction.The database scenario asks for ten random student records before requesting an age-distribution analysis.
- A. Cases of Chatting with DeepSeek-Coder-Instruct: Figure 5 reports that the model generated comprehensive, bug-free code accompanied by explanatory details for the database interaction.
- A. Cases of Chatting with DeepSeek-Coder-Instruct: The database response uses SQLite in Python to create a students table, insert ten random records, and display them.The table includes id, name, age, and grade columns.
- A. Cases of Chatting with DeepSeek-Coder-Instruct: The follow-up analysis uses pandas and matplotlib to read the database and plot a histogram of students’ age distribution.The script reads the data into a pandas DataFrame and labels the histogram by age and student count.
B. Benchmark curves during training of DeepSeek-Coder-Base
The training analysis presents benchmark curves for DeepSeek-Coder-Base models. Validation uses a curated subset of 8,000 code files, with performance metrics shown in the final two subfigures.
- B. Benchmark curves during training of DeepSeek-Coder-Base: Figure 7 presents benchmark curves showing DeepSeek-Coder-Base model performance during training.
- B. Benchmark curves during training of DeepSeek-Coder-Base: Validation uses 8,000 code files selected as a diverse and representative subset of the training corpus.
- B. Benchmark curves during training of DeepSeek-Coder-Base: The performance metrics are detailed in the final two subfigures of Figure 7.