Source-linked AI summary
Seed-Coder: Let the Code Model Curate Data for Itself
ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong, Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang, Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Shen, Liang Xiang, Yonghui Wu
TL;DR
Code-data curation for open-source code LLMs remains dependent on human-designed rules that are difficult to scale and maintain across languages. Seed-Coder replaces much of this effort with LLM-based scoring and filtering, then trains 8B base, instruct, and reasoning variants using specialized post-training. The resulting family is reported to achieve state-of-the-art performance across diverse coding benchmarks, while remaining limited in broader language understanding and difficult high-end reasoning tasks.
Problem
Human-centric code-data curation relies on hand-crafted rules and expertise that introduce subjectivity and are difficult to scale across programming languages.
Method
Seed-Coder uses an LLM-centered data pipeline for code-data scoring and filtering, followed by SFT and DPO for instruction tuning and LongCoT reinforcement learning for reasoning.
Results
Seed-Coder achieves state-of-the-art performance across diverse coding benchmarks, including 36.2% pass@1 on MHPP and 55.8% pass@1 on FullStack Bench for its 8B instruct model.
Takeaways & Limitations
The results support using model-centric curation to build lightweight open-source code models with strong coding and reasoning capabilities.
Takeaways & Limitations
Because Seed-Coder excludes general web data and uses fewer pretraining tokens than some models, its general language understanding and broader-task ability remain limited.
Abstract
from arXiv · showhide
Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks.
1. Introduction
Seed-Coder addresses the scalability, subjectivity, and maintenance costs of human-centric code-data curation with a model-centric pipeline. It introduces 8B base, instruct, and reasoning models built from LLM-filtered data and specialized post-training.
- The paper positions model-centric curation as a response to proprietary models’ limited disclosure of training data and open-source pipelines’ high-level reporting.
- Human-curated rules for code data are difficult to scale across programming languages and can introduce conflicts and subjective bias.
- The proposed pipeline uses LLMs instead of hand-crafted rules to score and filter code data while minimizing human involvement.It processes GitHub code, commits, and code-related web data into a 6-trillion-token pretraining corpus.
- Seed-Coder comprises open-source 8B base, instruct, and reasoning models.
- The instruct model uses synthetic-data fine-tuning and preference optimization, while the reasoning model uses LongCoT reinforcement learning.
2. Pretraining
Seed-Coder’s pretraining pipeline combines deduplicated GitHub and web data with LLM-powered quality filtering, multiple data granularities, and context-aware training formats. The pipeline also constructs large continued-pretraining corpora from web archives and curated data.
- The end-to-end pipeline decouples preprocessing and filtering modules so they can run independently on raw GitHub and web-archive data.
- GitHub Data: LLM-powered quality filters address broad code-quality variation more flexibly than rule-based filters, whose definitions and heuristics are difficult to scale.
- The filtered corpus contains file-level code, repository-level code, GitHub commits, and code-related web data.
- Regular pretraining uses file-level code and web data, while continued pretraining expands to all four categories for performance, alignment, and long-context learning.
- GitHub Data: A file-level scoring model replaces individually crafted standards by using oracle-assessed code files to train a scalable quality filter.
- Code-Related Web Data: The web-data pipeline produces approximately 1.2 trillion tokens, while continued-pretraining data construction yields approximately 130 billion tokens.
- Fill-in-the-Middle (FIM): Fill-in-the-Middle training supports context-aware completion, with the authors reporting slightly better training performance for SPM than PSM.
3. Post-training
Seed-Coder’s post-training combines supervised fine-tuning and preference optimization for instruction following with LongCoT reinforcement learning for difficult code reasoning. Its data pipelines emphasize diversity, difficulty, quality, and sandbox-based verification.
- Instruct Model: Seed-Coder-8B-Instruct is trained with supervised fine-tuning followed by direct preference optimization for code generation and reasoning.
- Instruct Model: The instruction dataset emphasizes diversity, quality, and difficulty through synthetic generation, varied seeds and styles, and code-related real-world prompts.
- Instruct Model: Quality filtering combines syntax-based rules with model scoring, while difficulty filtering removes instances scored below 3 out of 10.
- Instruct Model: Sandbox execution, generated unit tests, self-correction, and preference-pair construction provide verification signals for difficult examples and DPO training.
- Instruct Model: The two-stage instruction-training strategy is reported to improve robustness, reasoning ability, and code-generation performance.
- Reasoning Model: The reasoning model applies LongCoT warmup followed by GRPO-based reinforcement learning on challenging coding problems and verified solutions.
- Reasoning Model: The authors avoid further scaling distillation data to preserve the base model’s exploration space, despite observed gains from more data.
- Reasoning Model: Progressive scheduling improved early training efficiency, while larger rollout sizes may offer further gains according to preliminary observations.
4. Decontamination
The training corpus was decontaminated against benchmark overlap before evaluation.
- A 10-gram filtering process removed training data sharing any 10-gram overlap with key benchmark datasets.
5. Results
Seed-Coder models demonstrate strong performance across code generation, multilingual generation, code reasoning, and software-engineering evaluations. The 8B variants frequently match or outperform comparable and larger open-source models.
- Code Generation: Seed-Coder-8B-Base achieves impressive HumanEval(+) and MBPP(+) performance among similarly sized open-source models, surpassing some much larger models.The comparison uses EvalPlus, with chat-template handling for Qwen2.5-Coder and a normal base prompt for Seed-Coder.
- Code Generation: On 7 of 8 evaluated MultiPL-E languages, Seed-Coder-8B-Base outperforms comparable open-source models and sometimes exceeds models larger than 13B parameters.The evaluation covers eight mainstream languages translated from Python tasks and unit tests.
- Code Reasoning: Seed-Coder-8B-Instruct achieves top-tier CRUXEval performance in both Input-CoT and Output-CoT settings among approximately 8B models.Both Seed-Coder-8B-Instruct and Qwen2.5-Coder-7B surpass DeepSeek-Coder-33B-Base in these settings.
- Code Generation: 36.2% pass@1 on MHPP lets Seed-Coder-8B-Instruct surpass all other approximately 8B models and some much larger models, including DeepSeek-Coder-V2-Instruct.MHPP contains 210 human-curated problems targeting difficult code-generation challenges.
- Software Engineering: Seed-Coder-8B-Instruct records 55.8% pass@1 on FullStack Bench and 49.6% overall on NaturalCodeBench, outperforming several larger open-source models.It also shows strong multilingual and cross-domain results on MBXP, LiveCodeBench, code reasoning, and code-editing benchmarks.
6. Conclusion, Limitation, and Future Work
Seed-Coder is presented as a lightweight open-source family achieving state-of-the-art performance across diverse coding benchmarks with minimal human effort. The release remains limited in general natural-language understanding and broader tasks because training focuses on coding and uses less general-purpose data.
- Seed-Coder is a family of lightweight open-source code LLMs achieving state-of-the-art performance across diverse coding benchmarks.
- The models demonstrate that LLMs can curate training data with minimal human effort to enhance code intelligence and reasoning capabilities.
- Seed-Coder’s general natural-language understanding and ability to handle broader tasks remain limited because training excludes general web data.
- Seed-Coder’s 6 trillion-token pretraining volume is substantially smaller than Qwen3’s 36 trillion tokens, reflecting insufficient general-knowledge and mathematical data.
7. Contributions and Acknowledgments
The report credits contributors across core research, data, training, evaluation, infrastructure, management, and related work. It also identifies departed contributors with a star and marks core contributors with a dagger.
- Contributions: Contributors are organized across core contributions, pretraining and model training, data quality, code-related data, high-quality data, long-context training, post-training, reasoning, evaluation, data support, infrastructure, and management.
- Contributions: The acknowledgments identify additional contributors and distinguish contributors by their listed responsibilities.
- Acknowledgments: Names marked with ★ denote individuals who have departed from the team, while † indicates core contributors sorted alphabetically.
A.1. Programming Languages in GitHub Data
The GitHub pretraining data covers 89 programming languages. The listed languages span programming, markup, configuration, data, and domain-specific languages.
- The GitHub pretraining data uses 89 programming languages.
- The language list includes languages and formats ranging from C, C++, Java, and Python-related ecosystems to markup, configuration, and domain-specific languages.
A.2. Configuration of Quality Scorer
This appendix section provides implementation details for the quality scorer described earlier. The supplied passage does not specify the scorer’s configuration.
- The section provides implementation details of the quality scorer described in Section 2.2.1.
A.2.1. Prompt for GitHub Code Quality Scoring
The prompt operationalizes code-file quality scoring through explicit structural, functional, and maintainability criteria, followed by a standardized numerical rating. A zero-score policy takes precedence for certain low-value or unsuitable files.
- Evaluation procedure: The prompt asks an expert evaluator to assess each code file against quality criteria relevant to LLM pretraining.The assessment includes an explanation before the final rating.
- Output format: The evaluator rates code from 1 to 10, where 1 is the lowest quality and 10 is the highest, using the required format “Rating: [[X]]”.The prompt instructs evaluators to consider the zero-score policy first and skip other criteria when it applies.
- Quality criteria: The criteria cover documentation, formatting, naming, modularity, separation of concerns, reuse, and avoidance of repetition or debugging artifacts.They also require clear communication of code intentions and coherent module responsibilities.
- Quality criteria: The prompt additionally checks syntax and logical correctness, meaningful functionality, limited hard-coded data, and reasonable comments.These criteria target whether code is useful as pretraining data rather than merely syntactically present.
- Zero-score policy: Files meeting any zero-score condition are assigned 0 before the remaining criteria are considered.Zero-score cases include data-dominated or configuration-heavy files, code with little effective logic, and auto-generated code.
A.2.2. Ground-Truth Quality Scores and Comparison of Oracles
Ground-truth quality scores were collected across 21 programming languages and compared across three oracle models. The oracle responses were broadly consistent, and DeepSeek-V2-Chat was selected for scorer training to balance efficiency and accuracy.
- Ground-truth data: Ground-truth scores were collected for code files spanning 21 representative programming languages.The language distribution is reported in Table 17.
- Oracle comparison: GPT-4 Turbo, DeepSeek-Coder-33B, and DeepSeek-V2-Chat served as the three compared oracles.GPT-4 Turbo was used as the baseline.
- Oracle comparison: Despite minor discrepancies on zero-score decisions, all three oracles produced consistent per-sample scores and overall score distributions.Figure 12 presents the score distributions along the heatmap edges.
- Oracle selection: DeepSeek-V2-Chat was used as the training oracle for the quality scorer, balancing efficiency and accuracy.The choice followed the cross-oracle comparison.
A.2.3. Evaluation of Quality Scorer
The quality scorer is evaluated as a regression model whose predictions support flexible thresholding across score intervals. Its reported error metrics indicate reliable performance.
- Evaluation objective: The scorer filters low-quality code files and must predict accurately across arbitrary score intervals so different thresholds can be applied.Once a threshold is selected, the scorer functions as a classifier.
- Training objective: Mean squared error is used during training to fit predicted scores to ground-truth values.The scorer is trained as a regression model rather than with binary pass-or-fail labels.
- Error metrics: The evaluation reports MAE across all samples as 0.91, while also defining a class-balanced MAE for score groups.The class-balanced metric evaluates errors within subsets sharing each ground-truth score.
- Error metrics: Both reported metrics indicate that the quality scorer performs reliably.The class-balanced metric uses subsets C_i of samples with ground-truth score i, while ŷ_j denotes a predicted fractional score.