PaperlayerView exact revision on arXiv

Source-linked AI summary

OpenCoder: The Open Cookbook for Top-Tier Code Large Language Models

Siming Huang, Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang, Yuan Qi, Yinghui Xu, Wei Chu

arXiv:2411.04905v3cs.CLcs.PL

TL;DR

Open code LLMs still lack the transparency and reproducible materials needed for rigorous research. OpenCoder releases a code model alongside its datasets, processing pipeline, experiments, and training protocols, and reports performance on par with leading proprietary models while surpassing most prior open-source models at both 1B+ and 6B+ scales.

  • Problem

    Open code LLMs still trail state-of-the-art systems, while proprietary training datasets limit strong baselines and insight into top-tier code models.

    Source: §1 · p. 4

  • Method

    OpenCoder pairs a code LLM with reproducible training data, a complete data-processing pipeline, experimental ablations, and detailed training protocols.

    Source: §1 · p. 4, §8 · p. 20

  • Results

    OpenCoder is on par with leading proprietary models and surpasses most previous open-source models at both 1B+ and 6B+ parameter scales.

    Source: §8 · p. 20

  • Takeaways & Limitations

    The release provides researchers with a transparent baseline and materials to investigate training choices and develop customized code LLMs.

    Source: §1 · p. 4

    The annealing-data mixture ratio may not be ideal because of the available computing budget.

    Source: §2.2 · p. 9

Abstract

from arXiv · show

Large language models (LLMs) for code have become indispensable in various domains, including code generation, reasoning tasks and agent systems. While open-access code LLMs are increasingly approaching the performance levels of proprietary models, high-quality code LLMs suitable for rigorous scientific investigation, particularly those with reproducible data processing pipelines and transparent training protocols, remain limited. The scarcity is due to various challenges, including resource constraints, ethical considerations, and the competitive advantages of keeping models advanced. To address the gap, we introduce OpenCoder, a top-tier code LLM that not only achieves performance comparable to leading models but also serves as an "open cookbook" for the research community. Unlike most prior efforts, we release not only model weights and inference code, but also the reproducible training data, complete data processing pipeline, rigorous experimental ablation results, and detailed training protocols for open scientific research. Through this comprehensive release, we identify the key ingredients for building a top-tier code LLM: (1) code optimized heuristic rules for data cleaning and methods for data deduplication, (2) recall of text corpus related to code and (3) high-quality synthetic data in both annealing and supervised fine-tuning stages. By offering this level of openness, we aim to broaden access to all aspects of a top-tier code LLM, with OpenCoder serving as both a powerful model and an open foundation to accelerate research, and enable reproducible advancements in code AI.

1 INTRODUCTION

OpenCoder addresses the limited transparency of leading code LLMs by releasing a reproducible model and its development materials. Controlled experiments identify data-curation and training choices associated with strong code-LLM performance.

1 INTRODUCTION

  • Open-source code LLMs still trail state-of-the-art models, and proprietary training datasets limit strong baselines and insight into top-tier models.The authors identify dataset transparency as an obstacle to broader research.Source: §1 · p. 4
  • The release aims to support interpretability and data-distribution research, investigate curation pipelines, and enable customized solutions based on a transparent code LLM.The authors also aim to stimulate and accelerate the open-source code-LLM community.Source: §1 · p. 4
  • Experiments highlight data cleaning, file-level deduplication, diverse data, high-quality annealing data, and two-stage instruction tuning as important design choices.The reported tuning strategy first builds broad capabilities and then refines them with code-specific tasks, improving theoretical and practical coding performance.Source: §1 · p. 4
  • OpenCoder provides an open code LLM with a transparent, reproducible pipeline, dataset, SFT corpus, and intermediate checkpoints.The release is presented as an open cookbook for building a code LLM from scratch.Source: §1 · p. 4
  • The authors report that OpenCoder achieves top-tier results on multiple code-LLM benchmarks.The introduction describes its performance as exceeding expectations through its data processing and training methods.Source: §1 · p. 4

2 PRETRAINING DATA

OpenCoder’s pretraining data combines a curated raw-code corpus and code-related web data, with a separate annealing stage that adds algorithmic and synthetic material. The section describes the processing choices behind RefineCode and reports improved training efficiency over The Stack v2.

2.1 REFINECODE

  • RefineCode contains about 960 billion tokens across 607 programming languages, drawing on raw code and code-related web data.The corpus uses more than 130 language-specific rules with customized weight assignments.Source: §2.1 · p. 5

2.1.1 RAW CODE

  • File-level exact and fuzzy deduplication addresses GitHub code repetition, while transformations remove repetitive copyright notices and replace detected personal information with placeholders.Nearly 75% of files are completely duplicated; fuzzy deduplication removes 6% of file volume.Source: §2.1.1 · pp. 5–6
  • Raw-code filtering combines natural-language, general-code, and language-specific rules, with tailored criteria for eight commonly used languages.The rules target file size and line counts, code characteristics, and language-specific patterns; thresholds aim to remove harmful data without substantially changing its distribution.Source: §2.1.1 · pp. 6–7
  • PCA embeddings show The Stack v2 has more outliers than RefineCode, whose embeddings are more tightly clustered.The reported outliers often contain low-quality patterns such as pure-text comments, hexadecimal-only data, or very short code lacking computational logic.Source: Figure 3 · p. 7, §2.1.1 · p. 7

2.1.2 CODE-RELATED WEB DATA

  • The code-related web-data pipeline uses a seed corpus to recall and annotate relevant pages, then applies the process to additional web corpora and GitHub text.The process yields about 220GB from Common Crawl, 330GB from FineWeb, Skypile, and AutoMathText, and an additional 178GB from GitHub text.Source: §2.1.2 · pp. 7–8

2.1.3 SUMMARY

  • RefineCode significantly improves training efficiency over The Stack v2 in a comparison training 1.5B models up to 600B tokens.The comparison uses RefineCode and The Stack v2’s training subset.Source: §2.1.3 · p. 8

2.2 ANNEALING DATA

  • Annealing combines RefineCode-distribution data with Algorithmic Corpus and synthetic data; 84% comes from the original RefineCode distribution.The authors note that computing-budget limits may make this mixture ratio non-ideal.Source: §2.2 · pp. 8–9
  • The annealing additions include algorithmic examples and synthetic code snippets with tests, plus educational code-textbook snippets generated through LLM analysis.The algorithmic corpus is sampled using keywords such as “leetcode” and “def solution”; generated code snippets are retained when they pass their tests.Source: §2.2 · p. 9

3 PRETRAINING

OpenCoder provides 1.5B- and 8B-parameter models with different architectures and training configurations. Both use the WSD learning schedule, and training covered Chinese, English, and 607 programming languages.

3.1 MODEL ARCHITECTURE

  • OpenCoder offers 1.5B- and 8B-parameter models; the 1.5B model has 24 layers and a 4096-token context window, while the 8B model follows Llama-3.1-8B with 32 layers.The 1.5B model has hidden size 2240 and 14 attention and key/value heads; the 8B model has hidden size 4096 and 8 attention heads. Both use SwiGLU and a 96,640-token vocabulary.Source: §3.1 · p. 10

3.2 TRAINING DETAILS

  • The training dataset spans Chinese and English as well as 607 programming languages.Source: §3.2 · p. 10
  • The 1.5B model trained on 2 trillion tokens over four epochs, then annealed on 100 billion more, using WSD with a 2,000-step warm-up across 8 billion tokens.Its peak learning rate was 3e-4, decaying to 1e-5 during annealing; training used 256 H800 GPUs for 28,034 GPU hours.Source: §3.2 · p. 10
  • The 8B model trains for 3.5 epochs on 2.5 trillion tokens, then uses an additional 100 billion-token decay phase, incorporating code-related recall data absent from the 1.5B model.Training used 512 H100 GPUs for 96,000 GPU hours; the first 130,000 steps used a 4096-token sequence length and global batch size 2048.Source: §3.2 · pp. 10–11

4 POST TRAINING

OpenCoder builds instruction data from real-user interactions, curated code, library documentation, and LLM synthesis. It fine-tunes in two stages, moving from theoretical computer science to practical coding, and removes training examples overlapping evaluation tests.

4.1 DATA COMPOSITION

  • Instruction data combines public corpora and cleaned code-related real-user conversations, with low-quality responses regenerated to better reflect practical user scenarios.The sources include Evol-Instruct, Infinity-Instruct, McEval, WildChat, and Code-290k-ShareGPT.Source: §4.1 · p. 11
  • Educational instruction synthesis selects high-quality code snippets as seeds, generates test cases, and retains samples whose code passes execution tests.A scorer identifies seed quality, and a teacher model generates test cases for code sections in each problem.Source: §4.1 · pp. 11–12
  • Package-related instruction data uses PyDoc signatures and examples to prompt a teacher model for question-answer pairs reflecting current library usage.The stated motivation is to address outdated package syntax and external-library interfaces in pretraining data.Source: §4.1 · p. 12
  • Large-scale instruction synthesis generates diverse questions and answers from cleaned web context, then validates code with execution and unit tests before refining responses.Task specifications configure programming languages, difficulty, and task types; prompts use temperature T = 1.0.Source: §4.1 · p. 12

4.2 TWO-STAGE INSTRUCTION-TUNING

  • Two-stage fine-tuning first targets theoretical computer science, then shifts to practical coding with high-quality GitHub code.The stages aim to combine theoretical knowledge with practical coding capabilities rather than focusing on either alone.Source: §4.2 · pp. 12–13

4.3 TRAINING DETAILS

  • Stage 1 SFT runs for one epoch and Stage 2 for three epochs, using different batch sizes and learning rates.Stage 1 uses batch size 4096 and learning rate 2e-5; Stage 2 uses batch size 512 and learning rate 5e-5.Source: §4.3 · p. 13

4.4 DECONTAMINATION

  • SFT data is decontaminated by removing test-set entry points and examples with 10-gram overlap with HumanEval or MBPP tests.Source: §4.4 · p. 13

5 EXPERIMENTAL RESULTS

OpenCoder is evaluated on code completion, generation, and understanding tasks using public evaluation frameworks and a range of benchmarks. The reported multilingual results show stronger comparable-size open-source performance on McEval and effective bug finding and repair on MdEval.

5 EXPERIMENTAL RESULTS

  • The evaluation compares code models on generation, completion, and understanding tasks, including open-source and closed models.Source: §5 · p. 13

5.1 EVALUATION ON BASE MODELS

  • Base-model evaluation focuses on code completion, comparing OpenCoder-1.5B with state-of-the-art small language models using OpenCodeEval.Benchmarks include HumanEval, MBPP, and BigCodeBench; HumanEval uses zero-shot results, while MBPP reports three-shot results on 500 questions.Source: §5.1 · pp. 13–14, Table 6 · p. 14

5.2 EVALUATION ON INSTRUCT MODEL

  • Instruct models are evaluated on HumanEval, MBPP, BigCodeBench, LiveCodeBench, MultiPL-E, McEval, and MdEval, covering algorithmic tasks, multilingual generation, and debugging.LiveCodeBench uses the 2305-2409 split; MultiPL-E evaluates across multiple programming languages.Source: §5.2 · pp. 14–15, Table 7 · p. 15, Table 8 · p. 15
  • OpenCoder-8B-Instruct outperforms comparable-size open-source models on McEval multilingual generation and effectively finds and fixes bugs on MdEval multilingual repair.McEval covers 40 languages and nearly 2,000 samples; MdEval covers 18 languages and nearly 1.2K samples.Source: §5.2 · p. 15, Figure 6 · p. 16, Figure 7 · p. 16

6 ANALYSIS

The analyses compare deduplication levels, annealing-data compositions, star-based filtering, and instruction-tuning strategies. They report better downstream performance with file-level than repository-level deduplication, a drop when high-quality annealing data is removed, and consistent gains from two-stage SFT.

6.1 ANALYSIS OF THE DEDUPLICATION LEVEL

  • File-level deduplication performs substantially better than repository-level deduplication on HumanEval and MBPP, despite retaining fewer tokens.Repository-level deduplication retains almost three times as many tokens, yet 52 billion tokens have exact character-level duplicates; a further 68 billion tokens can be deduplicated by file-level post-processing. Chunk-level deduplication yields no observable benefits, and exact then file-level fuzzy deduplication is described as efficient and CPU-saving.Source: §6.1 · p. 17, Figure 8 · p. 17, Table 9 · p. 17

6.2 ANALYSIS ON THE IMPORTANCE OF HIGH-QUALITY DATA IN THE ANNEALING PHASE

  • Removing Algorithmic Corpus and Synthetic Data from annealing causes performance to drop substantially, supporting the value of high-quality data in that phase.Source: §6.2 · p. 17, Figure 9 · p. 18

6.3 ANALYSIS ON THE EFFECT OF GITHUB STARS

  • Star-filtered data yields lower training loss but worse model performance than original data, and the authors attribute the trade-off to reduced data diversity.The comparison filters by GitHub stars >= 5; the authors conclude that stars are not an optimal filtering criterion.Source: §6.3 · p. 18, Figure 10 · p. 18, Figure 11 · p. 19

6.4 ANALYSIS ON THE TWO-STAGE INSTRUCTION TUNING STRATEGY

  • Two-stage SFT consistently improves performance over the compared strategies on public benchmarks and real-world CodeArena scenarios.The comparison includes Stage1, Stage1+Stage2, and Mix Training; CodeArena uses nearly 400 human-created samples and GPT-4 judgments.Source: §6.4 · p. 19, Table 10 · p. 19

7 RELATED WORK

Prior work spans open code LLMs, benchmark development, and broader efforts to make models and training materials reproducible. The section also notes that inaccessible proprietary checkpoints limit further innovation.

Code Large Language Models

  • Open code LLMs such as CodeGen, StarCoder, CodeLlama, and DeepSeekCoder support continued innovation, while inaccessible proprietary checkpoints hinder it.The cited work also describes proprietary models as performing strongly on code benchmarks.Source: §7 · p. 19

Code Benchmarks

  • Code benchmarks assess tasks including code generation, retrieval, translation, efficiency, and repository-level completion.The section identifies multiple benchmark datasets for these task types.Source: §7 · pp. 19–20

Open Large Language Models

  • Open model and dataset releases include training details, while StarCoder and StarCoderV2 also release code pretraining corpora.Examples of broader open releases include model checkpoints, data-processing pipelines, pretraining data, and intermediate checkpoints.Source: §7 · p. 20

8 CONCLUSION & FUTURE WORK

OpenCoder releases training materials and reports top-tier code-intelligence performance, alongside ablation analyses intended to inform future training. The accompanying appendix describes a staged process for designing and inspecting heuristic filters, with example rules and thresholds.

8 CONCLUSION & FUTURE WORK

  • OpenCoder releases its complete data-processing pipeline, reproducible pretraining dataset, code SFT dataset, ablations, protocols, and intermediate checkpoints, with performance on par with leading proprietary models.It surpasses most previous open-source models at both the 1B+ and 6B+ parameter scales.Source: §8 · p. 20
  • Future work will update OpenCoder’s model and data, aiming to improve performance and expand its community influence.The authors also plan to incorporate user feedback and recent research findings.Source: §8 · p. 20

A.1 DESIGN OF FILTERING RULES

  • The filtering design separates quality-signal scoring from threshold-based retention, and recommends placing this adjustable stage late in the pipeline.Design proceeds from signal selection and coarse thresholds to rule-specific tuning, followed by manual and PPL-based inspection of sampled data.Source: §A.1 · p. 27

A.2 EXAMPLES OF FILTERING RULES

  • Tables 11 and 12 give examples of general and Python-specific filtering rules, whose thresholds may vary with programming language.The authors direct readers to pipeline implementation details for specific threshold values.Source: §A.2 · pp. 27–28, Table 11 · p. 28, Table 12 · p. 28

B ANALYSIS ON CHUNK-LEVEL DEDUPLICATION

Across the tested deduplication strategies, file-level deduplication yields the highest training efficiency, while adding chunk-level deduplication after repository-level improves on repository-level alone. The appendix also describes annotated Chinese code-like and math-like domains as seed sources for future CC datasets.

B ANALYSIS ON CHUNK-LEVEL DEDUPLICATION

  • File-level deduplication achieves the highest training efficiency; repository-level plus chunk-level deduplication outperforms repository-level alone.The authors conclude file-level deduplication is most suitable for GitHub data; Figure 12 compares Pass@1 on HumanEval and MBPP across the strategies.Source: Figure 12 · p. 29, B ANALYSIS ON CHUNK-LEVEL DEDUPLICATION · p. 29
  • Chunk-level deduplication alone is less effective than repository-level deduplication, and adding it after file-level removes only 0.04B additional data.The experiments use 4096-token chunks; Table 13 compares deduplication strategies on Python data.Source: B ANALYSIS ON CHUNK-LEVEL DEDUPLICATION · p. 29, Table 13 · p. 29

C.1 CHINESE CODE-LIKE DOMAINS ANNOTATION

  • The manually annotated Chinese code-like and math-like domains can seed page sampling for future Common Crawl datasets.Table 14 describes URL patterns using the percent sign as a wildcard.Source: §C.1 · p. 30, Table 14 · p. 30

C.2 CODE-RELATED DATA FROM GITHUB TEXT FILES

The appendix describes two ways to retain code-related content from GitHub text files and explains why it keeps structured Jupyter notebooks while discarding a redundant script format.

C.2 CODE-RELATED DATA FROM GITHUB TEXT FILES

  • For GitHub text files, filename-based filtering retained coding-context files and recalled 3% volume of the whole text part; a fastText model recalled an extra 7% file volume from the original text data.Source: §C.2 · p. 30

C.3 JUPYTER NOTEBOOKS

  • Jupyter notebooks are converted to a markdown-code-execution triplet format, while Jupyter-script format is discarded because of lower code-writing standards and high redundancy.The notebook data comes from GitHub and Meta Kaggle code.Source: §C.3 · p. 31

D COMPARISON OF REFINECODE WITH THE STACK SERIES

RefineCode is presented as a broader, more language-tailored alternative to The Stack series, with more tokens and rules, web data, and language-specific filtering. The appendix also distinguishes included programming-language types from excluded file types.

D COMPARISON OF REFINECODE WITH THE STACK SERIES

E.1 INCLUDED PROGRAMMING LANGUAGES

  • Included types span 470 code, 115 data, and 22 text types, with filtering thresholds varying slightly by data type.Code files are logic-rich, data files contain structured data, and text files are dominated by natural language.Source: §E.1 · pp. 31–32

E.2 EXCLUDED PROGRAMMING LANGUAGES

  • The appendix separately lists excluded file types, including CSV, SVG, STL, and robots.txt.Source: §E.2 · p. 32

F RAW CODE DATA COMPOSITION

The section reports that filtering and deduplication shift the representation of programming languages in RefineCode, increasing the share of commonly used languages. It also presents several prompt formats for synthesizing supervised instruction data.

F RAW CODE DATA COMPOSITION

  • After filtering, the language composition shifts significantly, with commonly used programming languages gaining representation.Figure 16 presents the top 85 languages after deduplication and filtering; Table 16 orders data-composition items by filtered file volume.Source: F RAW CODE DATA COMPOSITION · p. 33, Table 16 · p. 33

G PROMPTS FOR SFT SYNTHETIC DATA

Loading 2411.04905v3…