Source-linked AI summary
OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
Wasi Uddin Ahmad, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Vahid Noroozi, Somshubra Majumdar, Boris Ginsburg
TL;DR
Code LLM progress is limited by the scarcity of high-quality, public instruction-tuning data. OpenCodeInstruct constructs a 5-million-sample dataset with generated solutions, tests, execution feedback, and LLM assessments, then uses it to fine-tune Llama3 and Qwen2.5-Coder. Across major code-generation benchmarks, the resulting models substantially outperform their instruction-tuned counterparts, while filtering remains constrained by a diversity–correctness tradeoff.
Problem
High-quality code instruction data is difficult and resource intensive to produce, while many effective models do not disclose their instruction-tuning data or methods.
Method
OpenCodeInstruct combines large generic and algorithmic seed collections with synthetic instruction and solution generation, testing, execution feedback, and LLM-based quality assessment.
Results
OpenCodeInstruct fine-tuning substantially outperforms instruction-tuned counterparts across HumanEval, MBPP, LiveCodeBench, and BigCodeBench.
Takeaways & Limitations
The dataset and its analyses provide open resources and design insights for code instruction tuning research.
Takeaways & Limitations
Execution-based filtering can remove diverse questions with correct solutions when generated tests perform poorly, creating a diversity–correctness tradeoff.
Abstract
from arXiv · showhide
Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering.
1 Introduction
OpenCodeInstruct addresses the scarcity and cost of high-quality public code instruction data with a 5-million-sample dataset and extensive evaluations showing gains from fine-tuning.
- High-quality instruction data is important for aligning code LLMs with user intent, but producing it is challenging, resource intensive, and often dependent on proprietary models or data.
- OpenCodeInstruct provides 5 million synthetic Python coding samples from 1.43 million general instructions and 25,443 algorithmic questions, with tests, execution feedback, and LLM judgments.
- The dataset combines a larger and more diverse seed set with a scalable framework integrating SELF-INSTRUCT and EVOL-INSTRUCT for instruction and response generation.
- Fine-tuning Llama3 and Qwen2.5-Coder at 1B+, 3B+, and 7B+ scales substantially improved performance over their instruction-tuned counterparts.
- The study reports gains on HumanEval, MBPP, LiveCodeBench, and BigCodeBench, alongside analyses of scaling, generation techniques, seed sets, and instruction formatting.
- OpenCodeInstruct is presented as the largest publicly available code instruction tuning dataset, expanding resources for code instruction tuning and future research.
2 OPENCODEINSTRUCT: Large-scale Coding Instruction Tuning Dataset
OpenCodeInstruct builds a large coding-instruction corpus from complementary seed collections, evolutionary generation, response synthesis, testing, and quality assessment.
- 2 OPENCODEINSTRUCT: Large-scale Coding Instruction Tuning Dataset: The dataset pipeline starts from synthetic generic instructions and non-synthetic algorithmic coding questions as complementary seed collections.
- 2.1 Creation of the Initial Seed Collection: The initial-seed strategy addresses duplicate instructions by combining a broad generic collection with a smaller, high-quality algorithmic collection.
- 2.1 Creation of the Initial Seed Collection: The generic seed set contains 1.43 million instructions generated from filtered GitHub Python functions, while the algorithmic set contains 25,443 TACO questions.
- 2.2 Instruction Generation: GENETIC-INSTRUCT generates instructions and code solutions through mutation and crossover operations that combine ideas from EVOL-INSTRUCT and SELF-INSTRUCT.
- 2.2.1 Data Cleaning and Decontamination: The pipeline removes instructions containing Python code snippets and decontaminates overlaps with evaluation benchmarks before retaining approximately 5 million questions.
- 2.3 Response Generation: Solutions are generated with Qwen2.5-Coder-32B-Instruct, while alternative models are used to analyze the impact of the coder model.
- 2.3 Response Generation: The dataset adds automatically generated coding-skill metadata and ten assertion-style unit tests per question-solution pair, then records execution results and pass rates.
- 2.4 Test Case Generation and Execution: Unit-test outcomes are generally bimodal, with many solutions passing all tests or failing completely, while LLM-as-a-judge scores requirement conformance, logical correctness, and edge-case consideration.
3 Main Evaluation
The evaluation fine-tunes Llama3 and Qwen2.5-Coder variants with OPENCODEINSTRUCT and compares them across established code-generation benchmarks. Results show substantial gains in several settings, while improvements for smaller Llama3 models on LiveCodeBench are marginal.
- Evaluation setup: Llama3 and Qwen2.5-Coder 1B+, 3B+, and 7B+ variants were fine-tuned using OPENCODEINSTRUCT for evaluation.Training used three epochs, a batch size of 2048, and a maximum sequence length of 2048.
- Function-level generation: OPENCODEINSTRUCT substantially improves Llama3 performance on HumanEval, MBPP, HumanEval+, and MBPP+ over instruction-tuned counterparts.For Qwen2.5-Coder, fine-tuned scores were competitive with or exceeded instruction-tuned counterparts.
- LiveCodeBench: 713 coding problems comprise LiveCodeBench-v4, a contamination-free benchmark spanning newly collected competitive-programming challenges.The benchmark draws problems from platforms including LeetCode, AtCoder, and CodeForces.
- LiveCodeBench: OPENCODEINSTRUCT significantly enhances Qwen2.5-Coder models on LiveCodeBench, while gains for 1B+ and 3B+ Llama3 models are marginal.The authors attribute the smaller Llama3 gains likely to LiveCodeBench sample complexity requiring models larger than 7B.
- BigCodeBench-Instruct: BigCodeBench-Instruct evaluates complex function-calling tasks using 1,140 tasks, 5.6 test cases per task, 139 libraries, and 7 domains.The evaluation table compares instruction-tuned models across HumanEval, MBPP, LiveCodeBench, and the BigCodeBench instruct subset.
4 Analyses and Findings
Analyses show that filtering, dataset scale and composition, instruction format, solution-generator choice, and dataset quality all affect code-generation performance. Across these studies, LLM-based judgment, larger and more diverse seeds, NL-to-Code formatting, stronger generators, and OpenCodeInstruct improve outcomes within the tested settings.
- Effectiveness of LLM-based Filtering and Verification: LLM-as-a-judge filtering outperforms unit-test-based filtering and random selection, while execution pass rate correlates with judgment scores.Unit-test filtering improves over samples failing all tests but only marginally over random selection; LLM judgment is the strongest verifier reported.
- Impact of Synthetic Data Size: 500k samples already surpass the original instruct-tuned models, while performance on MBPP consistently improves through the full 5 million samples.The trend is shown for Qwen2.5-Coder-7B-Base and Llama-3.1-8B-Base; Llama3 gains more across sample sizes.
- Impact of Seed Population: Large-scale and combined algorithmic-plus-generic seed populations yield stronger results, although instruction-generation algorithms remain competitive and benchmark-dependent.The combined instruction set outperforms separate subsamples, while seed population size, coverage, and diversity affect synthetic-data quality.
- Impact of Instruction Formatting: NL-to-Code vs. Code-to-Code: NL-to-Code instructions significantly outperform Code-to-Code instructions across HumanEval, MBPP, and the other evaluated benchmarks.This comparison uses Qwen2.5-32B-Instruct after reformatting OpenCodeInstruct instructions with few-shot prompting.
- Impact of Code Generation Models: Using a stronger solution-generation model improves the benchmarks of models distilled from its generated solutions.Qwen2.5-Coder-32B-Instruct has the highest reported HumanEval, MBPP, and BigCodeBench scores among the compared generators.
- OSS-Instruct Samples vs. OpenCodeInstruct: With equal 4-million-sample budgets, OpenCodeInstruct improves Llama-3.1-8B by 9.8 HE+ and 5.6 MBPP+, and Qwen2.5-Coder-7B by 9.4 HE+ and 2.0 MBPP+ over OSS-Instruct.The comparison supports higher per-sample quality alongside OpenCodeInstruct's larger overall dataset size.
5 Related Work
Code instruction tuning builds on synthetic instruction-response generation to address the difficulty and expense of obtaining high-quality instructional data. Related work spans general synthetic-data methods and code-focused applications across software engineering tasks.
- Code LLMs have been applied to repository-level generation, automated program repair, performance optimization, and code translation.
- Instruction tuning improves LLMs by fine-tuning them on instruction-response pairs.
- Synthetic data generation has gained adoption because acquiring high-quality instructional data is difficult.
6 Conclusion
The paper presents OPENCODEINSTRUCT as the largest LLM-generated code instruction-tuning dataset and evaluates it across multiple model sizes and code-generation benchmarks. Fine-tuning with the dataset significantly outperforms corresponding instruction-tuned models and supports analysis of pipeline design choices.
- OPENCODEINSTRUCT is presented as the largest LLM-generated code instruction-tuning dataset to date.
- Fine-tuning Llama3 and Qwen2.5-Coder with OPENCODEINSTRUCT significantly outperforms their instruction-tuned counterparts on HumanEval, MBPP, LiveCodeBench, and BigCodeBench.
- The paper analyzes how OPENCODEINSTRUCT pipeline design choices affect downstream code-generation tasks.
- The dataset is intended to be fully open-sourced for future LLM-for-code research.
Supplementary Material: Appendices
The appendices document OPENCODEINSTRUCT’s data-generation and evaluation prompts, example code and assertions, quality-assessment rubric, and visualizations of sample quality and error patterns. Together, these materials expose how coding skills, tests, judgments, and generated solutions are represented and assessed.
- Quality visualizations: Figure 5 visualizes the distribution of average unit-test pass rates for OPENCODEINSTRUCT samples.
- Quality visualizations: Figure 6 visualizes the distribution of average LLM-as-a-judge scores for OPENCODEINSTRUCT samples.
- Test generation: The test-case prompt requires ten immediately executable assertion statements without testing-library imports or redundant formatting.
- Test generation: The example first_repeated_char solution is accompanied by assertions covering repeated symbols, repeated letters, empty and singleton strings, and uniform input.
- Generation prompts: The appendices include prompts for generating coding instructions, test cases, LLM judgments, and code-derived skills.
- Solution evaluation: The evaluation rubric scores requirement conformance, logical correctness, and edge-case consideration on five-level scales.