PaperlayerView exact revision on arXiv

Source-linked AI summary

KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Coding

Zhangchen Xu, Yang Liu, Yueqin Yin, Mingyuan Zhou, Radha Poovendran

arXiv:2503.02951v2cs.LGcs.AIcs.CL

TL;DR

Coding LLM training needs data that is both broad and difficult enough for varied coding tasks while retaining verifiable correctness. KODCODE synthesizes and self-verifies question-solution-test triplets, reformats them for post-training, and produces models with state-of-the-art performance across multiple coding benchmarks.

  • Problem

    Existing coding resources provide either limited-scale human-curated data or synthetic data that may lack diversity, complexity, and reliable response verification.

  • Method

    KODCODE synthesizes questions across diverse domains and difficulties, generates and executes tests with extra attempts for challenging problems, then reformats validated triplets and applies test-based response filtering.

  • Results

    KODCODE-tuned models achieve state-of-the-art performance across HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, outperforming larger models.

  • Takeaways & Limitations

    The dataset provides verified unit tests for supervised fine-tuning and potential reinforcement-learning use, while supporting diverse coding coverage.

  • Takeaways & Limitations

    Performance on LiveCodeBench-Hard remains limited, likely because highly challenging competition-level problems are insufficiently represented.

Abstract

from arXiv · show

We introduce KodCode, a synthetic dataset that addresses the persistent challenge of acquiring high-quality, verifiable training data across diverse difficulties and domains for training Large Language Models for coding. Existing code-focused resources typically fail to ensure either the breadth of coverage (e.g., spanning simple coding tasks to advanced algorithmic problems) or verifiable correctness (e.g., unit tests). In contrast, KodCode comprises question-solution-test triplets that are systematically validated via a self-verification procedure. Our pipeline begins by synthesizing a broad range of coding questions, then generates solutions and test cases with additional attempts allocated to challenging problems. Finally, post-training data synthesis is done by rewriting questions into diverse formats and generating responses under a test-based reject sampling procedure from a reasoning model (DeepSeek R1). This pipeline yields a large-scale, robust and diverse coding dataset. KodCode is suitable for supervised fine-tuning and the paired unit tests also provide great potential for RL tuning. Fine-tuning experiments on coding benchmarks (HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench) demonstrate that KodCode-tuned models achieve state-of-the-art performance, surpassing models like Qwen2.5-Coder-32B-Instruct and DeepSeek-R1-Distill-Llama-70B.

1 Introduction

KODCODE addresses the need for coding data that combines broad coverage, challenging problems, and verified solutions and tests. Its synthesis and evaluation pipeline produces a large verified dataset whose fine-tuned models achieve state-of-the-art results across coding benchmarks.

  • Human-curated coding datasets provide quality but limited scale, while synthetic datasets often lack diversity, complexity, or reliable response verification.
  • 447K verified question-solution-test triplets are generated through question synthesis, solution-and-test generation, and post-training data synthesis.
  • Self-verification executes generated unit tests against solutions, with additional attempts allocated to challenging questions.
  • The self-verification evaluation reports an error rate below 2.5% on human-written MBPP validation tests.
  • KODCODE-fine-tuned models achieve state-of-the-art performance across HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench.
  • The authors aim for KODCODE to support stronger supervised fine-tuning and reinforcement-learning post-training for coding models.

2 KODCODE : Synthesizing Diverse, Challenging, and Verifiable Correct Post-Training Data for Code

KODCODE constructs diverse coding questions, verifies solutions and tests through execution, and reformats validated questions for post-training. The pipeline preserves challenging problems, creates 447K triplets, and produces a test-filtered SFT dataset.

  • Coding Question Synthesis: KODCODE synthesizes questions across 12 subsets spanning domains from algorithms to package-specific knowledge and difficulty levels from basic exercises to competitive programming.
  • Coding Question Synthesis: Its Data Structures and Algorithms subsets transform Python DSA snippets into questions testing foundational understanding rather than direct code replication.
  • Solution & Test Generation: GPT-4o-0513 generates solutions and unit tests, which are executed for validation, while branch coverage analysis promotes diverse test cases.
  • Solution & Test Generation: Up to 10 fresh solution-and-test generation attempts preserve challenging questions and provide difficulty labels based on success rates.
  • Post-training Data Synthesis: 279K verified triplets emerge after Step 2, while question reformatting adds 168K triplets and raises the total to 447K for RL training.
  • Post-training Data Synthesis: DeepSeek R1 generates three responses per question, with test-based reject sampling producing the KODCODE-SFT dataset.

3 Analysis

KODCODE’s analyses assess self-verification reliability, scaling across attempts, dataset statistics, diversity, difficulty, contamination, and pipeline filtering. The results show strong verification success, broader topical coverage, varied difficulty, minimal contamination, and substantial filtering during construction.

  • Additional Attempts: Over 20% average pass-rate improvement from 1 to 5 attempts, followed by an additional 4% gain at 10 attempts.Gains are especially pronounced for challenging Codeforces and Docs tasks, while Prefill shows more modest improvement.
  • Diversity: KODCODE spans the full t-SNE space, whereas baseline datasets cluster mainly in the upper-left region.The Algorithm and Filter subsets provide comprehensive coverage across the visualization space.
  • Difficulty: Prefill is typically easiest, while Codeforces, Taco, and Code Contests contain higher failure rates and larger hard-question proportions.Difficulty levels are defined from self-verification success across 10 attempts: easy, medium, hard, and fail.
  • Contamination: Only 94 of 447K questions are potentially contaminated, with most overlaps involving Prefill and simple HumanEval/MBPP questions.Prefill has the highest average maximum cosine similarity, while Docs has the lowest; contaminated cases are excluded from performance evaluation.
  • Data Flow: Over 25% of instances are removed during Step 1 deduplication, with Prefill exceeding 50% discarded and harder subsets showing higher Step 2 rejection rates.Figure 7 tracks instances from initial subsets through deduplication and final reject-sampled sets.

4 Performance Evaluation

KODCODE is evaluated through supervised fine-tuning and reinforcement learning across multiple code-generation benchmarks, with ablations examining data difficulty and question-format diversity. Fine-tuned models outperform strong baselines, while hard and style-diverse examples improve selected results.

  • Experimental Setup: KODCODE evaluation uses SFT and GRPO-based RL across HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench.Baselines include non-reasoning and reasoning models, and evaluation follows benchmark-specific setups.
  • SFT Results: 59.8% (+1.9%) on BigCodeBench-C Full and 37.8% (+4.7%) on BigCodeBench-C Hard exceed the strongest baselines.For BigCodeBench-I, the model reaches 51.1% (+1.7%) on Full and 32.4% (+5.4%) on Hard.
  • SFT Results: 92.7% on HumanEval and 90.7% on LiveCodeBench-Easy contribute to the highest overall average score of 61.26%.The HumanEval result exceeds Qwen2.5-Coder-32B-Instruct’s 90.9% by 1.8%.
  • Data Selection Ablations: 31.8% versus 27.7% (+4.1%) on BigCodeBench-I Hard shows that hard-sample training outperforms random-sample training.Hard-sample training also improves BigCodeBench-I Full, BigCodeBench-C Hard, and LiveCodeBench Hard.
  • Data Selection Ablations: Removing style conversion lowers BigCodeBench-C Full from 61.1% to 60.3% and Hard from 38.5% to 35.1%.LiveCodeBench Easy also decreases from 87.8% to 86.4% without style conversion.
  • RL Results: GRPO with KODCODE-RL-10K produces significant gains on most benchmarks, with further improvement from additional training steps.The RL reward is 1 when a solution passes all unit tests and 0 otherwise.

5 Related Work

Related work covers synthetic code-data generation, execution feedback, and unit-test generation as approaches to improving coding-LLM training and evaluation. KODCODE’s evaluation also uses GRPO with unit-test-based rewards.

  • Synthetic Data Generation: Synthetic-data methods expand small sets of human-annotated coding instructions using LLM prompting.This line of work responds to the time and resource costs of human data collection.
  • Code Generation with Execution Feedback: Execution-feedback systems iteratively refine generated code using runtime results to address syntactic and logical errors.Prior work applies execution and refinement in Docker environments.
  • KODCODE Evaluation: KODCODE-RL-10K uses GRPO to train models with rewards based on whether generated solutions pass unit tests.The accompanying evaluation reports notable gains across most benchmarks.
  • LLM-based Unit Test Generation: Unit-test generation complements execution feedback by proactively assessing correctness, with EvalPlus increasing test diversity and rigor.The related work includes empirical studies of LLM-generated tests and mutation-based test expansion.

6 Conclusion and Future Work

KODCODE provides a 447K-question synthetic dataset with verified solutions and unit tests through a three-step synthesis pipeline. Experiments report state-of-the-art results across multiple coding benchmarks, including performance above larger models.

  • Conclusion: KODCODE contains 447K diverse coding questions paired with verified solutions and unit tests.The dataset is intended to provide both diversity and quality for coding-language-model training.
  • Conclusion: Its three-step pipeline generates coding questions, refines solutions and tests through self-verification, and synthesizes post-training data.The pipeline is presented as addressing both dataset diversity and training-data quality.
  • Conclusion: Fine-tuned models achieve state-of-the-art performance across HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench.The reported results also place them above larger models.
  • Future Work: Future work targets more challenging problems, improved post-training data selection, and repository-level synthetic data.The authors also identify highly challenging competition-level problems as an area for further synthesis.

Limitations

KODCODE models perform at state-of-the-art levels on most coding benchmarks, but LiveCodeBench-Hard remains a limitation. The authors attribute this gap likely to insufficient coverage of highly challenging competition-level problems.

  • Scope Boundary: KODCODE-tuned models have limited performance on LiveCodeBench-Hard despite state-of-the-art results across most coding benchmarks.The limitation concerns a particularly challenging competition-oriented evaluation setting.
  • Scope Boundary: The authors suggest insufficient representation of highly challenging competition-level problems as a likely source of the LiveCodeBench-Hard gap.Future work therefore considers synthesizing more highly challenging coding problems.

Ethical Statement

The authors state that KODCODE does not introduce or endorse harmful or misusable applications and identify no ethical concerns in the paper.

  • KODCODE does not introduce or endorse applications that could cause harm or be misused.
  • The paper states that it presents no ethical concerns.

A Additional Information of the Filter Subset.

The Filter subset is built from model-generated data filtered for Python content, then labeled for quality and difficulty; Figure 8 shows task categories before filtering.

  • Seven state-of-the-art models generate the Filter subset, which is restricted to Python-specific instructions and responses, yielding 186K instances.
  • Llama-3.1-8B-Instruct labels Filter instances for quality and difficulty before retaining high-quality data.
  • Figure 8 depicts the task-category distribution of collected Magpie Coding data before filtering.

B.1 Analysis on MBPP Validation Failures Cases with Self Verification

The MBPP validation failures illustrate how generated solutions can mishandle factor accumulation or assume a narrower input format than the ground-truth specification.

  • Task 511 fails because GPT-4o does not correctly accumulate all prime factors when multiple factors exist beyond a single pair.For example, 105 has factors 3, 5, and 7, whereas the ground-truth implementation sums valid divisors iteratively.
  • Task 525 concerns determining whether two coefficient tuples represent parallel lines.The ground-truth solution supports both two-element (a, b) and three-element (a, b, c) tuples.
  • GPT-4o’s determinant-based parallelism check fails when inputs use two-element tuples because it assumes three-element representations.The mismatch occurs for tuples (3, 3) and (5, 5), which implicitly use c = 0.
  • KODCODE includes examples similar to MBPP and HumanEval instances, shown in Figures 11 and 12.

C More on Experimental Setups

The appendix documents fine-tuning settings, prompt templates, subset-construction procedures, and representative KODCODE examples, including two MBPP validation failures.

  • Experimental Setup: SFT experiments use Llama Factory, train on 32 GPUs, and run for 2 epochs on datasets larger than 20K samples or 3 epochs on smaller datasets.
  • Experimental Setup: The fine-tuning system prompt is adapted to accommodate R1-style long chain-of-thought responses.
  • Prompt Templates: Appendix prompts cover question generation, quality and category labeling, solution and unit-test generation, and question-style conversion.
  • Examples: The appendix includes matrix multiplication, weather-dashboard, and Gridland sapling question examples alongside failed-task illustrations.
  • Subset Construction: The Prefill subset uses Qwen2.5-Coder-7B-Instruct with three alternative prompt prefillings to generate simple Python function questions.
Loading 2503.02951v2…