PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeDPO: Aligning Code Models with Self Generated and Verified Source Code

Kechi Zhang, Ge Li, Yihong Dong, Jingjing Xu, Jun Zhang, Jing Su, Yongfei Liu, Zhi Jin

arXiv:2410.05605v2cs.SE

TL;DR

Code-generation training still faces challenges in prioritizing correct code and optimizing execution efficiency. CodeDPO builds preference data through self-generated code and tests, mutual validation, and execution timing. It reports improvements over existing methods across five benchmarks, including an 83.5% HumanEval pass rate, while noting that comparison with advanced RL methods remains incomplete.

  • Problem

    SFT and existing code preference methods do not fully address preference for correct solutions and execution efficiency, which require objective code-generation measures.

    Source: §1 · p. 1

  • Method

    CodeDPO generates code and tests together, iteratively ranks them through mutual validation, and builds correctness- and efficiency-based preference data for DPO training.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    CodeDPO reports superiority over existing methods across five popular benchmarks and an 83.5% pass rate on HumanEval.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that self-validation can synthesize preference data without pre-existing tests or powerful generation models.

    Source: §6 · p. 9

    The study does not thoroughly compare CodeDPO with advanced RL alignment methods such as DeepSeek-R1.

    Source: Limitation · p. 9

Abstract

from arXiv · show

Code generation models have shown significant potential for programming tasks. However, existing training methods like supervised fine-tuning face key limitations: they do not effectively teach models to prioritize correct over incorrect solutions in ambiguous situations, nor do they effectively optimize the runtime efficiency of the generated code. To address these challenges, we propose CodeDPO, a framework that integrates preference learning into code generation to improve two key code preference factors: code correctness and efficiency. CodeDPO employs a novel dataset construction method, utilizing a self-generation-and-validation mechanism that simultaneously generates and evaluates code and test cases. The underlying assumption is that test cases executable by multiple code snippets provide more reliable validation, and code that passes more tests is more likely to be correct. Through this self-validation process, our PageRank-inspired algorithm iteratively updates the ranking score of each code snippet, ultimately creating a code preference optimization dataset based on correctness and efficiency. CodeDPO is flexible and scalable, generating diverse preference optimization data without depending on external resources. Through comprehensive evaluations of five widely used benchmarks, CodeDPO demonstrates significant improvements in correctness and efficiency compared to existing methods. Our experiments prove that CodeDPO enhances the capabilities of LLMs in code generation and provides a robust foundation for conducting code preference optimization in more complex and challenging real-world scenarios.

1 Introduction

Code generation still faces challenges in code correctness and efficiency, while SFT and existing preference methods do not fully address them. CodeDPO builds preference data through self-generation and validation, then uses it to optimize both factors.

1 Introduction

  • Code generation models still need improvements in both correctness and execution efficiency.SFT may increase the likelihood of preferred outputs while also increasing undesirable outputs, leading to performance saturation.Source: §1 · p. 1
  • CodeDPO generates and evaluates code and tests together, then iteratively updates their scores with a PageRank-inspired mutual-verification algorithm.The resulting preference dataset supports DPO training for correctness and efficiency.Source: §1 · p. 2
  • CodeDPO does not depend on high-quality test cases or powerful generation models, supporting diverse preference data when high-quality tests are sparse.The authors describe the framework as flexible and scalable.Source: §1 · p. 2
  • CodeDPO achieves an 83.5% pass rate on HumanEval and demonstrates superiority over existing methods across five popular benchmarks.The authors also report further gains after high-quality SFT.Source: §1 · p. 2

2 Related Work

Prior code-model training includes SFT and preference optimization, but these approaches leave gaps in preferring correct solutions and in addressing objective correctness and efficiency. CodeDPO is presented as an approach to create balanced preference pairs without external tests or powerful generation models.

2.1 Large Language Models for Code

  • SFT boosts code quality but does not fully train models to prefer correct solutions over incorrect ones.Self-instruct methods generate synthetic instruction data, while OSS-instruct draws inspiration from real-world code snippets.Source: §2.1 · pp. 2–3

2.2 Preference Optimization for Code Models

  • Code generation needs objective correctness and efficiency measures, while preference optimization has been less explored for code than for natural-language tasks.Prior work includes Code-Optimize and PLUM; the passage notes Code-Optimize uses the 384-problem MBPPtrain subset.Source: §2.2 · p. 3
  • The related-work discussion says PLUM uses limited tests, produces imbalanced data, uses KTO, and does not consider code efficiency.CodeDPO is introduced as not relying on external test cases or powerful models to generate its dataset.Source: §2.2 · p. 3

3 CodeDPO: Self-Verified Performance Optimization Code Generation Framework

CodeDPO constructs preference data from real-world code, self-generated code and tests, and execution times on credible tests, then trains code models with DPO. Its pipeline separately targets correctness and efficiency.

3 CodeDPO: Self-Verified Performance Optimization Code Generation Framework

  • The framework brings together real-world code seed construction, correctness optimization, efficiency optimization, and model optimization training.Code and tests are generated together for correctness scoring, while execution time on credible tests informs efficiency scoring.Source: §3 · p. 3

3.1 Data Seed Construction

  • Data seed construction extracts programming concepts from open-source repositories and uses them to generate task prompts.The seed is intended to let the model explore a wide range of scenarios.Source: §3.1 · pp. 3–4

3.2 Correctness Optimization with Self-Generation and Validation

  • The self-generation-and-validation loop generates candidate code and tests in parallel, executes the code against tests, and uses outcomes to update both sets of scores.A PageRank-inspired iterative algorithm ranks code snippets and test cases; its scores are updated over a fixed number of iterations.Source: §3.2 · p. 4

3.3 Execution Efficiency Optimization

  • Efficiency ranking measures total runtime on credible tests passed by top-performing code from the correctness stage, favoring snippets that pass them faster.Both fast and slow snippets are collected for efficiency optimization training.Source: §3.3 · pp. 4–5

3.4 Final Dataset and Model Optimization

  • CodeDPO combines correctness and efficiency preference datasets for DPO training, using 93k correctness samples and 21k efficiency samples.The final dataset pairs preferred and rejected solutions and filters samples with identical or near-identical ranking scores.Source: §3.4 · p. 5

4 Experiment Setup

The experiments assess correctness, execution efficiency, the self-generation-and-validation algorithm, preference optimization strategies, and data scaling. They use multiple base and SFT code models with specified training and inference settings.

4 Experiment Setup

  • The experiment plan tests correctness on six benchmarks, execution efficiency against baselines, and the effects of the validation algorithm, preference strategy, and training-data scale.The correctness question lists HumanEval, HumanEval+, MBPP, MBPP+, DS-1000, and LiveCodeBench.Source: §4 · p. 5

4.1 Backbone LLMs

  • Backbones include base Phi-2, DeepSeekCoder, and StarCoder2 models, as well as Magicoder SFT models based on CodeLlama and DeepSeekCoder.The study also evaluates base models because they can generate more diverse snippets and offer greater potential for improvement.Source: §4.1 · p. 5

4.2 Training and Inference Settings

  • Dataset construction samples 15 code solutions and tests per prompt from DeepSeekCoder-v2, while training runs for 10 epochs with a 5e-6 learning rate.The study focuses on a Python dataset, uses greedy decoding, and runs experiments on 16 A100 GPUs.Source: §4.2 · pp. 5–6

5 Results and Analyses

CodeDPO improves code correctness and runtime efficiency across the reported evaluations, while experiments examine its self-validation, training strategy, data scale, and benchmark overlap. The results also show where performance gains are observed and how dataset size and evaluation scope qualify them.

5.1 Code Correctness (RQ1)

  • 83.5% pass rate on HumanEval; CodeDPO also leads on HumanEval(+) and MBPP(+) across tested models, with a reported 36.1% relative improvement on StarCoder2-7B.The reported results include gains after high-quality SFT and on the stricter HumanEval+ benchmark.Source: §5.1 · p. 6, Table 1 · p. 7, Table 2 · p. 7
  • CodeDPO improves over baselines on DS-1000 and raises LiveCodeBench performance across difficulty levels, with particularly notable gains on medium and hard problems.The DS-1000 evaluation spans different libraries without incorporating prior knowledge of specific Python libraries; the authors note slight drops in Torch and TensorFlow settings.Source: §5.1 · p. 6, Table 4 · p. 7

5.2 Code Efficiency (RQ2)

  • 1.25–1.45× speed-up and around 20%–45% optimized solutions are reported on HumanEval+ and MBPP+ among problems solved before and after CodeDPO.A solution counts as optimized if it is at least 10% faster than its baseline; evaluations were repeated five times because runtime environments affect measurements.Source: §5.2 · p. 6, Figure 3 · p. 8

5.3.1 Self Generation and Validation Algorithm (RQ3)

  • Self-validation scores correlate highly with actual code accuracy, while mutual reinforcement updates code and test credibility to mitigate vulnerability to low-quality generated tests.The comparison evaluates ranking strategies using 15 sampled solutions and tests per HumanEval problem.Source: §5.3.1 · pp. 7–8, Table 5 · p. 8
  • The self-generation-and-validation algorithm is reported as essential to constructing a correct and reliable preference dataset.The experiments compare dataset construction strategies, including randomly selected preference pairs.Source: §5.3.1 · p. 8, Table 6 · p. 8

5.3.2 Impact of PO Training Strategy (RQ4)

  • DPO performs best among DPO, KTO, and SFT strategies; the authors attribute its benefit to balanced preference pairs that strengthen DPO’s contrastive mechanism.The SFT strategy uses the best code solution from the constructed dataset, while KTO replaces DPO in the framework.Source: §5.3.2 · p. 8

5.4 Data Scaling Law for CodeDPO (RQ5)

  • HumanEval scores rise from 32.92 at 25% data to 41.46 at 75%, with gains that gradually plateau as dataset size grows.The passage reports similar scaling trends on MBPP and says the experiments balance performance gains against training costs.Source: §5.4 · p. 9, Table 7 · p. 9

5.5 Overlap Avoidance with Existing Benchmarks

  • CodeDPO has comparable or lower overlap with HumanEval than most widely used datasets, supporting the authors’ assessment of evaluation validity and reliability.The study assesses overlap using average cosine similarity between HumanEval and synthetic data.Source: §5.5 · p. 9, Table 8 · p. 9

6 Conclusion

CodeDPO is a code-model preference optimization framework focused on correctness and efficiency. It uses iteratively updated self-validation scores to prioritize solutions and synthesize preference data without requiring pre-existing tests or powerful generation models.

6 Conclusion

  • CodeDPO is a preference optimization framework for code models that targets both correctness and efficiency.Its self-validation score is iteratively updated to prioritize solutions across those factors.Source: §6 · p. 9
  • The work validates self-validation as a way to synthesize preference data without complex resources such as pre-existing tests or powerful generation models.Source: §6 · p. 9
  • The authors propose the framework as a basis for further data synthesis and large-scale code-model preference optimization.Source: §6 · p. 9

Limitation

The stated limitations include a lack of thorough comparison with advanced online RL alignment and the scope of correctness-focused evaluation. Readability and security objectives remain future extensions.

Limitation

  • CodeDPO has not been comprehensively evaluated against advanced RL alignment methods such as DeepSeek-R1 on performance and efficiency.The paper contrasts its offline approach and lower resource demands with online RL, while identifying the comparison as future work.Source: Limitation · pp. 9–10
  • Current test-case-driven functional-correctness evaluation remains limited by reliance on high-quality tests or powerful models, motivating the paper’s self-validation approach.Source: Limitation · p. 10
  • Readability and security are outside the current framework’s objectives; the authors identify their integration as future work.Proposed directions include style and naming measures, static analysis, and vulnerability detection.Source: Limitation · p. 10

C Impact of PO Training Strategy (RQ4)

This appendix describes preference-training comparisons, CodeT’s ranking approach, the CodeDPO construction pipeline, and the prompts used to generate its data. It outlines the pipeline from source-code seeds through preference optimization training.

C Impact of PO Training Strategy (RQ4)

D Comparison with CodeT in details

  • CodeT ranks code using the product of passed-test count and the number of codes passing each test; the cited comparison uses 15 samples versus CodeT’s typical 100.The authors say their PageRank-based method offers finer-grained verification, particularly at smaller sample sizes, while noting resource constraints prevented using 100 samples.Source: D Comparison with CodeT in details · p. 13

E CodeDPO Dataset Construction algorithm description

  • The construction pipeline moves from real-world code seeds to self-validation-based correctness optimization, execution-time efficiency ranking, and DPO training.Source: E CodeDPO Dataset Construction… · p. 13

F LLM Prompts for CodeDPO dataset construction

  • The prompt workflow extracts concepts from Stack v1 code, generates task descriptions, then generates code solutions and assertion-only tests; construction takes around 40 hours on 32 CPUs.Source: F LLM Prompts for CodeDPO dataset… · p. 13

G Experiments on Challenging Code Efficiency Tasks

On EffiBench, CodeDPO reduces execution time and memory usage while keeping maximum memory usage comparable. Repeated measurements also show consistent runtime-efficiency improvements for Phi-2-2.7B.

G Experiments on Challenging Code Efficiency Tasks

  • The EffiBench analysis considers both absolute and normalized efficiency measures, while acknowledging that absolute values depend on the execution environment.The supplied passage does not provide specific numerical comparisons.Source: G Experiments on Challenging Code… · p. 15
  • CodeDPO significantly reduces execution time and memory usage, in absolute terms and after normalization, while maintaining comparable maximum memory usage.The reported efficiency analysis focuses on relative improvements because absolute results may vary by execution environment.Source: G Experiments on Challenging Code… · p. 15

H Execution Time for Code Efficiency Experiments

  • Repeated experiments in a stable environment yield averaged execution-time statistics for Phi-2-2.7B, with runtime conditions noted as a source of variation.The passage reports that these measurements demonstrate consistent execution-efficiency improvements and practical runtime benefits.Source: H Execution Time for Code Efficiency… · p. 15

I Ablation on Sample Number for code and test generation

The ablation varies the number of generated code solutions and test cases, examining score correlation and Phi-2-2.7B benchmark performance. The authors identify 15 as a favorable balance between diversity and computational feasibility.

I Ablation on Sample Number for code and test generation

  • sample_num=15 achieves a favorable trade-off between diversity and computational feasibility.Larger sample numbers provide marginal gains, but the passage does not specify the associated increased cost.Source: I Ablation on Sample Number for code… · p. 15
  • The study tests sample numbers of 5, 15, and 50, with results covering HumanEval score correlation and Phi-2-2.7B performance on HumanEval and HumanEval+.The table caption identifies the reported correlation and model-performance comparison across sample numbers.Source: I Ablation on Sample Number for code… · p. 15, Table 15 · p. 17
  • The sample-number and temperature choices are guided by practical considerations and empirical observations, with the setup aligned to Section 5.3.1.Similar trends are reported for other models.Source: I Ablation on Sample Number for code… · p. 15

J.1 Comparison of Dataset Statistics

CodeDPO's dataset has substantially greater unique-question diversity than PLUM and Code-Optimise despite similar sample sizes. The appendix also describes its self-validation score calculation and two mechanisms for mitigating weak-test effects.

J.1 Comparison of Dataset Statistics

  • CodeDPO provides significantly more diverse unique questions than PLUM and Code-Optimise despite similar dataset sizes.The comparison treats dataset size and unique-question count as important statistics; the table caption specifies these dimensions.Source: §J.1 · p. 17, Table 16 · p. 17

J.2.1 Python Implementation of the Self-Validation Scores

  • The Python example illustrates the iterative calculation of self-validation scores for the Figure 4 case, specifically its Step 2.Figure 4 is identified as a Python implementation of the self-validation scores in Figure 2.Source: §J.2.1 · p. 17, Figure 4 · p. 18

J.2.2 Handling Weak Test Cases

  • Weak tests that nearly all solutions pass contribute equally to their scores and minimally affect relative ranking; near-identical scores are filtered from the final dataset.The filtering addresses cases where repeated updates make code-solution scores identical or nearly identical.Source: §J.2.2 · pp. 17–18
Loading 2410.05605v2…