PaperlayerView exact revision on arXiv

Source-linked AI summary

Focused-DPO: Enhancing Code Generation Through Focused Preference Optimization on Error-Prone Points

Kechi Zhang, Ge Li, Jia Li, Yihong Dong, Jia Li, Zhi Jin

arXiv:2502.11475v2cs.SE

TL;DR

Code generation remains challenging because complex, long-reasoning tasks can fail through small errors concentrated at particular points. Focused-DPO directs preference optimization to those points and uses Error-Point Identification to construct data without costly human annotations. The paper reports improved results across code-generation benchmarks, including a 42.86% relative improvement on extremely hard LiveCodeBench competition problems.

  • Problem

    Code generation requires many small decisions to work together, yet errors concentrate at particular points and can sharply affect program accuracy.

    Source: §1 · pp. 1–2

  • Method

    Focused-DPO focuses preference optimization on error-prone points, using Error-Point Identification to construct datasets without costly human annotations.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    42.86% relative improvement on extremely hard competition-level LiveCodeBench problems, with significant improvements over existing methods on HumanEval(+), MBPP(+), and LiveCodeBench.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper concludes that fine-grained preference optimization on error-prone points can improve code quality even in heavily post-trained models.

    Source: §7 · p. 9

    The dataset pipeline uses PageRank and common-prefix/suffix matching to identify error-prone points; preliminary experiments suggest these choices support performance.

    Source: Limitation · p. 10

Abstract

from arXiv · show

Code generation models have shown significant potential for automating programming tasks. However, the challenge of generating accurate and reliable code persists due to the highly complex and long-reasoning nature of the task. Even state-of-the-art models often fail in code generation due to small errors, which can drastically affect the overall functionality of code. Our study identifies that current models tend to produce errors concentrated at specific error-prone points, which significantly impacts the accuracy of the generated code. To address this issue, we introduce Focused-DPO, a framework that enhances code generation by directing preference optimization towards these critical error-prone areas. This approach builds on Direct Preference Optimization, emphasizing accuracy in parts prone to errors. Additionally, we develop a method called Error-Point Identification, which constructs a dataset that targets these problematic points without requiring costly human annotations. Our experiments on benchmarks such as HumanEval(+), MBPP(+), and LiveCodeBench demonstrate that Focused-DPO significantly improves the precision and reliability of code generation, reducing common errors and enhancing overall code quality. By focusing on error-prone points, Focused-DPO advances the accuracy and functionality of model-generated code.

1 Introduction

Code generation remains difficult because many small decisions in a long reasoning process determine whether a program works, and errors cluster at particular code points. Focused-DPO targets those points during preference optimization and uses automatically identified error-prone points to build preference data without costly human input.

1 Introduction

  • Errors cluster at particular points in generated code, where small differences can affect final accuracy and program reliability.Middle code requiring more complex reasoning is more error-prone than familiar patterns such as function headers or return statements.Source: §1 · pp. 1–2
  • 90.02% versus 3.17% final accuracy follows from continuing generation at correct versus incorrect error-prone points, while non-highlighted parts show no comparable disparity.Generated outputs can share prefixes and suffixes while differing at the highlighted points.Source: Figure 1 · p. 1, §1 · p. 2
  • Focused-DPO directs preference optimization toward error-prone points rather than treating all code parts equally.The framework builds on Direct Preference Optimization and prioritizes areas where errors are most likely.Source: §1 · p. 2
  • Error-Point Identification constructs preference data from repository concepts, generated code and tests, PageRank-inspired ranking, and common-prefix/suffix matching, without costly human input.The matching step locates error-prone points in generated code.Source: §1 · p. 2
  • 42.86% relative improvement on extremely hard competition-level LiveCodeBench problems, with significant gains over existing methods across HumanEval(+), MBPP(+), and LiveCodeBench.The reported relative improvement is for Qwen2.5-Coder.Source: §1 · p. 2

2 Related Work

Related work describes preference optimization as aligning outputs with desired outcomes, but notes that existing methods do not directly target the most error-prone code points. Prior fine-grained methods offer step- or token-level signals, yet the paper argues their datasets or reward signals are inadequate for code generation.

2 Related Work

  • Existing DPO approaches align outputs with preferences but do not directly target error-prone code points that can affect final code quality and reliability.The paper identifies this as a gap in preference optimization for code generation.Source: §2 · pp. 2–3
  • Step-DPO (Lai et al., 2024) and Step-Controlled DPO (Lu et al., 2024) use step-wise preferences, while TDPO (Zeng et al., 2024) adds token-level forward-KL constraints.cDPO (Lin et al., 2024) identifies critical tokens in natural-language thought chains.Source: §2 · p. 3
  • The paper argues that fine-grained methods often struggle to construct adequate code datasets or provide strong enough fine-grained reward signals for code generation.It proposes self-generation and validation to identify code points for optimization.Source: §2 · p. 3

3 Focused-DPO

Focused-DPO builds a preference dataset around error-prone code spans, then modifies preference optimization to emphasize those spans. Its pipeline generates repository-derived prompts, verifies generated code with tests, and identifies contrasting code regions for focused learning.

3 Focused-DPO

  • Focused-DPO modifies DPO to emphasize high-impact error-prone code parts using a dataset that identifies those points.The framework combines synthetic prompt generation, fine-grained verification, and focused preference optimization.Source: §3 · p. 3, Figure 2 · p. 3

3.1 Synthetic Data Generation with Real-World Source Code

  • Programming prompts are generated by extracting concepts from diverse open-source code snippets and using those concepts to create tasks.The authors say this strategy allows exploration of a broad range of scenarios.Source: §3.1 · p. 4

3.2 Fine-Grained Verification to Identify Error-Prone Points

  • The verification pipeline generates code and tests, ranks outputs through test-based execution relationships, and uses the highest-ranked code’s passing tests as ground truth.Generated code is divided into correct and incorrect samples before common prefixes and suffixes are matched.Source: §3.2 · p. 4
  • Diff combines the rank difference between correct and incorrect code with shared prefix and suffix lengths to select error-prone middle spans.The resulting filtered dataset contains 5,000 training samples and 1,000 validation samples per policy model.Source: §3.2 · p. 4

3.3 Focused Preference Optimization Learning

  • Focused-DPO weights the middle span more heavily in preference rewards and downweights the rejected sample’s suffix contribution.The suffix adjustment reflects the authors’ observations that suffixes are often similar across samples and correlate weakly with overall accuracy.Source: §3.3 · p. 5

4 Experiment Setup

The experiments ask whether error-prone code regions affect correctness, whether Focused-DPO improves code generation, and how its components affect performance. They compare it with established training methods across code models and benchmarks under specified training and inference settings.

4 Experiment Setup

  • The experiments test whether error-prone points affect correctness, whether Focused-DPO improves generation quality, and how its loss components affect performance.The planned evaluation spans HumanEval, HumanEval+, MBPP, MBPP+, and LiveCodeBench.Source: §4 · p. 5

4.1 Baselines

  • Focused-DPO is compared with SFT, standard DPO, Step-DPO, and TDPO across base and instruction-tuned code models.SFT uses positive samples, while the other compared methods use preferred and rejected pairs.Source: §4.1 · pp. 5–6

4.2 Training and Inference Settings

  • Training uses 10 code-and-test candidates per prompt at temperature 1.5, 10 epochs on 8 V100 GPUs, and greedy-search inference.The best checkpoint is selected by lowest validation loss; experiments set wfocused = 2.Source: §4.2 · p. 6

5 Results and Analyses

Focused-DPO improves benchmark performance by emphasizing code regions where errors strongly affect correctness. The analyses support this focus, while ablations favor the proposed data construction and reward choices.

5.1 Exploration of Error-Prone Points in Code (RQ1)

  • Mid-code parts correlate more strongly with final correctness than prefixes or suffixes, which have little relation to correctness.The table describes frequencies in correct and incorrect code and correlations with overall correctness; the findings also report that suffixes in incorrect code are not a significant cause of errors.Source: §5.1 · p. 6, Table 1 · p. 6
  • Using chosen_mid at error-prone points yields around 90% pass@1, compared with just over 3% for rej_mid.The reported pass rates vary with the content supplied at the error-prone points.Source: §5.1 · p. 6, Table 2 · p. 6
  • A heavily post-trained Qwen2.5-Coder model shows little to no clear preference between chosen_mid and rej_mid when given common_prefix.The analysis measures the generation-probability difference and reports the distribution in Figure 3.Source: §5.1 · p. 7, Figure 3 · p. 7

5.2 Main Results (RQ2)

  • Focused-DPO improves relative accuracy by 4.79% on average over baselines on HumanEval(+) and MBPP(+), and consistently improves across LiveCodeBench difficulty levels.The reported comparisons cover Table 3 and Table 4; Focused-DPO also outperforms Step-DPO and TDPO.Source: Table 3 · p. 7, §5.2 · p. 7, Table 4 · p. 8
  • Applying Focused-DPO after SFT and an initial DPO stage yields the highest pass rates across the reported training stages.The comparison uses DeepSeekCoder-6.7B at the base, SFT, first-DPO, and Focused-DPO stages.Source: §5.2 · pp. 7–8, Table 5 · p. 8

5.3 Ablation Study (RQ3)

  • The proposed Error-Prone Identification dataset construction achieves the best experimental results among the tested approaches.The alternatives use Step-DPO construction or git-diff-defined differences as error-prone points.Source: §5.3 · p. 8, Table 6 · p. 8
  • Changing wfocused from its current value or including the suffix in the reward function degrades model performance.The authors relate the suffix result to its weak correlation with overall accuracy in incorrect code.Source: §5.3 · p. 8

6 Discussions

Error-Point Identification uses automated quality filtering and identifies some shared error-prone points across models. After Focused-DPO, the model favors more accurate continuations at these points, with reported gains on a hard benchmark.

6.1 Error-Prone Points Identification

  • Error-Point Identification uses policy-model generations and PageRank-based self-generation-and-validation to filter lower-quality test cases without costly human annotations.The authors distinguish this from approaches that use all test cases as ground truth and note that their policy model is less robust than GPT-4.Source: §6.1 · p. 9
  • DeepSeekCoder-instruct-6.7B and Qwen2.5-Coder-instruct-7B share a 32% overlap in identified error-prone points on the same problems.The construction process also filters problems with excessively high or low model accuracy to keep a consistent number of problems in each training dataset.Source: §6.1 · p. 9
  • The method uses prefix-and-suffix-based identification for its effectiveness and efficiency, while tested git-diff and Step-DPO constructions produce slightly different results.The authors plan to explore additional identification strategies.Source: §6.1 · p. 9

6.2 Improvement in Error-Prone Points

  • On LiveCodeBench-Hard, Focused-DPO improves correctness by 42.8% for Qwen2.5-Coder-Instruct and reaches performance on par with GPT-4o.After Focused-DPO, the model also shows a stronger preference for generating more accurate code at error-prone points than before training.Source: §6.2 · p. 9, Figure 4 · p. 9

7 Conclusion

Focused-DPO targets error-prone code points through preference optimization and uses Error-Point Identification to construct training datasets without costly human annotations. The paper reports improved code quality, including for heavily post-trained models.

7 Conclusion

  • Focused-DPO directs preference optimization toward error-prone points, which the paper identifies as critical to overall program correctness.Its Error-Point Identification method constructs targeted datasets without costly human annotations; evaluations report improved quality even for heavily post-trained models.Source: §7 · p. 9

Limitation

The authors identify limits in the scope of Focused-DPO’s comparisons and correctness-focused dataset construction, and report that enlarging the training dataset did not significantly improve results within the tested range.

Limitation

  • Focused-DPO is not extensively compared with advanced RL alignment techniques, leaving performance and efficiency comparisons with those methods for future investigation.The authors note that these online techniques typically require substantial resources, high-quality datasets, and complex reward environments.Source: Limitation · p. 10
  • Error-Prone Identification targets correctness in final code, while extending it to efficiency, readability, and security remains future work.The authors suggest static analysis, code-smell detection, and vulnerability identification as possible ways to detect and penalize insecure patterns.Source: Limitation · p. 10

A Data Scaling For Focused-DPO

  • Increasing the training dataset from 2.5k to 10k samples did not significantly improve Focused-DPO results within the tested data range.The authors report that fine-grained preference optimization converged efficiently across the examined data settings.Source: A Data Scaling For Focused-DPO · p. 12, Table 7 · p. 12

B Dataset Statistics

  • Table 8 is identified as reporting evaluation-benchmark statistics, while Table 9 reports training-dataset statistics for Qwen2.5-Coder-Instruct-7B.Source: Table 8 · p. 12, Table 9 · p. 12

C Scalability and Dataset Independence

The authors describe Focused-DPO as intended to scale with minimal dependence on external prompt resources and dataset-specific constraints. They report self-generated preference feedback and CodeContests evaluation as evidence for its broader applicability.

C Scalability and Dataset Independence

  • The authors envision using the policy model for question generation and solution alignment, aiming for self-alignment and self-improvement with minimal external prompt resources.They also describe avoiding constraints to specific datasets as an intended design goal.Source: C Scalability and Dataset Independence · p. 12
  • Focused-DPO’s preference feedback comes directly from the model itself, avoiding potential risks of transferring knowledge from a more capable model through externally distilled data.Source: C Scalability and Dataset Independence · p. 12
  • The authors report that CodeContests results demonstrate Focused-DPO is dataset-agnostic and scalable.The evaluation uses nearly 4,500 filtered CodeContests samples; the results are presented in Table 10.Source: C Scalability and Dataset Independence · p. 12, Table 10 · p. 12

D Evaluation of k’s Impact

Moderate k values provide the most consistent identification of error-prone regions and stable benchmark improvements. The authors also report that retaining part of the positive-sample learning component improves training robustness and stability.

D Evaluation of k’s Impact

E Suffix in Positive Loss

F Case Studies for Error-Prone Points

The case studies illustrate error-prone points in three code-generation tasks: converting to a valid variable name, parsing a version, and finding a maximum product.

F Case Studies for Error-Prone Points

Loading 2502.11475v2…