PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Better Correctness and Efficiency in Code Generation

Yunlong Feng, Yang Xu, Xiao Xu, Binyuan Hui, Junyang Lin

arXiv:2508.20124v1cs.SEcs.AI

TL;DR

Generated code can emphasize correctness while running 3 to 13 times slower than human-written code, limiting its practical use. The paper proposes a two-stage tuning strategy that establishes correctness before applying error-insensitive online reinforcement learning for efficiency; on a 7B model, it reports improvements in both correctness and runtime efficiency.

  • Problem

    Code-generation models often neglect runtime efficiency while focusing on correctness, and generated code can be 3 to 13 times slower than human-written code.

    Source: §1 · p. 1

  • Method

    The method first fine-tunes a base model for correctness, then applies error-insensitive online reinforcement learning with high-contrast efficiency signals.

    Source: §1 · p. 1

  • Results

    10.18% improvement in code correctness and 7.75% improvement in runtime efficiency are reported on a 7B model, with performance comparable to a much larger model.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Offline methods establish a correctness baseline, while error-insensitive online methods dynamically improve efficiency while maintaining that achieved correctness.

    Source: §6 · p. 7

    The model exploited evaluation loopholes, including repeated-run caching and hard-coded or memoized solutions for limited test cases, distorting the reward signal.

    Source: §4.4 · p. 6

Abstract

from arXiv · show

While code large language models have demonstrated remarkable progress in code generation, the generated code often exhibits poor runtime efficiency, limiting its practical application in performance-sensitive scenarios. To address this limitation, we propose an efficiency-oriented reinforcement learning framework guided by a novel performance reward. Based on this framework, we take a deeper dive into the code efficiency problem, identifying then proposing methods to overcome key bottlenecks: (1) Dynamic exploration overcomes the static data constraints of offline fine-tuning, enabling the discovery of more efficient code implementations. (2) The error-insensitive reinforcement learning method and high-contrast efficiency signals are crucial for mitigating systematic errors and achieving effective optimization. (3) Online exploration is most effective when starting from a high-correctness baseline, as this allows for efficiency improvements without sacrificing accuracy. With these discoveries, we finally propose a two-stage tuning method, which achieves high and balanced performance across correctness and efficiency. The results of experiments show the effectiveness of the method, which improves code correctness by 10.18\% and runtime efficiency by 7.75\% on a 7B model, achieving performance comparable to much larger model.

1 Introduction

Code-generation models often prioritize correctness over runtime efficiency, with generated code reported as 3 to 13 times slower than human-written code. The paper explores online reinforcement learning and proposes a two-stage tuning strategy that first establishes correctness and then optimizes efficiency.

1 Introduction

  • Offline fine-tuning is limited by static training data, while online optimization requires error-insensitive learning and high-contrast efficiency signals to mitigate noisy, error-prone rewards.A high-correctness starting model is identified as a better foundation for improving efficiency without sacrificing accuracy.Source: §1 · p. 1
  • The proposed two-stage strategy fine-tunes a base model for correctness before using online reinforcement learning to improve runtime efficiency while maintaining that correctness.The first stage can use ordinary offline fine-tuning; the second uses error-insensitive reinforcement learning and high-contrast efficiency signals.Source: §1 · p. 1
  • 10.18% higher code correctness and 7.75% higher runtime efficiency are reported for the two-stage method on a 7B model, with performance comparable to a much larger model.The strategy first builds a correctness foundation, then applies error-insensitive online reinforcement learning with high-contrast efficiency signals.Source: §1 · p. 2

2 Efficiency-oriented Reinforcement Learning

The framework assesses code efficiency through group comparisons and CPU instruction counts, then assigns rewards that favor faster correct code while penalizing errors. Its design addresses the difficulty of choosing an optimal reference implementation and stabilizing noisy efficiency signals.

2 Efficiency-oriented Reinforcement Learning

  • A single manually optimized reference is inadequate because identifying a definitively optimal implementation is often intractable and manual reference creation is costly.The paper therefore adopts group comparisons for performance assessment.Source: §2 · p. 2
  • CPU instruction counts replace wall-clock time to reduce nondeterminism, while high-complexity-differentiation inputs expose algorithmic efficiency differences.Logarithmic transformation compresses skewed counts and stabilizes their variance for training.Source: §2 · p. 2

2.1 Reward Function

  • The reward benchmarks a correct candidate against other correct codes for the same query, using C = -log(N) so higher C means faster execution.G contains the performance metrics of correct generated codes for that query; its maximum and minimum mark the fastest and slowest codes.Source: §2.1 · p. 2
  • Passing code receives a performance score from 1.0 for the slowest to up to 2.0 for the fastest, while failures receive penalties based on error type.Test errors score 0.0, missing-entity errors -0.5, and format errors -1.5.Source: §2.1 · p. 2

3 Identify the Bottlenecks in Code Efficiency

The preliminary experiments compare offline and online fine-tuning methods on Qwen-2.5-Coder-Instruct-7B using two benchmarks. The section frames these comparisons around balancing correctness and efficiency and describes the datasets and efficiency metrics used.

3.1 Experiments

  • The experiments compare offline methods that learn from static precollected data with online methods that generate data during training.The listed methods include SFT and DPO offline, and GRPO and RLOO online.Source: §3.1 · p. 2
  • The training subset pairs simpler Effi-Learner queries with Evalplus-like difficulty and EffiBench queries with difficulty more like Mercury.EffiBench queries, like Mercury’s, originate from LeetCode.Source: Table 1 · p. 3
  • DPSnorm and Beyond measure code efficiency, with higher values indicating better efficiency.The section identifies these as the efficiency metrics for evaluating methods.Source: §3.1 · p. 3

3.2 Results of Preliminary Experiment

  • Different methods were evaluated on the Qwen-2.5-Coder-Instruct-7B model, and their results are presented in Table 2; the authors analyze performance to identify scene challenges and guide future research.Source: §3.2 · p. 3, Table 2 · p. 3

3.3 Offline Methods Bottlenecked by Initial Data Sampling

Offline fine-tuning is constrained by the quality of the base model’s initial sample pool, limiting its ability to learn correct and efficient solutions for difficult problems. Experiments show that selecting correct samples can improve correctness while worsening efficiency, and DPO also reaches an efficiency ceiling on challenging tasks.

3.3 Offline Methods Bottlenecked by Initial Data Sampling

  • Offline SFT and DPO are bounded by the initial sample pool, which may lack solutions that are both correct and efficient for difficult problems.Without high-quality examples, offline tuning cannot learn solutions absent from the data.Source: §3.3 · p. 3

Limited by Inefficient Samples from Challenging Problems

  • Fine-tuning on the fastest correct base-model samples boosted correctness but significantly reduced runtime efficiency on the challenging Mercury benchmark.The base model’s correctness was 62.99% on difficult EffiBench LeetCode problems versus 81.39% on simpler Effi-Learner problems; the authors suggest correct samples may be naive, inefficient implementations.Source: Limited by Inefficient Samples from… · p. 3

DPO Hits an Efficiency Ceiling

  • DPO uses negative samples more effectively than SFT, improving correctness without compromising efficiency, but both methods remain constrained by the base model on challenging problems.The authors report that this limitation hinders DPO’s ability to further improve efficiency on complex tasks.Source: DPO Hits an Efficiency Ceiling · p. 3

3.4 Data Composition Dictates the Trade-off Between Correctness and Efficiency

DPO training data composition shapes a trade-off between code correctness and runtime efficiency. Increasing the share of efficiency pairs helps efficiency up to a point, while using only efficiency pairs degrades both metrics.

3.4 Data Composition Dictates the Trade-off Between Correctness and Efficiency

  • As efficiency-pair proportions rise from 0.0 to 0.9, runtime efficiency steadily improves while code correctness declines.The authors describe the proportion as a way to steer the model’s focus.Source: §3.4 · pp. 3–4
  • At a 1.0 efficiency-pair proportion, both metrics deteriorate, with correctness falling below baseline.The paper characterizes this as pursuing efficiency at the complete expense of correctness.Source: §3.4 · p. 4
  • The authors conclude that efficiency can receive substantial emphasis at p ≤0.9 without substantially degrading correctness, but not when efficiency is pursued exclusively at p = 1.0.They conjecture that dynamic rewards in online reinforcement learning can manage this balance.Source: §3.4 · p. 4

3.5 Online RL is Constrained by Initialization, Reward, and Algorithm

The paper examines how synthetic inputs, model initialization, and RL algorithm choice affect online efficiency optimization. Its method first establishes correctness with DPO, then optimizes runtime efficiency using online RL, including error-insensitive algorithms such as RLOO.

Inputs with High Complexity Differentiation Gain Better Efficiency

  • Synthetic inputs with high complexity differentiation are used to make efficiency rewards better distinguish runtime performance, but this approach may compromise correctness when test coverage is insufficient.The EvalPerf results are cited as evidence of efficiency optimization conflicting with functional accuracy in that setting.Source: Inputs with High Complexity… · p. 4

Initialization from Models with Higher Diversity or Accuracy Gain Better Performance

  • The authors report that base models’ greater output diversity can be leveraged by online RL, while DPO initialization with high accuracy at p = 0.1 lets RL focus on efficiency without significantly compromising accuracy.The proposed two-phase approach first uses DPO to establish correctness, then optimizes runtime efficiency with online RL.Source: Initialization from Models with Higher… · p. 4, Figure 3 · p. 5

Error-Insensitive RL Algorithms can Maintain Accuracy better

  • RLOO generally achieves higher average accuracy, while GRPO tends to achieve higher code execution efficiency.The paper suggests GRPO’s regularization can magnify errors in efficiency rewards, potentially causing over-optimization and reduced correctness.Source: Error-Insensitive RL Algorithms can… · pp. 4–5

4 Towards Better Correctness and Efficiency

The method builds a high-correctness starting point before online efficiency optimization, reports results on two benchmarks, and illustrates both successful optimization and reward hacking.

4.1 How can we gain better performance?

  • RLOO is adopted for greater late-training stability than GRPO, while training starts from a higher-accuracy DPO checkpoint and uses discriminative inputs for efficiency rewards.The method sets the efficiency-pair proportion to 0.1 for this stage.Source: §4.1 · p. 5

4.2 Method

  • Stage 1 uses DPO data weighted 90% toward correctness pairs and 10% toward efficiency pairs; Stage 2 applies RLOO to improve efficiency while preserving prior correctness.The training process is described as yielding stable efficiency improvement while retaining the high correctness achieved in Stage 1.Source: §4.2 · p. 6, Figure 5 · p. 6

4.3 Results

  • 88.82 DPSnorm and 84.32 Pass@1 on EvalPerf, and 79.04 Beyond and 90.78 Pass@1 on Mercury, demonstrate gains on both benchmarks.The authors describe these results as validating an effective path toward better correctness and efficiency in code generation.Source: §4.3 · p. 6

4.4 Case Study

  • A successful case moves from a brute-force solution and its bottlenecks to a more efficient algorithm and complete implementation.Figure 4 illustrates this intended optimization pattern.Source: §4.4 · p. 6, Figure 4 · p. 6
  • Limited test cases and evaluation environments that retain state enable reward hacking, including hard-coded answers and LRU caching that inflate measured efficiency.Some hard-coded solutions pass existing tests but fail on new inputs or produce incorrect results.Source: §4.4 · pp. 6–7, Figure 6 · p. 7

5 Related Work

Related work addresses code efficiency through generation workflows, fine-tuning strategies, and benchmarks for evaluating execution performance.

Benchmarks

Workflow-based methods

  • Workflow methods include ECCO’s self-refinement, SOAP’s execution-feedback loop, and LLM4EFFI’s logic-then-code process (Ye et al. 2025).These approaches engineer the generation process through critique, execution feedback, or algorithm selection before implementation.Source: §5 · p. 7

Fine-tuning strategies

  • Data-centric methods fine-tune on efficient solutions or optimized code, while PIE (Shypula et al. 2024) reports a trade-off in functional correctness.Examples include Mercury’s SFT and DPO, Effi-Learner’s SOAP-generated code, and ACECode’s PPO with human-written upper-bound references.Source: §5 · p. 7

6 Conclusion

The authors conclude that offline methods establish correctness but are constrained by static data, while error-insensitive online methods can dynamically improve efficiency from that foundation.

Conclusion

  • Offline approaches are constrained by static initial data but can establish a strong correctness baseline.This finding informs the strategy of securing correctness before pursuing efficiency gains.Source: §6 · p. 7
  • Error-insensitive online methods dynamically improve runtime efficiency while maintaining the correctness level already achieved.Source: §6 · p. 7
  • The two-stage approach achieves significant improvements in both correctness and efficiency, supporting an effective path toward better code generation.Source: §6 · p. 7

Appendix

The appendix describes dataset preparation, benchmark metrics, and procedures for constructing training data and tuning models.

6.1 Data Prepare

  • Problems were deduplicated by function signature across Effi-Learner, EvalPerf, EffiBench, and Mercury test sets.Source: §6.1 · p. 10

6.2 Metrics

  • DPSnorm measures the percentage of reference solutions a solution beats, while Beyond normalizes runtime against the task’s solution distribution for cross-platform comparison.Source: §6.2 · p. 10

6.3 Training Setup

  • The authors sampled and tested 100 instances per problem with Qwen-2.5-Coder-Instruct-7B, grouping AST-consistent snippets by function AST before selecting responses.Source: §6.3 · p. 10
  • SFT uses 5,000 selected correct samples with the highest runtime efficiency, while DPO pairs correct with incorrect samples and compares correct samples using CPU instruction counts.The appendix identifies separate hyperparameter tables for SFT and DPO.Source: §6.3 · p. 10, Table 8 · p. 10, Table 9 · p. 10

RLOO/GRPO

The RLOO/GRPO section reports that adding a rounding mechanism improves GRPO’s correctness stability, bringing it closer to RLOO. It also describes the training setup and explains how rounding addresses errors amplified by GRPO’s reward normalization.

RLOO/GRPO

  • RLOO/GRPO training uses all samples from the Train Dataset, runs with Verl, and uses the hyperparameters listed in Table 10.Source: RLOO/GRPO · p. 10

6.4 GRPO with Round

  • GRPO’s division by reward standard deviation can amplify small errors; rounding is proposed to mitigate this effect.Source: §6.4 · p. 10
  • Figure 7 reports that rounding substantially improves GRPO’s correctness stability, making it similar to RLOO and advantageous over logarithmically scaled Instruction Count rewards.Source: Figure 7 · p. 10
Loading 2508.20124v1…