PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-Optimise: Self-Generated Preference Data for Correctness and Efficiency

Leonidas Gee, Milan Gritta, Gerasimos Lampouras, Ignacio Iacobacci

arXiv:2406.12502v2cs.CL

TL;DR

Code language models have generally prioritised correctness, while execution-optimisation work has reported correctness drops. Code-Optimise trains models with self-generated correctness and runtime preferences, and reports improved pass@k alongside lower runtimes than competitive baselines.

  • Problem

    Prior execution optimisation of code models has produced runtime gains but also correctness drops, while subsequent work has focused mostly on correctness.

    Source: §1 · p. 1

  • Method

    Code-Optimise uses self-generated passed/failed and quick/slow preferences to fine-tune CLMs, with dynamic solution selection during training.

    Source: §1 · p. 2

  • Results

    Functional correctness improves significantly, particularly for smaller models, while runtimes decrease by up to 6% for MBPP and 3% for HumanEval over baseline CLMs.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Generated code is shorter by up to 48% for MBPP and 23% for HumanEval, reducing inference cost and improving response time.

    Source: §5 · p. 9

    Runtime measurement of short programs remains challenging, and the experiments used Python, leaving other programming languages for follow-up work.

    Source: §6 · p. 9

Abstract

from arXiv · show

Code Language Models have been trained to generate accurate solutions, typically with no regard for runtime. On the other hand, previous works that explored execution optimisation have observed corresponding drops in functional correctness. To that end, we introduce Code-Optimise, a framework that incorporates both correctness (passed, failed) and runtime (quick, slow) as learning signals via self-generated preference data. Our framework is both lightweight and robust as it dynamically selects solutions to reduce overfitting while avoiding a reliance on larger models for learning signals. Code-Optimise achieves significant improvements in pass@k while decreasing the competitive baseline runtimes by an additional 6% for in-domain data and up to 3% for out-of-domain data. As a by-product, the average length of the generated solutions is reduced by up to 48% on MBPP and 23% on HumanEval, resulting in faster and cheaper inference. The generated data and codebase is open-sourced at https://github.com/huawei-noah/HEBO/tree/Code_Optimise.

1 Introduction

Code-Optimise addresses the trade-off between code correctness and runtime by training CLMs on self-generated preference data. It reports improved correctness and reduced runtimes over competitive baselines.

1 Introduction

  • Prior code-optimisation work reported runtime gains alongside correctness drops of up to about 30%, while subsequent research focused mostly on correctness.The authors position their work as the first to show improvements in both correctness and efficiency for code generation.Source: §1 · pp. 1–2
  • SFT optimises positive examples but cannot directly reduce the likelihood of undesirable code; the authors choose DPO as a simpler alternative to RL.They cite legal, financial, or privacy constraints as reasons proprietary data may not be feasible.Source: §1 · p. 2
  • Code-Optimise uses self-generated preferences for passed versus failed and quick versus slow solutions, with dynamic selection during training to reduce overfitting.The framework samples solutions, automatically annotates correctness and runtime, then fine-tunes the CLM.Source: §1 · p. 2
  • The framework contributes a code preference dataset and an extensible recipe for jointly optimising CLM correctness and runtime efficiency.The data is generated to provide learning signals without relying on larger models.Source: §1 · p. 2
  • Correctness improves significantly, particularly for smaller CLMs and lower pass@k, while runtimes fall by up to 6% on MBPP and 3% on HumanEval versus competitive baseline CLMs.The authors also report that scores improve further with Dynamic Solution Selection.Source: §1 · p. 2

2 Code-Optimise

Code-Optimise fine-tunes pretrained code language models using self-generated correctness and runtime preferences, with dynamic solution selection intended to reduce overfitting. Its DPO models show higher pass@k correctness and lower runtimes than SFT and competitive Base models.

2 Code-Optimise

  • Code-Optimise assumes pretrained models already contain accurate and efficient solutions, and seeks to guide selection rather than learn new information.The authors say exploring this approach alongside learning new information is beyond the work’s scope.Source: §2 · p. 2

2.1 Sampling

  • For each problem, the pretrained model generates 100 diverse solutions using multinomial sampling at temperature 0.6.The sampling expands training data beyond the limited supplied solutions.Source: §2.1 · p. 3

2.2 Annotation

  • Solutions are tested for correctness and timed repeatedly; runtimes are accepted at CoV ≤0.1, and problems lacking two passing and one failing solution are removed.Each solution is run 50 times, with repeats up to 1K when timing is unstable; five timing runs are averaged.Source: §2.2 · p. 3

2.3 Optimisation

  • DPO fine-tunes on preference data, while SFT trains on the fastest 25% or all solutions, and DPO pairs can encode runtime, correctness, or both.The configurations are DPOQvS, DPOPvF, and DPOAll; DPO is chosen for simplicity and wide adoption.Source: §2.3 · p. 3

2.4 Dynamic Solution Selection

  • DPO models consistently achieve higher functional correctness than Base and SFT across all k on MBPP and HumanEval.Figure 2 averages scores across model sizes.Source: Figure 2 · p. 4
  • DPO models reduce runtime versus SFT and competitive Base models, while code length falls by 10%–20% across both datasets.Figure 3 reports median runtime and code length as percentage changes relative to Base.Source: Figure 3 · p. 4
  • The framework dynamically selects a new preference pair per problem each DPO epoch, or a working solution each SFT epoch, to reduce overfitting.Sampling multiple completions lets prompts appear with different solutions during training.Source: §2.4 · p. 4

3 Results

The results evaluate functional correctness, runtime, and code length across MBPP and HumanEval. DPO configurations generally improve correctness and efficiency, with effects varying by dataset and optimization objective.

3.1 Evaluation Metrics

  • Runtime is measured as the median for working solutions, while code length is the median character count for working solutions.Runtime comparisons use the same subset of solved problems across models; code length is not an explicit fine-tuning objective.Source: §3.1 · pp. 4–5

3.2 Functional Correctness

  • DPO models show higher functional correctness than Base and SFT on both datasets, with larger gains on in-domain data, especially at lower k.SFT improves marginally at k = 1 but shows no improvement or a small decrease at higher k.Source: §3.2 · p. 5

3.3 Code Efficiency

  • DPOQvS and DPOAll reduce runtimes by up to 6% on in-domain data and 3% on out-of-domain data, while the best DPO models shorten median code length by up to 48% on MBPP and 23% on HumanEval.SFT generally increases runtimes and tends to generate longer solutions.Source: §3.3 · pp. 5–6, Figure 5 · p. 6, Figure 6 · p. 6

3.4 Model Scaling

  • Larger DPO models improve correctness more than Base and SFT, particularly on in-domain problems; code-length reductions persist across datasets and reach 48% in-domain and 23% out-of-domain.DPOPvF tends toward slower runtimes because it optimizes correctness alone; HumanEval runtime patterns vary more with model size.Source: §3.4 · pp. 6–7

3.5 Qualitative Analysis

  • Efficiency gains generally come from brevity and simpler, faster routines; DPO solutions also tend to have lower complexity and better unit-test coverage.The qualitative analysis finds no clear winner among DPO models, although DPOQvS leads in aggregate results.Source: §3.5 · p. 7

3.6 Importance of Solution Selection

  • Dynamic Solution Selection improves pass@1 for every StarCoder-1B model and benefits DPO more strongly than SFT; gains also hold across pass@k values.It selects unique solution pairs for each prompt at the start of each epoch, rather than fixing pairs at the start of training.Source: §3.6 · p. 7, Figure 7 · p. 7

3.7 Which configuration to choose?

  • DPOPvF is preferred for functional correctness, DPOQvS for runtime efficiency, and DPOAll improves both without being optimal on either objective.The authors recommend choosing a configuration to match the practitioner’s objective.Source: §3.7 · p. 8

4 Related Work

Prior work uses distilled supervision to improve correctness and code editing to reduce runtime, while reinforcement learning can propagate negative rewards. These approaches have limits involving proprietary data, complexity, instability, or correctness trade-offs.

Distilled supervision

  • Distilled supervised fine-tuning has been used to improve correctness, but approaches such as MagiCoder and WizardCoder rely on large proprietary models for training data.Such reliance may be impermissible in settings with financial, legal, or privacy constraints.Source: §4 · p. 8

Reinforcement learning and DPO

  • Reinforcement learning can propagate negative rewards for dysfunctional code, but typically adds complexity and instability; the cited approaches consider correctness, not runtime.Direct Preference Optimisation is presented as offline RL that optimizes trajectories without a separate reward model.Source: §4 · p. 8

Code-efficiency optimization

  • Code-efficiency editing research reports runtime gains alongside correctness reductions of up to 30%, with smaller CLMs suffering larger degradation in many configurations.The authors do not compare directly because that work assumes a correct but inefficient program as input, whereas this paper generates solutions from scratch.Source: §4 · p. 8

5 Conclusions

Code-Optimise uses self-generated preference data to improve code correctness and runtime. Experiments report higher functional correctness, reduced runtime, and shorter generated solutions.

5 Conclusions

  • The framework trains on self-generated preference data contrasting quick versus slow and passed versus failed solutions.Source: §5 · p. 9
  • Runtime falls by up to 6% for MBPP and 3% for HumanEval over the baseline CLM.The conclusion also reports significantly improved functional correctness, particularly for smaller models.Source: §5 · p. 9
  • Generated code length falls by up to 48% for MBPP and 23% for HumanEval, reducing inference cost and improving response time.Source: §5 · p. 9
  • Dynamic solution selection during training provides an additional benefit by reducing overfitting.Source: §5 · p. 9

6 Limitations

The authors identify runtime measurement quality, prompt sourcing, task scope, and programming-language coverage as limitations or areas for follow-up.

6 Limitations

  • Accurate timing of short programs is challenging, and additional software engineering could improve runtime measurements.Source: §6 · p. 9
  • Training data generation used only about 200 open-source prompts, while higher-quality problems free from proprietary or licensing issues may yield better results.Source: §6 · p. 9
  • Experiments used Python; other programming languages remain to be analysed in follow-up work.Source: §6 · p. 9
  • Investigating runtime and inference improvements for other code-related tasks amenable to the methodology was outside the paper's scope.Source: §6 · p. 9

A.1 Dataset

The evaluation uses MBPP for in-domain testing and HumanEval as an out-of-domain set to assess robustness to handwritten prompts.

A.1 Dataset

  • MBPP contains 974 crowdsourced Python programming challenges, with training and validation splits used for optimisation and the test split used for in-domain evaluation.Source: §A.1 · p. 11
  • HumanEval contains 164 Python programming challenges with handwritten descriptions, example solutions, and unit tests.Source: §A.1 · p. 11
  • HumanEval serves as the out-of-domain test set to evaluate robustness to handwritten prompts.The passage contrasts its writing style with MBPP descriptions.Source: §A.1 · p. 11

A.2 Training

The appendix specifies model and training settings, then points to additional qualitative examples and solution-selection ablation scores.

A.2 Training

  • Training uses pretrained StarCoder models at 1B and 3B and CodeLlama models at 7B and 13B, fine-tuned for 30 epochs with best checkpoints selected by validation loss.The setup uses a 5e−7 learning rate, linear scheduler, 10% warm-up, and 2048-token maximum sequence length.Source: §A.2 · p. 11

B.1 Additional Qualitative Examples

  • Figures 10 and 11 provide additional qualitative examples from each configuration.Source: §B.1 · p. 11

B.2 Additional Ablation Scores

  • Figure 12 reports additional pass@10 and pass@100 scores for StarCoder-1B on MBPP and HumanEval, ablating solution selection.Source: §B.2 · p. 11

B.3 Fastest Solution Analysis

Best@k compares the fastest of k sampled solutions, and DPO models yield the best runtimes under this fastest-solution view. The analysis cautions that this metric can overstate runtime improvements.

B.3 Fastest Solution Analysis

  • Best@k considers only the fastest solution among k samples; the analysis uses k = 100, covering all generated solutions.Source: §B.3 · p. 11
  • DPO models produce fastest solutions 2%–5% faster on in-domain and out-of-domain problems, and show the best runtimes on MBPP and HumanEval.The Figure 9 comparison averages best@100 scores across model sizes and considers each problem’s fastest solution.Source: §B.3 · p. 11, Figure 9 · p. 11
  • Best@k may overestimate runtime gains, so experiments also evaluate the median of all working solutions as a less biased measure.Source: §B.3 · p. 11

C Complete Results

The complete results section reports sampling-stage functional correctness and optimization performance, including a dynamic-selection comparison and example solutions.

C.1 Sampling

C.2 Optimisation

  • Tables 4–7 report full test-set performance on MBPP and HumanEval for StarCoder-1B, StarCoder-3B, CodeLlama-7B, and CodeLlama-13B, with CoV beside each runtime.Source: §C.2 · p. 11, Table 4 · p. 15, Table 5 · p. 15
  • Figures 10 and 11 show example solutions for a HumanEval problem and an MBPP problem, respectively.Source: Figure 10 · p. 12, Figure 11 · p. 13
  • For StarCoder-1B, dynamic solution selection improves both pass@10 and pass@100 scores and distributions compared with static selection.Source: Figure 12 · p. 14
Loading 2406.12502v2…