PaperlayerView exact revision on arXiv

Source-linked AI summary

RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learning

Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Quentin Carbonneaux, Taco Cohen, Gabriel Synnaeve

arXiv:2410.02089v2cs.CLcs.AI

TL;DR

LLM agents often fail to improve code iteratively from execution feedback, despite needing such feedback to achieve reliable multi-step outcomes. RLEF trains code-generation models with reinforcement learning over repeated attempts and execution feedback, achieving stronger competitive-programming results with far fewer samples while generalizing across benchmarks and inference settings.

  • Problem

    Code-generation LLMs have struggled to use execution feedback effectively, with independent sampling often outperforming iterative refinement at fixed inference budgets.

  • Method

    RLEF fine-tunes LLMs with reinforcement learning in an iterative code-synthesis environment where public-test feedback informs later generations and private tests determine final reward.

  • Results

    RLEF achieves substantial CodeContests solve-rate improvements, surpasses prior state-of-the-art results with 8B and 70B models, and reduces required generations by an order of magnitude.

  • Takeaways & Limitations

    RLEF-trained models can leverage automatic inference-time feedback over multiple steps and generalize improvements to HumanEval+ and MBPP+ and larger sample budgets.

  • Takeaways & Limitations

    The study improves a single solution to a problem; extending RLEF to larger tasks requiring decomposition remains future work, and unit-test iteration requires available test cases.

Abstract

from arXiv · show

Large language models (LLMs) deployed as agents solve user-specified tasks over multiple steps while keeping the required manual engagement to a minimum. Crucially, such LLMs need to ground their generations in any feedback obtained to reliably achieve the desired outcomes. We propose an end-to-end reinforcement learning method for teaching models to leverage execution feedback in the realm of code synthesis, where state-of-the-art LLMs struggle to improve code iteratively compared to independent sampling. We benchmark on competitive programming tasks, where we achieve new state-of-the art results with both small (8B parameters) and large (70B) models while reducing the amount of samples required by an order of magnitude. Our analysis of inference-time behavior demonstrates that our method produces LLMs that effectively leverage automatic feedback over multiple steps.

1 INTRODUCTION

LLM agents must both follow user intent and use intermediate execution feedback to ground later actions. RLEF addresses this challenge in code synthesis, where independent sampling has often outperformed iterative feedback use under fixed inference budgets.

  • 1 INTRODUCTION: LLM agents need instruction following and the ability to incorporate feedback from intermediate actions to reach desired outcomes.In code generation, feedback can expose implementation bugs and unstated platform or dependency constraints.
  • 1 INTRODUCTION: Execution feedback is crucial for grounding code-generation outputs in the concrete conditions encountered at inference time.
  • 1 INTRODUCTION: RLEF frames repeated code generation and execution feedback as a reinforcement-learning problem for improving feedback use in code synthesis.
  • 1 INTRODUCTION: Independent sampling has often achieved higher accuracy than iterative feedback use when computational inference budgets are fixed.
  • 1 INTRODUCTION: RLEF improves CodeContests performance, reduces required generations by an order of magnitude, and generalizes to HumanEval+ and MBPP+ and larger sample budgets.

2 METHOD

The method treats iterative code synthesis as a multi-turn interaction in which generated solutions receive public-test feedback before later attempts. It optimizes this process with PPO while using private tests for final reward evaluation and a hybrid token-level policy with turn-level value estimation.

  • 2 METHOD: The code-synthesis task is structured as a multi-turn conversation where each generated solution receives automatically produced execution feedback.
  • 2 METHOD: Public tests provide feedback after each attempt, while private tests determine final correctness and help prevent optimization shortcuts.
  • 2 METHOD: The environment is modeled as a partially observable MDP whose observations include prior dialogue and public-test feedback.Episodes end when public tests pass or a step limit is reached, followed by a scalar reward based on public and private tests.
  • 2 METHOD: PPO optimizes the language-model policy with a KL-regularized reward signal and a learned value-function baseline.
  • 2 METHOD: The implementation uses a token-level policy together with a value function estimated over whole turns.This hybrid choice performed best in the authors’ early experiments.

3 EXPERIMENTAL RESULTS

RLEF substantially improves competitive-programming code generation, especially under iterative inference, while enabling models to use execution feedback for targeted repair. Gains appear across sample budgets and benchmarks, with stronger performance than independent sampling and prior methods.

  • 3.2 MAIN RESULTS: 38.0 versus 29: the RLEF-trained 70B model beats AlphaCodium with GPT-4 on the test set using one rollout instead of 100 samples.
  • 3.2 MAIN RESULTS: RLEF improves the Llama 3.0 8B model’s 1@3 solve rates from 4.1 to 12.5 on validation and from 3.2 to 12.1 on test.
  • 3.3 INFERENCE-TIME BEHAVIOR: Unlike base models, RLEF-trained 8B and 70B models benefit from execution feedback in multi-turn generation and generalize gains to HumanEval+ and MBPP+.The 8B model is an exception on CodeContests and MBPP+, where single-turn performance drops, but the broader multi-turn benefits transfer across benchmarks.
  • 3.3 INFERENCE-TIME BEHAVIOR: RLEF improves repair behavior: models recover more reliably from errors, make larger code edits, and are impaired when feedback is randomized.Without RLEF, models often repeat prior solutions or make only minimal edits despite feedback identifying errors.
  • 3.3 INFERENCE-TIME BEHAVIOR: After RLEF, 3-, 5-, and 10-turn rollouts outperform independent sampling at fixed budgets, with best performance at 5 turns and no benefit from 10 turns.
  • 3.4.2 SINGLE-TURN TRAINING: RLEF is the most effective method for acquiring iterative code-synthesis capabilities, while multi-turn RL training generally outperforms single-turn training.A repair model combined with single-turn training reaches 12.6 on test, below the corresponding multi-turn model’s 16.0.

4 RELATED WORK

Prior work improves code generation through prompting, execution-based verification, repair, and reinforcement learning, but independent sampling can outperform iterative repair under equal inference cost. RLEF differs from approaches such as SCoRe by training models to use execution feedback during inference.

  • Code generation: Code-generation research has broadly combined large-scale code pre-training with instruction-following fine-tuning to improve synthesis from natural-language descriptions.This work forms the broader setting for later prompting, repair, and reinforcement-learning methods.
  • Inference-time code improvement: Prompting and flow-engineering methods compile or execute generated code, then use tests and error analysis to guide successive generations.These approaches include unit-test feedback, model-generated error analysis, and iterative code-repair scaffolds.
  • Inference-time code improvement: Independent sampling can outperform repair-based scaffolds when inference cost is controlled, while large models and repeated repair rounds remain problematic.Prior analyses report that independent sampling is competitive with faulty-code repair and that multiple repair rounds are ineffective.
  • Reinforcement learning: Execution-derived rewards have supported reinforcement-learning approaches for code synthesis, including policy-gradient training and binary or fine-grained unit-test rewards.Earlier methods also trained auxiliary predictors or correction models from rollout outcomes, sometimes without explicitly exposing execution output.
  • Reinforcement learning: SCoRe trains two successive solutions without inference-time execution feedback, enabling self-correction where automatic feedback is unavailable but excluding information from execution.This contrasts with methods that condition later generations on environment feedback.
  • Grounded decision-making: Reinforcement learning has also been studied for grounding LLMs in longer-horizon environments such as text-based navigation, games, and purchasing tasks.These studies emphasize successful task completion or interaction with environmental feedback.

5 CONCLUSION

RLEF trains LLMs to ground future code generations in execution feedback and improves iterative code synthesis across several benchmarks and turn limits. The study finds that first-turn correctness and generation diversity contribute substantially, while models also resolve execution errors over multiple turns.

  • Conclusion: RLEF grounds future generations in environment feedback and improves solve rates while reducing the inference sample budget on CodeContests.The method is applied to iterative code synthesis and produces substantial benchmark improvements.
  • Conclusion: RLEF-trained models generalize to more turns and to HumanEval+ and MBPP+, which use simpler problems and different execution-feedback formats.This extends evaluation beyond the primary competitive-programming benchmark.
  • Limitations: The evaluated code-synthesis task improves a single solution rather than decomposing larger tasks into subtasks.Generalization to manually scaffolded or self-directed task decomposition remains future work.
  • Limitations: Iterating on unit-test execution results requires test cases, which may not always be available.The paper identifies automatic unit-test generation as a possible direction for further experiments.
  • Broader impact: The study confines execution of generated source code to local sandboxes and emphasizes quality control and guard-railing for deployed LLM applications.The broader-impact discussion connects code-generation grounding with software development and quality-control use cases.
  • Reproducibility: Experiments use publicly available models and datasets, with training, evaluation, and infrastructure details distributed across the main text and appendices.The reproducibility statement identifies where data processing, model versions, metrics, hyperparameters, and compute details are documented.

A EXPERIMENTAL DETAILS

The experimental setup uses PPO-style policy and value optimization with specified clipping, discounting, regularization, sampling, rollout, evaluation, and code-execution settings. Evaluation applies benchmark-provided limits when available and otherwise uses fixed memory and wall-clock limits.

  • Optimization: PPO uses AdamW, a 2e−7 learning rate, 0.1 weight decay, 50-step warm-up, and KL regularization factor β=0.05.Separate policy and value networks are initialized from pretrained or instruction-tuned LLMs, with a new linear output projection for the value function.
  • Policy optimization: The policy objective uses normalized advantages, the behavior policy, and clipping threshold ϵ=0.2.The cited setup identifies the model parameters θ, normalized advantage Â_t, and behavior policy π_b as components of the objective.
  • Value optimization: The value function uses a clipped value loss with discount factor γ=1 and value clipping threshold α=0.2.The reward function is defined over state-action pairs in the cited formulation.
  • Training and evaluation: Training samples at temperature 1.0 without top-p or top-k sampling, collects 1024 rollouts, and performs four updates on batches of 256 sequences.Models are evaluated every 800 updates and selected using validation performance.
  • Code execution: CodeContests evaluation applies specified memory and time limits, or otherwise uses 1GB memory and 10 seconds per test case.The accompanying benchmark codebase evaluates candidate solutions under these execution constraints.

A.3 SUPERVISED FINE-TUNING

The supervised fine-tuning ablations build an iterative training corpus from 70B rollouts and optimize next-token prediction primarily on successful final responses. Table 4 compares base-model SFT with additional Instruct-model ablations involving feedback, value modeling, and repair.

  • Dataset construction: 313,639 successful trajectories are collected from 100 multi-turn rollouts per training problem using Llama 3.1 70B Instruct.The iterative code-generation setup uses top-p 0.95 and temperatures sampled uniformly from 0.1 to 1.0.
  • Training procedure: SFT computes next-token loss on the last response passing both public and private tests, which performs slightly better than training on all responses.The ablations sweep learning rates 5e−6 and 2e−6 across two and three epochs.
  • Ablations: Table 4a reports 1@3 solve rates for few-shot prompting and SFT with Llama 3.1 Base models on CodeContests.Table 4b reports Instruct 8B ablations that withhold feedback, learn token-level values, or apply a dedicated repair model.

B ADDITIONAL EXPERIMENTAL RESULTS

Few-shot prompting and supervised fine-tuning from pretrained Llama 3.1 models perform substantially worse than corresponding Instruct models on all evaluated cases.

  • Few-shot prompting from pretrained Llama 3.1 models is evaluated using solve rates.
  • Supervised fine-tuning from pretrained Llama 3.1 models is also evaluated using solve rates.
  • Both pretrained-model approaches have significantly lower performance than the corresponding Instruct models in all cases.

B.2 FEEDBACK FROM PRIVATE TESTS

The private-test feedback experiment evaluates broader inference-time feedback and compares it with public-test-only feedback, while also examining an extra repair model. Results vary by model size and split, and the integrated RLEF approach remains stronger than the repair-model variant.

  • B.2 FEEDBACK FROM PRIVATE TESTS: The main evaluation provides public-test feedback during inference while estimating solve rates on private and generated tests.Public test counts range from 1-7, with a median of 1.
  • B.2 FEEDBACK FROM PRIVATE TESTS: 20 test cases with feedback from up to 8 failures improve the 8B model from 17.2 to 18.1 on validation but reduce test performance from 16.0 to 14.4.These are 1@3 solve rates at temperature 0.2 with a three-turn limit.
  • B.2 FEEDBACK FROM PRIVATE TESTS: For the 70B model, larger test feedback improves validation performance from 37.5 to 40.4 and test performance from 38.0 to 41.2.The comparison uses feedback limited to public tests as the baseline.
  • B.3 EXTRA REPAIR MODEL: An extra repair model reaches 14.8 on validation and 12.6 on test, exceeding single-turn RL but remaining below the corresponding RLEF model’s 17.2 and 16.0.The repair-model evaluation generates an initial program followed by up to two independent repair samples.

B.4 RL TRAINING WITHOUT PUBLIC TEST EXECUTION FEEDBACK

Removing public-test execution feedback from subsequent prompts weakens RLEF performance, while replacing turn-level values with token-level values also lowers solve rates. The ablation therefore compares feedback use and value-function granularity against the main setup.

  • B.4 RL TRAINING WITHOUT PUBLIC TEST EXECUTION FEEDBACK: The no-feedback ablation removes execution feedback from subsequent prompts while retaining the reward definition and requesting two follow-up solutions.The setup produces three solutions per problem.
  • B.4 RL TRAINING WITHOUT PUBLIC TEST EXECUTION FEEDBACK: 12.2 validation and 10.9 test solve rates exceed the initial Instruct model but remain below RLEF’s 17.2 and 16.0.
  • B.5 TOKEN-LEVEL VALUE FUNCTION: The token-level value-function variant predicts a value for each response token instead of for the whole response.Its reward formulation remains unchanged, with separate per-token advantages.
  • B.5 TOKEN-LEVEL VALUE FUNCTION: 13.1 validation and 13.7 test solve rates are below the turn-level value-function results of 17.2 and 16.0.

C.2 RANDOM FEEDBACK ABLATION

The random-feedback ablation presents the model with execution feedback obtained from an unrelated incorrect solution, testing whether feedback relevance matters.

  • C.2 RANDOM FEEDBACK ABLATION: Random feedback is generated by evaluating an incorrect solution from a different problem against the corresponding public tests.If no incorrect solution fails, the procedure evaluates raise NotImplementedError().
  • C.2 RANDOM FEEDBACK ABLATION: The unrelated feedback is then presented to the model as execution feedback.
  • C.2 RANDOM FEEDBACK ABLATION: This ablation directly tests model behavior when feedback does not belong to the problem being solved.

D EXAMPLES

The examples show models progressively improving code with execution feedback, from incorrect or inefficient attempts to solutions that pass public and private tests. Across two competitive-programming tasks, later responses correct logic or efficiency problems identified in earlier attempts.

  • Iterative correction: The first example progresses from a wrong print value to a naive but inefficient algorithm, then to code that is both correct and fast.The intermediate loop from l to r would take about two minutes on the last public-test line.
  • Evaluation: The corrected solution in the first task passes both public and private tests.The example explicitly marks the final solution as passing both test sets.
  • Iterative correction: The second example shows a later attempt fixing failure to reset pk, while both attempts retain an unused variable.A dynamic-programming initialization from p_k−1 is preferred, but the corrected attempt still passes all test cases.
  • Evaluation: In the second task, an initial solution fails with incorrect outputs, while a subsequent solution passes public and private tests.The reported failures include expected outputs 0 2 12 22 and 0 3 5 versus incorrect outputs 0 2 14 36 and 0 3 8.
Loading 2410.02089v2…