PaperlayerView exact revision on arXiv

Source-linked AI summary

RLTF: Reinforcement Learning from Unit Test Feedback

Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, Deheng Ye

arXiv:2307.04349v2cs.AIcs.CLcs.LG

TL;DR

Prior code-generation RL methods are described as limited by offline training or coarse use of unit-test signals that misses error locations. RLTF combines online data generation with multi-granularity test feedback, and achieves state-of-the-art performance on APPS and MBPP.

  • Problem

    Prior code-generation RL methods either rely on offline frameworks that limit exploration of new sample spaces or use unit-test signals without accounting for specific error locations.

    Source: §1 · p. 2

  • Method

    RLTF generates program data online during training and uses fine-grained error feedback and adaptive penalties based on the proportion of passed test cases.

    Source: §1 · pp. 2–3

  • Results

    RLTF achieves state-of-the-art performance on APPS and MBPP and surpasses existing RL methods for programming.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The framework is reported as applicable to different LLMs, including CodeT5 and CodeGen.

    Source: §5 · p. 12

    Manual categorization of sub-error types makes transfer to other programming languages challenging.

    Source: §5 · p. 13

Abstract

from arXiv · show

The goal of program synthesis, or code generation, is to generate executable code based on given descriptions. Recently, there has been an increasing number of studies employing reinforcement learning (RL) to improve the performance of large language models (LLMs) for code. However, current representative works either rely solely on offline frameworks, limiting the exploration of new sample spaces, or fall short in the utilization of unit test signals, not accounting for specific error locations within the code. To address these issues, we propose RLTF, i.e., Reinforcement Learning from Unit Test Feedback, a novel online RL framework with unit test feedback of multi-granularity for refining code LLMs. Our approach generates data in real-time during training and simultaneously utilizes fine-grained feedback signals to guide the model towards producing higher-quality code. Extensive experiments show that RLTF achieves state-of-the-art performance on the APPS and the MBPP benchmarks. Our code is available at: https://github.com/Zyq-scut/RLTF.

1 Introduction

RLTF addresses limitations in code-model reinforcement learning by combining online training with multi-granularity unit-test feedback. It achieves state-of-the-art results on APPS and MBPP, with reported applicability across different base models.

1 Introduction

  • Existing code LLMs can struggle to ensure syntactic and functional correctness, while prior RL methods use coarse whole-program test rewards or rely on offline training.Offline training uses pre-collected data, while coarse rewards do not distinguish correct code portions from buggy ones.Source: §1 · p. 2
  • RLTF combines online reinforcement learning with multi-granularity unit-test feedback to refine pretrained LLMs for program synthesis.Its framework generates programs using the latest model parameters and adds generated data to an online buffer for subsequent training.Source: §1 · pp. 2–3
  • Fine-grained feedback uses reported error types and locations to penalize erroneous code parts, while adaptive feedback varies rewards with the proportion of tests passed.The feedback is automatically extracted from unit tests.Source: §1 · p. 3
  • RLTF achieves state-of-the-art results on APPS and MBPP, and tests with CodeT5 and CodeGen illustrate its applicability across base models.The paper also reports that its ablation study demonstrates the effectiveness of the approach.Source: §1 · p. 3

2 Related Work

Related work covers pretrained code models and reinforcement-learning approaches to program synthesis, highlighting correctness challenges and limitations in prior feedback and training frameworks.

2.1 Pretrained LLMs for Program Synthesis

  • Pretrained code language models can generate visually impressive, well-structured code, but guaranteeing syntactic and functional correctness remains challenging.The surveyed models include encoder-only, decoder-only, and encoder-decoder architectures.Source: §2.1 · p. 3
  • Figure 1 depicts two shared-weight LLM roles: one generates compiler-tested programs for an online buffer, and the other trains on buffer and ground-truth data.The buffer retains newly generated data and deletes older data as fresh data arrives.Source: Figure 1 · p. 4

2.2 Reinforcement Learning for Program Synthesis

  • Earlier program-synthesis RL methods include CodeRL and PPOCoder, but prior approaches are described as offline or as using unit-test feedback that overlooks specific error information and code locations.RLTF is presented as adopting an online framework and multi-grained feedback in response.Source: §2.2 · p. 4

3 Approach

RLTF formulates program synthesis as generating code from a high-level description and trains code LLMs with online reinforcement learning and multi-granularity unit-test feedback. Its approach combines real-time sample generation, error-sensitive rewards, and supervised and reinforcement-learning objectives.

3.1 Task Definition

  • RLTF formulates program synthesis as generating code that solves a problem from a high-level description of desired behavior.The objective is to maximize the conditional probability of generating correct code given the description.Source: §3.1 · p. 5

3.2 Online Reinforcement Learning Framework

  • Two shared-weight LLMs train from a continuously refreshed buffer while the other generates code with compiler feedback, enabling online sample generation.Buffer entries pair a problem description, the latest model-generated code, and compiler feedback.Source: §3.2 · p. 5

3.3 Reinforcement Learning from Unit Test Feedback

  • RLTF assigns rewards to explored code according to feedback, with reward coefficients and penalized code-span positions determined by feedback category.The framework derives unit-test feedback from the compiler and groups outcomes as Error, Failure, or Pass.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.1 Feedback Category

  • Compiler feedback distinguishes code that errors, runs but fails requirements, or passes all unit tests.Table 2 groups Python sub-error types as Uglobal, Uline, and Uignore; these correspond to whole-code penalties, line-specific penalties, or temporarily ignored errors when location is unclear.Source: §3.3.1 · pp. 5–6, Table 2 · p. 6, §3.3.2 · p. 7

3.3.2 Multi-granularity Feedback

  • Coarse feedback signals correctness without locating errors, while fine feedback targets specific error locations or broader logic problems and adaptive feedback rewards higher unit-test pass rates.Errors are categorized for whole-code, line-specific, or temporarily withheld penalties; unfinished code caused by the token limit is treated as a global error, otherwise a syntax error is line-specific.Source: §3.3.2 · pp. 6–7

3.4 Optimization Objective

  • The optimization objective combines supervised learning loss with coarse, fine, and adaptive reinforcement-learning losses.Cross-entropy is used for the supervised loss, and an adaptive weight balances fine-feedback variation from different span lengths; a dynamically updated historical-best code baseline is used for coarse and adaptive feedback, but not fine feedback.Source: §3.4 · pp. 7–8

4 Experiments

Experiments compare RLTF with prior RL-based code models across APPS and MBPP, then examine its framework, feedback, sampling temperature, and backbone. RLTF achieves reported state-of-the-art performance on both benchmarks, while ablations and unit-test outcomes characterize its behavior.

4.1 Benchmarks

  • The experiments use APPS and MBPP alongside state-of-the-art RL code-model baselines, with APPS spanning three difficulty levels and MBPP comprising shorter problems.The APPS benchmark has 10,000 problems; MBPP has 974 instances, including 500 test instances.Source: §4 · p. 8, §4.1 · pp. 8–9

4.2 Quantitative Evaluation on APPS

  • On APPS, RLTF surpasses CodeT5-based CodeRL and PPOCoder, achieving state-of-the-art performance; it also outperforms original PPOCoder results except at pass@1.The authors note discrepancies in reported PPOCoder results and include scaled and original results for comparison.Source: §4.2 · pp. 9–10, Table 3 · p. 9

4.3 Ablation Studies

  • The online framework combined with RLTF performs best, while the offline framework without RLTF performs worst; the RLTF benefit is larger online.The authors attribute the larger online benefit to real-time sample generation helping identify more potential program errors.Source: §4.3 · p. 10, Table 4 · p. 10
  • Combining coarse-grained, fine-grained, and adaptive feedback gives the best results, and fine-grained feedback contributes the largest performance boost among evaluated rewards.Removing coarse feedback reduces metrics, while tested fine-grained penalty values each outperform using no penalty.Source: §4.3 · pp. 10–11
  • Training temperature 1 performs better than 0.2, and the authors associate higher-temperature sampling with greater diversity and broader sample-space exploration.The paper reports that more diverse training samples facilitate RL training.Source: §4.3 · p. 11, Table 7 · p. 11
  • RLTF improves pass@10 by almost 1% on CodeGen 2.7B, and the authors report larger performance boosts for larger base models.The experiments evaluate RLTF with CodeGen 2.7B in addition to CodeT5.Source: §4.3 · p. 11, Table 8 · p. 12

4.4 Quantitative Evaluation on MBPP

  • On MBPP zero-shot evaluation, RLTF outperforms different-sized GPT models and achieves state-of-the-art performance among CodeT5-based methods.The CodeT5-based methods, including RLTF, were assessed zero-shot; the GPT-based models were fine-tuned on MBPP.Source: §4.4 · p. 11, Table 9 · p. 12

4.5 Qualitative Analysis by Unit Test Outcomes

  • On APPS, RLTF reduces error outcomes and increases passes, especially on introductory problems; syntax errors decline, while timeout errors rise slightly.The authors interpret the increased failure rate as error codes being fixed into programs that then pass or fail, and report runtime and compiler errors as easier to address than semantic errors.Source: §4.5 · p. 12

5 Conclusions and Future Work

RLTF combines online training with multi-granularity unit-test feedback and reports state-of-the-art results on APPS and MBPP, while noting benchmark-data and cross-language transfer constraints.

  • RLTF surpasses existing RL methods for programming, applies to CodeT5 and CodeGen, and achieves state-of-the-art performance on APPS and MBPP.Source: §5 · p. 12
  • RLTF generates training data on the fly and uses finer-grained feedback signals to guide code generation toward higher quality.Source: §5 · p. 12
  • Benchmark examples may lack diversity, and programs generated from hidden input-output examples may not be correct final versions, potentially restricting performance.Source: §5 · p. 12
  • Manual sub-error categorization makes transfer to other programming languages challenging; automated categorization is proposed to broaden applicability.Source: §5 · p. 13
Loading 2307.04349v2…