PaperlayerView exact revision on arXiv

Source-linked AI summary

RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs

Afra Feyza Akyürek, Ekin Akyürek, Aman Madaan, Ashwin Kalyan, Peter Clark, Derry Wijaya, Niket Tandon

arXiv:2305.08844v2cs.CL

TL;DR

Human critiques are costly, and fine-tuning-dependent approaches do not fit limited-access task models. RL4F trains a separate critique generator against a fixed model and reports superior final performance across three domains, with relative improvements up to 10%.

  • Problem

    Human critiques are costly to collect online, while learned critique approaches that require task-model fine-tuning do not fit limited-access models.

    Source: §1 · p. 2

  • Method

    RL4F uses reinforcement learning to train a separate critique generator to improve a fixed task model’s end-task performance without updating its parameters.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    RL4F yields superior final performance across three domains, with relative improvements up to 10%.

    Source: §1 · p. 2, §8 · p. 9

  • Takeaways & Limitations

    Its fixed-task-model design addresses settings where task-model fine-tuning is costly or impossible.

    Source: §1 · p. 2

    The analysis is limited to GPT-3, and the method has no explicit restraint against semantic drift.

    Source: §9 · p. 9

Abstract

from arXiv · show

Despite their unprecedented success, even the largest language models make mistakes. Similar to how humans learn and improve using feedback, previous work proposed providing language models with natural language feedback to guide them in repairing their outputs. Because human-generated critiques are expensive to obtain, researchers have devised learned critique generators in lieu of human critics while assuming one can train downstream models to utilize generated feedback. However, this approach does not apply to black-box or limited access models such as ChatGPT, as they cannot be fine-tuned. Moreover, in the era of large general-purpose language agents, fine-tuning is neither computationally nor spatially efficient as it results in multiple copies of the network. In this work, we introduce RL4F (Reinforcement Learning for Feedback), a multi-agent collaborative framework where the critique generator is trained to maximize end-task performance of GPT-3, a fixed model more than 200 times its size. RL4F produces critiques that help GPT-3 revise its outputs. We study three datasets for action planning, summarization and alphabetization and show relative improvements up to 10% in multiple text similarity metrics over other learned, retrieval-augmented or prompting-based critique generators.

1 Introduction

The introduction presents RL4F as a way to train a critique generator to improve a fixed task model, addressing the cost of human feedback and limits on fine-tuning. It reports gains across three tasks and highlights scaling and iterative use.

1 Introduction

  • Helpful critiques may identify one way to improve an answer rather than reproduce the entire answer.The revised prediction is conditioned on the critique.Source: §1 · p. 2
  • Human critiques are costly to collect, especially for online settings requiring a rapid stream of outputs.The paper asks whether an external agent can critique predictions while the language model remains intact.Source: §1 · p. 2
  • RL4F trains a critique generator to maximize a fixed task model’s end-task performance without updating the task model’s parameters.The critique generator supplies feedback for the task model to revise its initial output.Source: §1 · p. 2
  • The framework uses GPT-3 as the task model because smaller models may lack the ability to understand and act on natural-language critiques.The paper focuses on a single critique model for simplicity.Source: §1 · p. 2
  • Relative improvements reach up to 10% across topic-based summarization, action planning, and alphabetization.The introduction also reports promising scaling properties and usefulness when applied iteratively.Source: §1 · p. 2

2 Related Works

The related-work section reviews feedback types and uses, then situates RL4F alongside adapter and discrete-prompt approaches.

2 Related Works

  • The review organizes prior work around feedback type and origin, feedback use, and connections to adapters and discrete prompt learning.These are the topics covered in the section’s subsections.Source: §2 · p. 2

2.1 What kind of feedback is used and where does it originate?

  • Prior feedback includes binary judgments, natural-language critiques, and textual refinements, with approaches using human or model feedback (Ouyang et al., 2022; Bai et al., 2022; Scheurer et al., 2022).Some methods use human critiques with associated refinements, while synthetic critiques may also be used when human critiques are absent.Source: §2.1 · p. 3

2.2 How is feedback used?

  • Unlike approaches that fine-tune task models or retrieve critiques, RL4F trains only the critique generator via reinforcement learning to improve final outcomes (Tandon et al., 2022; Madaan et al., 2022).Its policy network is trained to bootstrap task success, making the proposal orthogonal to RLHF; the experiments use an RLHF fine-tuned checkpoint.Source: §2.2 · p. 3

2.3 Adapters & Discrete Prompt Learning

  • RL4F can be viewed as an alternative adapter when only textual access to the task model is available, and differs from discrete-prompt methods by conditioning critique generation on both input and model output.The cited discrete-prompt methods condition their policy solely on the input.Source: §2.3 · p. 3

3 Background

The section separates predicting, critiquing, and refining outputs, then motivates training a separate critique generator while leaving the task model unchanged. It positions direct refinement as a strong baseline for large models.

3 Background

  • The study focuses on generating critiques because GPT-3 responds well to ground-truth critiques, while generating useful critiques remains a central challenge.The stated goal is to reach and eventually exceed human-level critique performance using machines.Source: §3 · p. 3
  • PREDICT maps inputs to outputs, CRITIQUE generates feedback from an input and initial prediction, and REFINE uses that feedback to produce the target output.DIRECTREFINE repairs an initial answer without a critique.Source: §3 · p. 4

3.1 SUPERVISED: Supervised Learning for Critique Generation

  • SUPERVISED fine-tunes a separate critique model on natural-language critiques, while GPT-3 performs prediction and refinement through in-context learning.Unlike Saunders et al. (2022), this implementation keeps critique generation separate from the other tasks.Source: §3.1 · p. 4
  • Keeping the task model frozen supports already-deployed or expensive models and avoids task-specific changes to a general-purpose model.The critique model is described as multiple orders of magnitude smaller than GPT-3, making its training more computationally efficient.Source: §3.1 · p. 4

3.2 Direct-Refinement

  • Prior work reports that direct refinement can outperform refinement with self-generated critiques for models larger than 50B, motivating its use as a baseline.Saunders et al. (2022) and Bai et al. (2022) attribute this pattern partly to larger models' initial answers becoming harder to self-critique.Source: §3.2 · p. 4

4 RL4F: Reinforcement Learning for Feedback Generation

RL4F continues supervised training of a critique generator with policy gradients, using end-task performance to guide its critiques while keeping the task model fixed. GPT-3 refines the outputs, and reward is based on task performance.

4 RL4F: Reinforcement Learning for Feedback Generation

  • RL4F follows supervised critique training with policy-gradient learning on end-task data, training only LMcritique while treating LMtask as fixed.This addresses the limitation that supervised critique training does not use typically more abundant task data and may not tailor critiques to improving the target model.Source: §4 · p. 5
  • LMcritique takes an input and an initial prediction and produces a natural-language critique; GPT-3 is used as LMtask to refine the prediction.GPT-3 is selected for its adaptability to new tasks through few-shot prompting.Source: §4 · p. 5
  • RL4F warm-starts LMcritique with supervised CRITIQUE training, then uses Proximal Policy Optimization to maximize reward; the authors report PPO worked best in initial experiments.The paper notes that other policy-gradient approaches, such as REINFORCE, could also be used.Source: §4 · p. 5
  • The algorithm samples critiques, has LMtask refine the initial outputs using in-context examples, computes KL-regularized rewards, and updates LMcritique with PPO.The refinement examples are tuples of input, initial prediction, critique, and target output.Source: §4 · p. 5
  • Planning and summarization use ROUGE-1/2/L as a fast reward, collected only when generation ends or reaches its token limit.Reward collection occurs at the terminal stage.Source: §4 · p. 5

5 Datasets

The paper evaluates critique generation on topic-based summarization, action planning, and a synthetic alphabetization task. The datasets range from human-annotated critiques to low-resource action plans and synthetically distorted word lists.

5.1 Topic-Based Summarization

  • Topic-based summarization uses crowd-sourced critiques and improved summaries for model-generated answers, with 14,230 training, 1,150 validation, and 2,658 test tuples.Each passage has multiple questions targeting different aspects, and multiple summaries are sampled for each passage-question input.Source: §5.1 · p. 5

5.2 Interscript

  • Interscript pairs everyday-task goals with reference action sequences, erroneous plans, and natural-language critiques suggesting fixes.Its train, validation, and test sets contain 253, 45, and 169 examples, respectively.Source: §5.2 · pp. 5–6

5.3 Synthetic Task: Alphabetization

  • The synthetic alphabetization task distorts sorted word lists using REORDER, REPLACE, ADD, REPEAT, or REMOVE, while leaving most lists intact with a correctness critique.Lists contain words of lengths 3–12 drawn from a 43K-word lexicon; the dataset has 40K critique warm-start examples and 10K/1K/1K PPO train/dev/test examples.Source: §5.3 · p. 6
  • Alphabetization reward uses inverse-Levenstein distance measured with word-level operations; higher scores indicate closer sorting, and a sorted list receives reward 1.The reward is defined in the passage as inverse-Levenstein distance.Source: §5.3 · p. 6

6 Experiments and Results

The experiments assess critique generators by the final performance of a fixed GPT-3 task model, across planning, summarization, and alphabetization. RL4F performs close to human-critique outcomes on planning and summarization and improves substantially over supervised critiques on alphabetization.

6 Experiments and Results

  • The evaluation judges critiques by final task performance, since different critiques can lead to the same correct answer.Table 1 reports ROUGE, BERTScore, and BLEURT for planning and summarization; alphabetization uses exact match and inverse-Levenshtein scores.Source: §6 · p. 6, §6.1 · p. 8, §6.2 · p. 8
  • RL4F samples critiques from a T5-large-initialized LMcritique using the input and initial prediction, while Figure 3 illustrates GPT-3's critique-conditioned summarization prompt.Direct-Refinement instead supplies the fixed critique “Improve the answer.”Source: §6 · pp. 6–7, Figure 3 · p. 7
  • GPT-3 performs both the task and critique-conditioned refinement using few-shot prompts, with 3, 1, and 6 in-context examples for planning, summarization, and alphabetization.The experiments use code-davinci-002; only the 175B-parameter GPT-3 model can handle REFINE.Source: §6 · p. 6

6.1 Planning and Summarization

  • In action planning, GPT-3 edits the initial plan with Insert, Remove, or Reorder operations; summarization compares repaired outputs with human-written summaries.Planning reward uses a generic lexical-similarity metric.Source: §6 · pp. 6–7

6.1 Planning and Summarization; 6.2 Alphabetization

  • Across planning and summarization, RL4F is among the closest methods to human critiques across metrics; on alphabetization, it gains 27 absolute points over SUPERVISED.The comparisons cover Interscript and topic-based summarization in Table 1, and alphabetization in Table 2.Source: Table 1 · p. 7, Table 2 · p. 7, §6.1 · p. 8

6 Experiments and Results

  • The comparison includes SUPERVISED with unchanged task-model parameters, Direct-Refinement, BM25-based MemPrompt retrieval, and human-written critiques as an upper bound.For tasks where the learned understanding retriever is redundant, MemPrompt uses BM25 instead; all methods start from the same initial predictions.Source: §6 · p. 7

6.2 Alphabetization

  • For alphabetization, RL4F also achieves significantly better accuracy than full fine-tuning while training substantially fewer parameters.The synthetic training triplets may not reflect the errors GPT-3 makes, and SUPERVISED fails to improve on initial outputs.Source: §6.2 · p. 8

7 Analysis

The analysis examines scaling, critique quality, and repeated refinement. RL4F benefits from larger critique models, shows minimal observed semantic drift, and produces additional corrections when applied iteratively.

6 Analysis

  • MemPrompt performs exceptionally on seven cases with a one-concept train–test difference, but fails on the remaining test and validation examples.The cited examples preserve the goal and steps otherwise, with a concept such as “noodle” replaced by “food.”Source: §7 · p. 8

Scaling Properties

  • On Interscript, RL4F benefits greatly as LMcritique scales from 60M to 770M parameters, measured by the average of three ROUGE scores.The figure compares SUPERVISED and RL4F across model sizes.Source: Figure 4 · p. 8, §7 · p. 8

Semantic Drift

  • RL4F does not guarantee natural-language critiques, although the authors observe minimal signs of drift in fluency and naturalness and report that most sampled critiques are intelligible.The authors speculate GPT-3's preference for natural language may explain this, and call for closer study.Source: §7 · p. 8

Iterative Improvement

  • With iterative alphabetization refinement, RL4F yields up to 7 more corrections, while Direct-Refinement sometimes scrambles already correct orderings without adding points.The evaluation does not remove solved examples, and LMcritique may declare a solution correct.Source: Figure 5 · p. 8, §7 · p. 9

8 Conclusion

RL4F trains a critique generator with policy gradients while treating the task model as a black box. The authors report superior final performance across three domains and say the critiques remain fluent and natural.

8 Conclusion

  • RL4F trains the critique generator with policy gradients while treating the task model as a black box.Source: §8 · p. 9
  • RL4F leads to superior final performance across three domains compared with strong baselines.Source: §8 · p. 9
  • The authors report that RL4F critiques remain fluent and natural, and suggest future work on a mixture of experts for critique generation.Source: §8 · p. 9

9 Limitations

The paper identifies scope and design limitations, alongside dataset-processing and experimental details. Its analysis focuses on GPT-3 in a setting where training the task model is inefficient or impossible.

9 Limitations

  • The analysis is limited to GPT-3 and assumes task-model training is inefficient or impossible; the authors note this may be conservative for other settings.Source: §9 · p. 9

A Dataset Processing

  • Interscript uses a subset with distractors removed; alphabetization predictions can contain multiple distortions, but critique warm-starting uses one-step distortions.Source: A Dataset Processing · p. 11

B Experiment Details

  • Experiments use code-davinci-002 as GPT-3 unless otherwise specified, with ROUGE computed using the datasets implementation and stemming for summarization and Interscript.Source: B Experiment Details · p. 12

B.1 Data Formats

  • The listed critique-model inputs pair passages with questions and answers, goals with steps, or unsorted lists with initial answers; separate T5-large models are trained for each dataset.Source: §B.1 · p. 12, • Alphabetization · p. 12

B.2 Prompts for GPT-3

B.3 Standard Deviations

  • In Interscript, standard deviations of R1/R2/RL scores across five runs are 1.4/0.1/1.1 for SUPERVISED and 0.5/0.4/0.5 for RL4F.Source: §B.3 · p. 12

B.4 Hyperparameters

  • The paper directs readers to Tables 3, 4, and 5 for RL4LMs hyperparameters and reports temperature 0 for GPT-3 prompting, except 0.5 when sampling initial alphabetization predictions.Source: §B.4 · p. 12

B.5 Results for Self-Refine (Madaan et al., 2023) on Alphabetization

Self-generated critiques do not consistently improve alphabetization refinements: results vary with the model generating them. The section compares these outcomes with RL4F’s initialized critique model.

B.5 Results for Self-Refine (Madaan et al., 2023) on Alphabetization

  • The few-shot prompt asks a model to describe what is wrong with a supplied word ordering.Source: §B.5 · pp. 12–13
  • 21.6% exact match accuracy with code-davinci-002 sampled critiques is below its initial performance; text-davinci-002 raises exact match to 58.6% but still hurts its initial accuracy of 63.6%.Source: §B.5 · p. 13
  • Self-Refine may improve over initial performance when critiques come from a more capable model such as text-davinci-003 (Madaan et al., 2023).Source: §B.5 · p. 13
  • RL4F’s T5-large-initialized critique model produces useful feedback despite being significantly smaller than text-davinci-003.Source: §B.5 · p. 13

C Sample Predictions

The appendix presents sample predictions for three tasks and describes the learning curves tracked during RL4F training. Manual inspection finds task-specific differences in critique quality.

C Sample Predictions

  • Close to 100% of alphabetization and action-planning critiques are grammatical, while topic-based summarization critiques often repeat themselves or offer generic repair calls.Source: C Sample Predictions · p. 13
  • Tables 6–8 provide sample predictions for Interscript, alphabetization, and topic-based summarization, respectively.Source: Table 6 · p. 14, Table 7 · p. 14, Table 8 · p. 15

D Learning Curves for Reinforcement Learning

Loading 2305.08844v2…