PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Language Models with Language Feedback at Scale

Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez

arXiv:2303.16755v3cs.CLcs.AIcs.LG

TL;DR

Pairwise comparisons provide limited information about human preferences. ILF trains models from language-feedback-guided refinements; on summarization, combining ILF with binary feedback achieved roughly human-level performance.

  • Problem

    Pairwise comparisons between model outputs convey limited information about human preferences, although models can generate harmful or factually incorrect text.

    Source: §1 · p. 1

  • Method

    ILF generates refinements conditioned on an input, an initial model output, and language feedback, selects the refinement that best incorporates feedback, then finetunes on it.

    Source: §1 · p. 2

  • Results

    Combining ILF with binary feedback achieved roughly human-level summarization, with OPT-RM best-of-64 on FeedME summaries reaching a 50.8 ± 1.9% win rate using 5K training samples.

    Source: §4.3.3 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    ILF and binary feedback each capture valuable preference information that can be cumulative when used together.

    Source: §1 · p. 2, §4.3.3 · p. 7

    Initial results for multiple ILF iterations are promising, but further experimentation is needed to understand them better.

    Source: §4.3.3 · p. 7

Abstract

from arXiv · show

Pretrained language models often generate outputs that are not in line with human preferences, such as harmful text or factually incorrect summaries. Recent work approaches the above issues by learning from a simple form of human feedback: comparisons between pairs of model-generated outputs. However, comparison feedback only conveys limited information about human preferences. In this paper, we introduce Imitation learning from Language Feedback (ILF), a new approach that utilizes more informative language feedback. ILF consists of three steps that are applied iteratively: first, conditioning the language model on the input, an initial LM output, and feedback to generate refinements. Second, selecting the refinement incorporating the most feedback. Third, finetuning the language model to maximize the likelihood of the chosen refinement given the input. We show theoretically that ILF can be viewed as Bayesian Inference, similar to Reinforcement Learning from human feedback. We evaluate ILF's effectiveness on a carefully-controlled toy task and a realistic summarization task. Our experiments demonstrate that large language models accurately incorporate feedback and that finetuning with ILF scales well with the dataset size, even outperforming finetuning on human summaries. Learning from both language and comparison feedback outperforms learning from each alone, achieving human-level summarization performance.

1. Introduction

Pairwise preference feedback conveys limited information, motivating ILF, which uses language feedback to refine and select outputs for finetuning. In summarization, combining ILF with comparison feedback achieved human-level performance.

1. Introduction

  • Unlike pairwise comparisons, which convey limited information about preferences, language feedback offers a more informative and natural training signal.The paper presents ILF as an alternative to approaches using reinforcement learning or auxiliary losses.Source: §1 · pp. 1–2
  • ILF generates multiple refinements from the input, an initial output, and written feedback, selects the refinement that best incorporates feedback, then finetunes on it.The process can be repeated to gather feedback on outputs from the updated model.Source: §1 · p. 2
  • The paper views ILF as Bayesian inference, and its controlled word-removal task found that 175B-parameter GPT-3-based models accurately refined outputs.The authors also report that ILF improves summaries and that language and comparison feedback together yield human-level performance.Source: §1 · p. 2
  • Combining ILF with comparison feedback outperformed either feedback type alone, with human evaluators preferring its summaries to high-quality human references 50.8% of the time.ILF alone improved monotonically with feedback up to 5k samples and was comparable to or better than finetuning on human-written summaries across data regimes.Source: §1 · p. 2

2. Methods

ILF trains a language model to approximate a distribution of high-quality outputs using evidence from language feedback. Its sampling-and-finetuning procedure is linked to Bayesian inference and reinforcement learning from human feedback.

2. Methods

  • Given context, an initial output, and language feedback, a refinement model proposes a revised output; the model is then trained on feedback-selected refinements.The paper frames the goal as updating a language model based on evidence provided by language feedback.Source: §2 · pp. 2–3
  • Because the proposal distribution cannot be evaluated directly, ILF uses importance sampling and assumes its samples are equally good under high-quality human feedback.The importance weight for each sampled refinement is used in approximating the objective.Source: §2 · p. 3
  • The method operationalizes feedback incorporation with an instruction-finetuned language model that answers a binary yes-or-no question about whether a refinement addresses the feedback.The probability of a positive answer serves as the reward.Source: §2 · p. 3
  • ILF fits a model to a reward-proportional distribution of high-quality outputs, which the paper relates to Bayesian inference and reinforcement learning from human feedback.The resulting distribution supports drawing diverse high-quality outputs, giving downstream users and systems more control over preferred or avoided aspects.Source: §2 · pp. 2–3

3. Can Language Models Use Feedback?

The authors test whether language models can apply feedback in a controlled word-removal task. Only the largest models incorporated feedback to a non-negligible extent, motivating their use in later experiments.

3. Can Language Models Use Feedback?

  • The refinement test addresses a prerequisite for ILF: language models must be able to incorporate feedback accurately to generate useful refinements.The experiment compares model sizes to determine which model to use for refining outputs.Source: §3 · pp. 3–4
  • The controlled task asked models to remove up to three specified offensive words from automatically generated sentences containing up to ten, and measured exact target matches.Models produced one greedy-decoded output per sample, without best-of-N sampling.Source: §3 · p. 3
  • Only the largest GPT-3 and FeedME models, at 175B parameters, incorporated feedback in a non-negligible amount of time; later experiments therefore used 175B FeedME.Table 1 reports percentage of exact string matches with the target for the offensive-word removal task.Source: §3 · p. 4, Table 1 · p. 4

4. Summarization from Language Feedback

The section evaluates language-feedback refinement for TL;DR summarization, comparing refinement-selection strategies and learning from language feedback, binary feedback, and reference summaries. It reports that combining language and binary feedback reaches human-level performance, while ILF alone outperforms finetuning on human summaries.

4.1. Summarization with Language Feedback Dataset

  • The study uses the TL;DR Reddit summarization dataset and evaluates methods on 698 test samples with human rankings against human-written reference summaries.Annotators supplied feedback on an initial FeedME summary and also wrote an ideal summary; the dataset includes 5,000 training samples.Source: §4.1 · p. 4, §4.3.2 · p. 6

4.2. Comparing Refinement Ranking Methods

  • InstructRM Ensemble achieves a 56.0 ± 3.0% win rate against random refinement selection; embedding similarity does not outperform random selection on this dataset.The authors use InstructRM Ensemble for subsequent refinement selection; they suggest the dataset’s more varied crowdsourced feedback may explain the embedding method’s weaker performance.Source: Table 2 · p. 4, §4.2 · p. 5

4.3.1. METHODS

  • ILF uses one refinement-learning iteration; the comparison includes finetuning on refinements, initial summaries, or human summaries, plus reward-model best-of-N sampling and a combined approach.The combined method finetunes on refinements, then uses OPT-RM to rank 64 sampled summaries at test time.Source: §4.3.1 · pp. 5–6

4.3.3. RESULTS

  • 50.8 ± 1.9% win rate against human summaries: ILF + OPT-RM (best-of-64) reaches similar quality to human summaries with 5K training samples.OPT-RM best-of-64 on FeedME summaries outperforms the finetuning and sampling approaches across sample sizes, at the cost of added inference time.Source: Figure 3 · p. 5, §4.3.3 · p. 7
  • 31.3 ± 1.7% win rate against human summaries: ILF finetuned on 5K refinements outperforms finetuning on human summaries (28.9 ± 1.7%), initial summaries (27.3 ± 1.7%), and FeedME (22.5 ± 1.6%).ILF outperformed finetuning on human summaries across sample sizes, although human-written summaries were generally higher quality.Source: §4.3.3 · p. 6

4.4. Does Language Feedback Improve Refinements?

  • 69.1 ± 1.9% win rate over initial summaries: Refinement with Feedback + Best of N exceeds feedback-only refinement (63.9 ± 2.0%) and refinement without feedback (59.4 ± 2.1%).The authors conclude language feedback improves refinement quality, especially when combined with Best-of-N sampling.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Do Refinements Incorporate the Feedback?

  • 57.4 ± 2.2% of refinements from Feedback + Best of N incorporate the feedback’s most important point, versus 49.6 ± 2.2% for feedback-only refinement.Human summaries written from scratch address that point 74.0 ± 1.9% of the time; refinements without feedback do so 30.8 ± 2.1% of the time.Source: §4.5 · pp. 7–8

4.6. Which Finetuning Dataset Changes Models Most?

  • Finetuning on refinements produces the largest forward and reverse KL divergence from the pretrained model, followed by human-summary finetuning and then initial-summary finetuning.The authors suggest the larger divergence with ILF may partly account for its larger gains in human evaluations.Source: §4.6 · p. 8

5. Related Work

Related work distinguishes feedback that guides how to improve an output from explanations that justify a labeled answer, and situates ILF among feedback-learning and Bayesian-inference approaches.

5. Related Work

6. Conclusion

The conclusion presents ILF as an iterative way to train language models from language feedback, validated on a controlled word-removal task and summarization.

6. Conclusion

  • Combining ILF with binary feedback brought GPT-3 to roughly human-level summarization, while ILF alone outperformed finetuning on human summaries.On the controlled word-removal task, only large models (175B parameters) accurately incorporated feedback.Source: §6 · p. 9

A. Additional derivations

ILF is framed as Bayesian inference: language feedback informs a proposal distribution for high-quality outputs, while model updates correspond to supervised learning. Unlike prior language-in-RL uses described here, ILF uses language to correct behavior during training without human intervention at test time.

A.1. Imitation Learning from Language Feedback as Bayesian Inference

  • The derivation maximizes the marginal log probability of high-quality outputs across contexts using an importance-sampling proposal and an evidence lower bound, optimized by alternating E- and M-steps.The E-step updates the proposal distribution; the M-step updates the language model.Source: §A.1 · p. 13
  • In the E-step, an LM generates an initial output, a human supplies feedback, and the LM produces a refinement whose quality is modeled using a reward-based Boltzmann distribution.The reward depends on the context, initial output, feedback, and refinement; β is its temperature hyperparameter.Source: §A.1 · p. 13
  • Figure 5 contrasts the target distribution over context, high-quality output, and human-preference judgment with the proposal distribution using an initial output and its feedback.The target is approximated by the algorithm; the proposal is used for importance sampling.Source: Figure 5 · p. 13
  • The M-step is supervised learning on data defined by q, and ILF alternates it with the E-step to update the language model.In practice, the previous-iteration model is used to generate initial outputs and refinements, while an instruction-finetuned LM scores feedback incorporation.Source: §A.1 · pp. 13–14
  • The proposal q weights refined outputs using the modeled probability that they are high quality, with importance sampling approximated by best-of-N refinement sampling.The account assumes informative feedback and a reward function that favors refinements improving on the initial output with respect to the context and feedback.Source: §A.1 · p. 14
  • ILF can be understood as Bayesian inference, corresponding to reinforcement learning with human feedback that has also been shown equivalent to Bayesian inference.The interpretation describes updating an LM based on evidence provided by language feedback.Source: §A.1 · p. 14

B. Additional Related Work on Language in RL Settings

  • Prior work uses language in RL to specify tasks, drive exploration, infer rewards, or provide supervision and trajectory descriptions; this work instead uses language to correct faulty behavior during training without test-time human intervention.Other cited work uses feedback at test time for image segmentation or code generation.Source: B. Additional Related Work on Language… · p. 14

C. Dataset Collection and Analysis

The dataset section describes annotation quality, feedback content, annotation times, and procedures for constructing targeted word-removal data and ranking summaries.

C. Dataset Collection and Analysis

  • 81.0% author-annotator agreement and 70% annotator-annotator agreement were reported for binary comparisons on a 10-post gold-standard sample.Seventeen annotators labeled the sample; 31 qualified annotators were selected for the annotation tasks.Source: C. Dataset Collection and Analysis · pp. 14–15
  • 92.0% of training-sample feedback was reported complete, and feedback categories were coverage (77.0%), accuracy (16.0%), coherence (5.0%), and other (2.0%).Average lengths were 41.0 ± 0.1 tokens for human-written summaries, 32.5 ± 0.1 for extracted Reddit summaries, 29.3 ± 0.1 for initial FeedME summaries, and 20.4 ± 0.2 for feedback.Source: C. Dataset Collection and Analysis · p. 15
  • 61.5 ± 5.3 seconds for binary comparisons, 182.5 ± 6.3 for feedback, and 195.5 ± 6.1 for ideal summaries were the reported mean annotation times.Figure 6 plots development-set annotation times; samples under 20 seconds or over 420 seconds were excluded.Source: C. Dataset Collection and Analysis · p. 15, Figure 6 · p. 15

D. Targeted Word Removal Details

  • For k = 1 to 10, 50 offensive sentences were sampled using k words drawn uniformly without replacement from a fixed set of 25; the task removed l = 1, 2, or 3 words, with k ≥ l.Each offensive sentence also included the words nice person; removing l = k offensive words still left a target sentence that intuitively makes sense.Source: D. Targeted Word Removal Details · pp. 15–16

E. Details about Ranking Procedure

F. Reward Model

The section compares scalar and language-output reward models and selects OPT-RM for subsequent comparison with ILF. It also specifies the shared summary-generation and post-processing setup.

F. Reward Model

  • The language-output reward model predicts a good/bad label while jointly optimizing label likelihood and summary likelihood; the standard reward model predicts a scalar preference score.The comparison reward model instead predicts which of two summaries is preferred; the evaluated language-output OPT-RM uses an instruction asking whether a summary is excellent.Source: F. Reward Model · p. 16
  • 74.2 ± 2.0% accuracy with GPT-3 Binary and 73.4 ± 1.9% with OPT-RM; both language-loss OPT models outperform the standard OPT reward model’s 71.8 ± 2.0%.OPT comparison reaches 72.6 ± 1.9%; the results evaluate prediction of human-preferred summaries.Source: F. Reward Model · p. 16, Table 4 · p. 17
  • OPT-RM reaches a 63.3 ± 2.7% development-set win rate versus 61.8 ± 2.9% for GPT-3 Binary, leading the authors to select OPT-RM for comparisons with ILF.Considering validation and development results together, the authors report similar performance for OPT-RM and GPT-3 Binary.Source: F. Reward Model · pp. 16–17
  • Reward-model performance depends strongly on the train and test preference distributions, and mismatches can make models sample-inefficient even with more training examples.For example, OPT-RM trained on 90K examples from Stiennon et al. (2020) reaches 68.6 ± 2.0% on this paper’s validation set, below 73.4 ± 1.9% for the model trained on 5K in-house examples.Source: F. Reward Model · p. 17

G.1. Generating Refinements

  • Summarization generations use nucleus sampling with p = 0.95 and temperature 1.0, then remove empty outputs and incomplete ending sentences.The same settings and post-processing apply to initial summaries, refinements, and test summaries.Source: §G.1 · p. 18

G.2. Finetuning on Summaries

The appendix reports summary-finetuning choices and qualifications, settings for multiple ILF iterations and reward-model training, and an analysis of finetuned models. In the reported likelihood evaluation, refinements have lower negative log-likelihood than human summaries.

G.2. Finetuning on Summaries

  • For summary finetuning, hyperparameters were swept separately at 100, 1K, and 5K examples, and the authors found that training beyond one epoch resulted in overfitting.They selected hyperparameters using human summaries before applying the same choices to refinements and initial summaries.Source: §G.2 · p. 18
  • The 1K-sample refinement and initial-summary runs used prompt loss weight 0 instead of the intended 0.05, though the authors report only a minimal perplexity difference between settings.The reported perplexities are 6.71 at weight 0 and 6.68 at weight 0.05; the method still outperformed human-summary finetuning in that regime.Source: §G.2 · p. 18

G.3. Multiple Iterations of ILF

  • Multiple-iteration ILF experiments finetune GPT-3-175B on datasets of 200 and 300 refinements, with separate hyperparameter choices for each size.Both selected settings use one epoch; prompt-loss weights are 0.05 for 200 refinements and 0 for 300.Source: §G.3 · p. 18

G.4. Finetuning Reward Models

  • Reward-model hyperparameters were tuned separately by model type; GPT-3 tuning used 1K training examples for cost reasons, then reused its settings for 5K training with early stopping.For GPT-3-175B classification, the binary annotations double the effective training set to 2K examples; OPT tuning used 200 development examples.Source: §G.4 · p. 19

H.1. Analyis of Finetuned Models

  • The finetuned-model analysis evaluates likelihood on summaries from each training distribution and measures forward and reverse KL divergence against pretrained GPT-3-175B.For the reverse KL, samples come from the finetuned models; for the forward KL, they come from GPT-3-175B.Source: §H.1 · pp. 19–20, Table 6 · p. 20
  • The model finetuned on refinements has significantly lower negative log-likelihood than the model finetuned on human summaries in the validation evaluation.The comparison evaluates models trained on 5K initial summaries, refinements, or human summaries on 500 examples from the corresponding validation datasets.Source: Figure 8 · p. 20

H.2. Results: ILF + OPT-RM

On 5K samples, combining ILF with OPT-RM best-of-64 reaches a 50.8% win rate against human summaries, while finetuning on feedback plus refinements performs worse than refinements alone. The section describes the combined approach and its evaluation scope.

H.2. Results: ILF + OPT-RM

  • The reported evaluation ranks test summaries by human annotators and computes win rate against human-written summaries; all methods compared here use 5K samples.The compared methods differ from those in Figure 3, and the authors note that human annotation noise can also make results vary across comparisons.Source: §H.2 · pp. 20–21
  • The feedback-plus-refinements method trains on initial summaries, feedback, and generated refinements, predicting both feedback and refinement during training.At test time it starts from FeedME-generated initial summaries and generates feedback and refinements before post-processing to the 48-token limit.Source: §H.2 · pp. 20–21
  • 50.8 ± 1.9% win rate for ILF + OPT-RM best-of-64 achieves human-level summarization performance, versus 36.0 ± 1.8% for finetuning on refinements and 35.4 ± 1.8% for human-summary finetuning.All methods in this comparison use 5K samples; OPT-RM best-of-64 FeedME reaches 45.1 ± 1.9%, while feedback-plus-refinements finetuning reaches 26.1 ± 1.7%.Source: Figure 9 · p. 21, §H.2 · p. 21

H.3. Multiple Iterations of ILF

Across multiple ILF iterations, finetuning from scratch on combined refinement datasets improved the baseline at 300 samples, but not at 200; continuous finetuning did not improve the baseline. The authors caution that the comparisons are affected by differing training approaches and unresolved hyperparameter and dataset-size effects.

H.3. Multiple Iterations of ILF

  • 53.2 ± 1.9% win rate favored two ILF iterations over one, suggesting multiple rounds can improve the data distribution.This comparison evaluates two iterations against one; the later experiment uses a more realistic baseline.Source: §H.3 · p. 22
  • The comparisons may be affected by different hyperparameters and dataset sizes, and the authors call for further experiments to clarify their effects and the best iteration count.The authors also plan to test a direct implementation that uses the continuously finetuned model to generate later-round summaries.Source: §H.3 · pp. 22–23
  • The scratch approximation concatenates refinements from successive iterations, whereas direct ILF continuously finetunes the model on additional refinements.The scratch-trained models approximate, rather than directly implement, the distribution induced by ILF.Source: §H.3 · p. 23
  • On the test dataset, ILF approximated by finetuning from scratch on distributions induced by ILF beat the baseline at 300 samples (55.6 ± 1.9%) but not 200 (48.9 ± 1.9%).Direct continuous ILF did not improve on the baseline, with win rates of 49.4 ± 1.9% and 50.9 ± 1.9% for 200 and 300 samples, respectively.Source: §H.3 · p. 23
  • The authors hypothesize that continuous finetuning may cause catastrophic forgetting, while scratch finetuning may retain gains from improved refinement distributions.They leave unexplained why the 200-sample scratch model does not benefit from the improved distribution.Source: §H.3 · p. 23

H.4. Part-of-Speech Distribution for Finetuning Datasets

The three finetuning datasets have similar word distributions, but content words have a larger influence on normalized mean loss for refinements.

H.4. Part-of-Speech Distribution for Finetuning Datasets

  • The initial-summary, refinement, and human-summary datasets have very similar distributions of function, content, and other words.The analysis uses 1K-sample finetuning datasets and evaluates summaries rather than prompts.Source: §H.4 · p. 23
  • Content words have a bigger influence on normalized mean loss for the refinements dataset.The authors believe this relates to their finding that finetuning on refinements gives the best results.Source: §H.4 · p. 23
  • The analysis groups tokens by part of speech, keeps numbers and symbols in an “others” group, and normalizes loss values to mean 0 and standard deviation 1.The authors note that average loss is higher for human summaries.Source: §H.4 · p. 23

H.5. Comparison to Results of Scheurer et al. (2022)

Compared with Scheurer et al. (2022), this paper reports different relative outcomes for finetuning and refinement selection, alongside differences in baselines and evaluation setup. Its appendices describe annotation instructions, prompt templates, and reward-model prompts.

H.5. Comparison to Results of Scheurer et al. (2022)

  • 19.6 ± 1.5% win rate against human summaries was reported for both finetuning on 100 refinements and finetuning on initial summaries; FeedME reached 20.8 ± 1.5%.Unlike the relative ordering reported by Scheurer et al. (2022), both finetuned methods performed equally and worse than FeedME in this paper.Source: §H.5 · p. 24
  • 69.1 ± 1.9% win rate for Refine with Feedback + Best-of-5, 63.9 ± 2.0% for Refine with Feedback, and 59.4 ± 2.0% for Refinement without Feedback were reported; human summaries reached 83.2 ± 1.7%.The passage compares these values with Scheurer et al. (2022) results and notes the human-summary difference was expected given the stronger summaries used here.Source: §H.5 · p. 24
  • Embedding similarity worked well for refinement selection in Scheurer et al. (2022) but not in this setting, which the authors associate with more diverse feedback from many annotators.The studies also differ in evaluation dataset size, annotators, prompts, hyperparameters, and human-summary quality.Source: §H.5 · pp. 24–25

I. Annotator Instructions

  • Annotation instructions were task-specific and updated over time; the authors provide the dataset-creation and summary-evaluation instructions but omit others for brevity.Source: §I · p. 25

J.1. Summarization Prompts

J.2. InstructRM Prompts

  • Five InstructRM prompt templates are reported; one author without domain knowledge or prior prompt information wrote prompts aimed at selecting high-quality refinements.Source: §J.2 · pp. 28–29, Table 9 · p. 31

J.3. Finetuning Prompts

  • The finetuning templates support completions from initial summaries, refinements, or human summaries; the feedback-plus-refinement format provides feedback before the refinement.Source: §J.3 · p. 31, Table 10 · p. 32

J.4. Reward Model Prompts

  • Table 11 reports prompts for training the reward model with language-model loss, including both classification and comparison prompts.Source: Table 11 · p. 32
Loading 2303.16755v3…