Source-linked AI summary
Training Language Models with Language Feedback
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez
TL;DR
Language models can generate outputs that violate human preferences, while pairwise comparisons provide limited information per evaluation. The paper learns from natural-language feedback by generating, selecting, and finetuning on refinements; only 175B-parameter models accurately incorporate feedback, and 100 feedback samples yield roughly human-level summarization.
Problem
Language models often produce offensive, misleading, or factually incorrect text, while pairwise comparison feedback provides limited information per human evaluation.
Method
The method generates many feedback-conditioned refinements, selects the one with highest feedback similarity, and finetunes a language model on the selected refinements.
Results
Only 175B-parameter models accurately incorporate feedback, while 100 human-written feedback samples bring GPT-3 to roughly human-level summarization ability.
Takeaways & Limitations
Natural-language feedback is a promising avenue for learning from human preferences with relatively few human feedback samples.
Takeaways & Limitations
The authors plan larger-scale human evaluations to confirm the initial findings.
Abstract
from arXiv · showhide
Pretrained language models often do not perform tasks in ways that are in line with our preferences, e.g., generating offensive text or factually incorrect summaries. Recent work approaches the above issue by learning from a simple form of human evaluation: comparisons between pairs of model-generated task outputs. Comparison feedback conveys limited information about human preferences per human evaluation. Here, we propose to learn from natural language feedback, which conveys more information per human evaluation. We learn from language feedback on model outputs using a three-step learning algorithm. First, we condition the language model on the initial output and feedback to generate many refinements. Second, we choose the refinement with the highest similarity to the feedback. Third, we finetune a language model to maximize the likelihood of the chosen refinement given the input. In synthetic experiments, we first evaluate whether language models accurately incorporate feedback to produce refinements, finding that only large language models (175B parameters) do so. Using only 100 samples of human-written feedback, our learning algorithm finetunes a GPT-3 model to roughly human-level summarization ability.
1 Introduction
The paper addresses language-model outputs that violate human preferences by replacing sparse pairwise comparisons with more informative natural-language feedback. It proposes a three-step refinement-and-finetuning algorithm and finds strong results with large models and only 100 feedback samples.
- Natural-language feedback conveys more information per human evaluation than pairwise comparisons of model outputs.
- The algorithm samples refinements conditioned on the input, output, and feedback, selects the refinement most similar to the feedback, then finetunes an LM on the selected refinements.
- Only 175B-parameter GPT-3-based models accurately refine outputs in the synthetic offensive-word-removal task.
- Using 100 samples of human-written feedback, the algorithm finetunes GPT-3 to roughly human-level summarization ability.
2 Method
The method formalizes feedback-driven refinement as learning outputs preferred by humans, using a pretrained LM to generate and score candidate refinements before finetuning on the selected ones.
- Given input x, initial output y, and natural-language feedback f, the goal is to generate an output y that is high quality according to human preferences.
- The LM generates N candidate refinements of the initial output conditioned on the input and feedback.
- The method scores candidate refinements for how effectively they incorporate feedback, selects the highest-scoring refinement, and finetunes a model on the selected refinements given x.
3 Experiments
Experiments show that only 175B-parameter models reliably incorporate feedback, while the full algorithm produces human-level summarization results from 100 feedback samples. Feedback-based refinements improve summaries, especially when initial summaries are weaker, and their quality tracks incorporation of specific feedback points.
- 3.2.1 Experimental Setup: Only 175B-parameter GPT-3 and InstructGPT models incorporate feedback a nonnegligible amount of the time.The authors therefore use 175B-parameter Davinci models in the remaining experiments.
- 3.2.1 Experimental Setup: The summarization evaluation uses 100 unseen Reddit posts, human rankings with ties allowed, and a win rate that counts ties as half wins.The study learns from 100 samples of Reddit data and reports mean and standard error for results.
- 3.2.2 Main Results: 51.0 ± 5.0% win rate over human summaries makes feedback-based finetuning roughly comparable to human quality.The strongest baseline, finetuning on INITIAL SUMMARIES, achieves 44.0 ± 5.0%, while the proposed method reaches 57.0 ± 5.0% against that baseline.
- 3.2.3 Analysis: 67.0 ± 3.1% preference over INITIAL SUMMARIES shows that REFINEMENT WITH FEEDBACK + BEST OF N improves initial outputs.The method is preferred to human summaries 54.0 ± 3.5% of the time, whereas INITIAL SUMMARIES are preferred only 39.3 ± 3.4%.
- 3.2.3 Analysis: Feedback-based refinements help more when initial summaries are lower quality, and both feedback use and refinement scoring matter.The ablations compare random refinement selection and refinement without feedback to assess these two components.
- 3.2.3 Analysis: The algorithm incorporates at least one feedback point 72.0 ± 4.5% of the time, versus 15.0 ± 3.6% without feedback.This supports the authors’ explanation that refinement quality is related to incorporating specific points from human feedback.
4 Additional Related Work
Related work has largely used explanations for labeled classification outputs, whereas this paper studies natural-language feedback for improving generated text. Other work uses language for task specification, exploration, reward inference, or test-time correction, while this approach uses feedback for training without test-time human intervention.
- 4 Additional Related Work: Prior NLP work mainly studies explanations for classification labels, whereas this paper studies feedback for improving candidate outputs in general text generation.Explanations describe why an output is correct; feedback describes how to improve it.
- 4 Additional Related Work: Unlike explanation-based classification methods, this work does not assume gold-labeled outputs and targets the more general text-generation setting.The paper notes that classification tasks can also be formulated as text generation.
- 4 Additional Related Work: In reinforcement learning and related domains, language has been used for instruction following, exploration, reward inference, and test-time correction, while this paper uses feedback to train models.The proposed approach does not require human intervention at test time.
5 Conclusion
The paper proposes training language models from natural-language feedback and finds that large models can incorporate feedback, enabling roughly human-level summarization with only 100 feedback samples.
- Using natural-language feedback, the finetuning algorithm brought a GPT-3 model to roughly human-level summarization ability with only 100 samples of human feedback.
- In the targeted word-removal task, only large language models with 175B parameters accurately incorporated feedback.
- Language feedback may make it easier for many people to provide informative, high-quality feedback.
- The approach suggests future applications of language feedback in domains including code generation and conversational assistance.
A Additional Results
Additional results show that feedback-based refinement is especially beneficial when initial summaries are poor, while its advantage reverses for higher-quality initial summaries.
- 83.0±3.9% versus 49.0 ± 5.4%: with rank-4 initial summaries, REFINEMENT WITH FEEDBACK has a higher win rate than REFINEMENT WITHOUT FEEDBACK.
- 7.8±4.0% versus 31.25 ± 5.8%: with rank-2 initial summaries, REFINEMENT WITH FEEDBACK has a lower win rate than REFINEMENTS WITHOUT FEEDBACK.
- The authors explain that feedback on bad summaries is more helpful because bad summaries have more room for improvement.
- Figure 6 plots methods’ win rates against initial summaries by initial-summary ranking and shows that poorer initial summaries yield better refinements.
- B Targeted Word Removal Details: The targeted word-removal setup prompts a model to remove specified offensive words while otherwise preserving the sentence.
C Human Feedback and Evaluation
The evaluation uses human-written feedback and human judgments across several refinement and learning comparisons because automated summarization metrics do not reliably track human preferences.
- Because ROUGE does not correlate well with human summarization preferences, the authors conduct human evaluations of the methods.
- Two authors wrote feedback on the initial summaries after viewing the title, post, and initial summary.
- One author evaluated how often refinements incorporated feedback, while two uninvolved authors evaluated generated refinements for Figures 3, 4, and 6.
- Human evaluations also covered Figures 2 and 5, with one author conducting those evaluations.
D Details about Ranking Procedure
The ranking procedure assigns ranks from 1 to 5 while accommodating tied summaries through adjusted average ranks.
- Each set of five summaries receives ranks between 1 and 5, inclusive.
- Because REFINEMENT WITHOUT FEEDBACK often copies the initial summary exactly, the ranking scheme permits ties.
- Tied summaries receive an adjusted rank r′ based on their shared rank r and the number n of tied elements.
E.1 Generating Refinements
Summarization experiments sample up to 48 tokens with nucleus sampling, while targeted word-removal experiments use greedy decoding up to 200 tokens or a termination token.
- E.1 Generating Refinements: Summarization experiments sample up to 48 tokens with nucleus sampling at p = 0.9.The procedure strips leading nonalphanumeric characters from sampled summaries.
- E.1 Generating Refinements: Targeted word-removal experiments use greedy decoding until 200 tokens or a termination token is generated.The supplied passage presents this as a separate decoding setup from summarization.
- E.1 Generating Refinements: Because of the maximum token length, sampled summaries can end with incomplete sentences that are removed from the output.
E.2 Finetuning
The finetuning experiments train GPT-3 with 175B parameters on refinements with feedback plus best-of-N and initial summaries, using a small hyperparameter search over 100 examples.
- E.2 Finetuning: GPT-3 with 175B parameters is finetuned on refinements with feedback plus best-of-N and initial summaries.
- E.2 Finetuning: Hyperparameters are selected with 5-fold cross-validation on a 100-example training dataset.The search uses the learning-rate multiplier and prompt-loss weight.
- E.2 Finetuning: The experiments use a batch size of 1 and four epochs with OpenAI's default API settings where applicable.
- E.2 Finetuning: The learning-rate multiplier sweep selects 0.05, while the prompt-loss-weight sweep selects 0.01.The model is then finetuned on all 100 examples using the selected settings.
G Examples
The examples compare initial, feedback-based, feedback-free, and human summaries, illustrating how feedback adds requested details and removes unsupported wording.
- G Examples: The examples organize prompts and outputs across initial summaries, human feedback, several refinement methods, and human summaries.Table 2 covers prompt templates, while Table 3 presents example summaries, feedback, and generated refinements.
- G Examples: Feedback-based refinements add requested details such as relationship timing, prior denials, and earlier lies.One example changes an initial summary about an old flame into a more detailed account of the author's concern.
- G Examples: For the Mother's Day example, feedback adds that the relationship's future is unsettled and that the author cannot spend the holiday with her own mother.The feedback-based refinement instead recommends attending to get to know the boyfriend's parents better.