PaperlayerView exact revision on arXiv

Source-linked AI summary

Chain of Hindsight Aligns Language Models with Feedback

Hao Liu, Carmelo Sferrazza, Pieter Abbeel

arXiv:2302.02676v8cs.LGcs.CL

TL;DR

Language-model alignment methods either rely on positively rated examples or use reinforcement learning with potentially imperfect rewards and difficult optimization. Chain of Hindsight converts feedback into language sequences and conditions on feedback paired with model generations. It substantially outperforms RLHF and other baselines on summarization and dialogue tasks, with strong results in human evaluations.

  • Problem

    Existing alignment methods rely on positively rated data or reinforcement learning with potentially imperfect reward functions and difficult optimization.

  • Method

    Chain of Hindsight converts human feedback into language sequences and fine-tunes models conditioned on model generations paired with feedback.

  • Results

    CoH substantially outperforms SFT variants, RLHF, and other baselines on summarization and dialogue across automated and human evaluations.

  • Takeaways & Limitations

    CoH enables language models to use feedback regardless of preference polarity while identifying and correcting errors or negative attributes.

  • Takeaways & Limitations

    Constructing CoH sequences can produce long sequences and increase training computational expenses, especially with multiple feedback instances.

Abstract

from arXiv · show

Learning from human preferences is important for language models to match human needs and to align with human and social values. Prior works have achieved remarkable successes by learning from human feedback to understand and follow instructions. Nonetheless, these methods are either founded on hand-picked model generations that are favored by human annotators, rendering them inefficient in terms of data utilization and challenging to apply in general, or they depend on reinforcement learning, which often suffers from imperfect reward functions and relies on extremely challenging optimizations. In this work, we propose a novel technique, Chain of Hindsight, that is easy to optimize and can learn from any form of feedback, regardless of its polarity. Our idea is inspired by how humans learn from extensive feedback presented in the form of languages. We convert all types of feedback into sequences of sentences, which are then used to fine-tune the model, allowing us to take advantage of the language comprehension capabilities of language models. We condition the model on a sequence of model generations paired with feedback. By doing so, the model is trained to generate outputs based on feedback, while learning to identify and correct negative attributes or errors. Applying our method to large language models, we observed that Chain of Hindsight significantly surpasses previous methods in aligning language models with human preferences. We report significant improvements on summarization and dialogue benchmarks, with our approach markedly preferred in human evaluations.

1 Introduction

Chain of Hindsight addresses limitations of SFT and RLHF by conditioning language models on sequences of outputs paired with natural-language feedback. It substantially improves alignment performance on summarization and dialogue tasks in automated and human evaluations.

  • Motivation: SFT depends on labeled, positively rated data, while RLHF requires a potentially imperfect reward function and difficult reinforcement-learning optimization.These limitations motivate a method that can use feedback with either polarity without reinforcement learning.
  • Core idea: CoH conditions language models on sequences of generations paired with feedback so they can identify and correct errors and negative attributes.The approach converts human feedback into sequences and fine-tunes the model to use comparisons among outputs.
  • Core idea: The method presents feedback such as “Bad” and “Good” and trains the model to generate outputs that better match the preferred feedback.Natural-language comparisons can also provide task-specific guidance beyond a binary preference label.
  • Results: CoH substantially outperforms SFT variants and RLHF across summarization and dialogue in automated assessments and human evaluations.Figure 1 specifically reports human pairwise comparisons against RLHF and supervised fine-tuning baselines.
  • Method overview: Figure 2 depicts responses being ranked by human preference, converted into natural-language feedback sequences, and used for fine-tuning.The construction combines model outputs with feedback while preserving the pretraining-style objective.
  • Contribution: CoH uses all available feedback data without relying on RLHF and retains the pretraining training objective, making it straightforward to train and scalable.The authors present this as a principal contribution alongside extensive comparisons with existing baselines.

2 Chain of Hindsight

Chain of Hindsight represents model outputs and human feedback as sequences for autoregressive fine-tuning. Compared with prior preference-learning paradigms, it uses both positive and non-positive data while conditioning on richer feedback-example sequences.

  • Sequence construction: CoH constructs training sequences by combining multiple model outputs with feedback and uses them for instruction fine-tuning.The sequence contains generated responses paired with feedback based on human ratings.
  • Feedback representation: CoH uses feedback templates such as “A good summary” and “a worse summary” to express positive and negative ratings in natural language.The study uses predetermined templates based on ratings rather than open-ended human feedback.
  • Inference: At inference, positive feedback such as “Good” guides the model toward favorable outputs.The inference procedure prompts the model with positive feedback to guide generation.
  • Objective: CoH masks loss on feedback tokens and predicts model-generated tokens so feedback can guide generation at inference time.The authors state that applying loss to other tokens would hinder generation during inference.
  • Training: The model is trained autoregressively with cross-entropy loss to predict the constructed input sequence.The training procedure samples output-rating minibatches, creates hindsight feedback, and optimizes the resulting sequences.
  • Relation to prior paradigms: Unlike SFT, CoH incorporates both positive and non-positive rated data; unlike conditional SFT, it conditions on a sequence of feedback-example pairs.Relative to RLHF, the paper characterizes CoH as having a simpler training process.

3 Evaluation Setup

The evaluation uses automatic and human assessments on summarization and dialogue benchmarks, with comparisons against tuned SFT variants and RLHF. Summarization is measured on TL;DR using ROUGE scores, while dialogue evaluation uses helpfulness, harmlessness, and human preference ratings.

  • Training Data: Training uses WebGPT, HH, and summarization datasets containing human preferences over answers, conversations, and summaries.WebGPT contains 19,578 comparisons, while HH and summarization provide pairwise human preferences for dialogues and summaries.
  • Evaluation Benchmarks: The study evaluates models with both automatic and human evaluation on summarization and dialogue benchmarks.These evaluations cover the TL;DR summarization dataset and Anthropic’s Helpful and Harmless dialogue dataset.
  • Evaluation Benchmarks: The summarization benchmark uses the filtered TL;DR validation set containing 123,169 Reddit posts and evaluates summary quality with coverage, quality, and ROUGE-related measures.The supplied figure caption specifically identifies ROUGE scores on the TL;DR summarization task.
  • Evaluation Benchmarks: The dialogue benchmark evaluates helpfulness and harmlessness primarily through labelers’ preference ratings on pseudo-dialogues constructed from positive examples.Pseudo-dialogues replace prior model responses with outputs conditioned on human responses and earlier model outputs.
  • Baselines: The primary baselines are SFT, SFT with unlikelihood, conditional SFT, and RLHF, implemented with GPT-J 6B and OPT base models.PPO is used for the RLHF baseline, and hyperparameters are tuned across baselines for comparison.

4 Results

CoH is evaluated against SFT variants and RLHF on summarization and dialogue, using automatic and human assessments. It substantially outperforms these baselines, while natural-language feedback and larger model sizes further improve results.

  • Summarization: CoH substantially outperforms pretrained, SFT, conditional SFT, SFT with unlikelihood, and RLHF baselines on summarization ROUGE scores.RLHF ranks second, followed closely by conditional SFT.
  • Summarization: 75 human labelers compared CoH with baselines on summarization accuracy, coherency, and coverage, with CoH substantially outperforming RLHF and conditional SFT.The evaluation used pairwise comparisons and allowed ties.
  • Dialogue: CoH substantially outperforms RLHF and SFT baselines when classifying which dialogue in a pair is preferred.The dialogue evaluation uses the Helpful and Harmless dataset and measures preferred-dialogue classification accuracy.
  • Language feedback: Natural-language feedback raises human preference from 11.6% without language feedback to 14.1% with it, while both CoH variants outperform RLHF.This ablation evaluates summarization using 75 human evaluators.
  • Model scaling: CoH may slightly underperform SFT at smaller model sizes, but consistently surpasses SFT and RLHF as model size increases, showing a positive scaling trend.The scaling comparison is conducted on the summarization benchmark.

5 Related Work

Related work places CoH at the intersection of hindsight-based learning, human-feedback alignment, and instruction finetuning. Unlike reinforcement-learning-based hindsight methods, CoH uses a simpler conditioning-and-finetuning approach.

  • Learning from hindsight: Hindsight experience replay retroactively relabels rewards and transitions in goal-conditioned reinforcement learning, motivating hindsight-based learning.The paper positions CoH as learning from chains of hindsight with human feedback.
  • Learning from hindsight: HIR is described as a one-step chain-of-hindsight special case of CoH, but uses likelihood, contrastive, and entropy losses in a more complex training process.CoH is presented as straightforward and easy to implement by comparison.
  • Learning from human feedback: Prior human-feedback methods span supervised finetuning on filtered annotations and reward-function learning for reinforcement learning across tasks including summarization and dialogue.The cited applications also include translation, semantic parsing, story generation, review generation, evidence extraction, and instruction following.
  • Instruction finetuning and conditional training: Instruction finetuning commonly reformats NLP benchmark examples as instructions, whereas CoH finetunes on chains pairing model outputs with human feedback.The paper relates CoH to instruction finetuning through pretrained models’ in-context learning abilities.

6 Conclusion

Chain of Hindsight conditions language models on hindsight feedback so they can use examples regardless of preference. The paper reports strong performance while identifying computational and evaluation-cost limitations and future extensions.

  • CoH conditions language models on sequences of hindsight feedback, allowing them to leverage examples regardless of preference.
  • CoH substantially outperforms RLHF and other baselines on summarization and dialogue datasets.
  • Constructing CoH can create long sequences with multiple feedback instances, increasing training computational expenses.
  • The evaluation relies heavily on hired human labelers because they are considered more reliable than automated metrics, incurring substantial costs.
  • Future work includes integrating external environment feedback such as unit tests and extending CoH to other domains.

A Human Evaluation Instructions

Human evaluations use pairwise preference judgments with task-specific instructions and definitions, while the appendix documents example natural-language feedback formats.

  • A Human Evaluation Instructions: Human labelers select the preferred output using supplied metric definitions and evaluation instructions.
  • A Human Evaluation Instructions: The summarization instructions are derived from Stiennon et al., while dialogue instructions are based on Bai et al.
  • A Human Evaluation Instructions: Table 6 presents examples of natural-language feedback, with task prompts omitted for simplicity.
  • B Natural Language Feedback: Training explored natural-language feedback with greater semantic meaning than the simple positive tokens used during inference.
  • B Natural Language Feedback: The natural-language feedback is tailored to each task.
  • B Natural Language Feedback: The natural-language feedback offers increased diversity during training.

C Hyperparameters

Training uses specified optimizer settings, batch sizes, and regularization while reporting labeling interfaces for dialogue and summary evaluation.

  • Adam training uses β1 = 0.9, β2 = 0.95, and epsilon 1.0e−8.
  • The human-feedback batch size is 512, while the pretraining-data batch size is 2048.
  • The value of λ is 1.5, setting the relative gradient strength of human-feedback and pretraining datasets.
  • Pretraining regularization uses the Pile dataset, and dropout is not used because random past-token masking is applied.
  • Figure 7 shows the dialogue-rating interface, where labelers choose a preferred dialogue for each metric.
  • Figure 8 shows the summary-rating interface, where labelers choose a preferred summary for each metric.

D Human evaluation web interface

The web interface lets labelers choose a preferred model output or mark a tie when two outputs appear similar in quality.

  • Labelers can choose the preferred model output or select tie when two outputs seem similar in quality.

E.1 Evaluation on Controllable Generation

CoH supports controllable generation by conditioning improved-summary instructions on earlier outputs, while RLHF does not follow these iterative instructions as effectively. Across few-shot benchmarks, CoH showed moderate improvements over pretrained and supervised fine-tuned models, suggesting less susceptibility to alignment tax.

  • Controllable generation: CoH follows successive instructions to generate improved summaries conditioned on previous model outputs, unlike the RLHF comparison model.The sequence begins with a standard-summary instruction, followed by two instructions requesting improved summaries.
  • Evaluation setup: The controllable-generation evaluation compares CoH and RLHF using three instructions that progressively request higher-quality summaries.The first instruction requests a standard summary; the second and third request improvements based on earlier summaries.
  • Few-shot evaluation: CoH showed moderate improvements over both pretrained and supervised fine-tuned models on diverse few-shot tasks.The evaluation used commonly studied few-shot NLP tasks and the Language Model Evaluation Harness.
  • Alignment tax: Supervised fine-tuning reduced average few-shot performance after alignment, whereas CoH appeared less susceptible to alignment tax.Table 7 reports results averaged over five random seeds.
  • Qualitative examples: Qualitative summarization examples compare GPT-J with CoH-tuned GPT-J on validation data derived from the TL;DR Reddit dataset.The examples are presented in Tables 8 and 9.
Loading 2302.02676v8…