PaperlayerView exact revision on arXiv

Source-linked AI summary

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi

arXiv:2306.01693v2cs.CL

TL;DR

Holistic RLHF gives limited, sparse information about which parts of long outputs contain specific errors. The paper introduces Fine-Grained RLHF, which trains and combines dense, category-specific reward models; experiments on detoxification and long-form QA report improved performance and customizable behaviors.

  • Problem

    Holistic RLHF feedback is sparse for long outputs and does not localize which output parts contain which error types.

  • Method

    Fine-Grained RLHF trains separate reward models for error categories, provides rewards at associated segment densities, and integrates them into PPO.

  • Results

    Experiments on detoxification and long-form QA show improved performance with fine-grained rewards under automatic and human evaluation, including gains over holistic-reward RLHF.

  • Takeaways & Limitations

    Different combinations and weights of fine-grained reward models can customize language-model behavior for specific desired needs.

  • Takeaways & Limitations

    Fine-grained rewards require additional computation and task-specific manual effort to define feedback types, densities, and reward models.

Abstract

from arXiv · show

Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at https://FineGrainedRLHF.github.io.

1 Introduction

The paper addresses the limited, sparse information in holistic RLHF feedback by training language models with feedback localized to error spans and differentiated by error type. Fine-Grained RLHF applies these signals to detoxification and long-form question answering, improving error-category performance and enabling customized behaviors.

  • Motivation: Holistic RLHF provides sparse feedback on long outputs and does not identify which parts contain specific errors.This makes it difficult to use user judgments about mixed behaviors as a localized training signal.
  • Approach: Fine-Grained RLHF associates each undesired-behavior category with a text span at a chosen density, such as sentence or sub-sentence level.The framework trains a separate reward model for each category and integrates the models into PPO.
  • Experiments: Detoxification experiments use a toxicity reward after every generated sentence and show efficacy and data efficiency compared with a holistic sequence-level reward.The toxicity reward model is PERSPECTIVE, trained on millions of human annotations.
  • Experiments: Long-form QA experiments examine fine-grained rewards across irrelevance, repetition, incoherence, factuality, and information completeness at different densities.The authors construct the QA-FEEDBACK dataset and collect corresponding human feedback.
  • Findings: Across the experiments, fine-grained feedback improves each targeted error category under automatic and human evaluation and supports customized behavior combinations.Different reward-model combinations can be weighted to steer training toward desired behaviors.

2 FINE-GRAINED RLHF

Fine-Grained RLHF replaces a single sequence-level reward with dense, category-specific rewards and combines them within PPO. Rewards are assigned to the relevant generated segments, while a KL penalty supports fluency during optimization.

  • Framework: Fine-Grained RLHF trains reward functions for different feedback types and incorporates one or multiple fine-grained reward models into an RL algorithm.This contrasts with prior RLHF formulations using a single holistic reward.
  • Environment: Language generation is modeled as an MDP in which a policy generates tokens until an end condition and receives rewards before or at sequence completion.The reward functions are supplied by the fine-grained reward models.
  • Learning algorithm: PPO optimizes the policy and value models against rewards acquired from one or multiple learned reward models.The reward models are not optimized during PPO training.
  • Fine-grained rewards: A holistic reward assigns one scalar only to the final token, whereas fine-grained rewards are provided densely for subsequences and distinct undesired-behavior categories.Each category is associated with an individual reward model.
  • Fine-grained rewards: Each fine-grained reward model segments the generated output at its associated density and assigns its segment reward to the segment’s final token.The combined token reward weights the category-specific rewards and includes an approximate KL divergence penalty with weight β.
  • Reward combination: The combined reward uses weights w_k for the K category-specific reward models before PPO optimization.The weights determine how the individual reward models contribute to each token’s reward.

3 Task 1: Detoxification

The detoxification task reduces toxicity in model generations, and this study compares sentence-level fine-grained rewards with holistic rewards. Fine-Grained RLHF achieves lower toxicity and perplexity while preserving similar diversity, and reduces toxicity faster during training.

  • Detoxification aims to reduce toxicity in model generations, using toxicity as the sole undesired behavior.
  • The experiments use REALTOXICITYPROMPTS, a 100K-prompt dataset designed to elicit problematic GPT-2 generations.
  • Sentence-level rewards query PERSPECTIVE after each sentence and assign reward according to the change in toxicity from generating that sentence.
  • FINE-GRAINED RLHF attains the lowest toxicity and perplexity among compared methods while maintaining similar diversity.
  • FINE-GRAINED RLHF reduces toxicity faster while keeping perplexity low, indicating greater sample efficiency than holistic reward training.
  • The stronger localized training signal requires more reward-model queries per example.

4 Task 2: Long-Form Question Answering (QA)

Long-form QA uses QA-FEEDBACK to train and evaluate fine-grained rewards at sub-sentence, sentence, and sequence densities. Fine-Grained RLHF improves error-related outcomes over comparison systems, supports controllable behavior, and reveals competition among reward objectives.

  • 4.1 QA-FEEDBACK: Long Form QA with Human Feedback: QA-FEEDBACK reformulates ASQA into reading comprehension with questions, oracle knowledge passages, and long-form responses averaging roughly 65 words.The dataset targets ambiguous factoid questions in an open-domain setting.
  • 4.1 QA-FEEDBACK: Long Form QA with Human Feedback: Workers mark spans for irrelevance, repetition, incoherence, incorrect or unverifiable facts, and incomplete information in model outputs.The feedback records each error category together with its annotated text span.
  • 4.1 QA-FEEDBACK: Long Form QA with Human Feedback: Human agreement determines reward density: sub-sentence for C1, sentence for C2, and full sequence for C3.Exact span-boundary agreement is low, while agreement on whether units contain C1 or C2 is reasonably high.
  • 4.4 Main Results: Fine-Grained RLHF outperforms SFT and Preference RLHF on all evaluated error types in human evaluation.It produces more factual and complete responses and fewer irrelevance, repetition, and incoherence errors.
  • 4.4 Main Results: FINE-GRAINED RLHF was rated better than Preference RLHF in 30.5% of examples, worse in 24.5%, and tied in 45%.These ratings compare overall generation quality between the two systems.
  • 4.4 Main Results: RLHF reduces factual errors more reliably than irrelevance, repetition, or incoherence errors in the reported human evaluation.Both Fine-Grained RLHF and Preference RLHF reduce factual errors, while the other error types show little or no improvement.
  • 4.5 LM Customization with FINE-GRAINED RLHF: Changing reward-model weights produces distinct generation styles: higher relevance weight yields shorter, more relevant but less factual and complete outputs.The contrasting long configuration gives the most factual and complete generations in manual inspection.
  • 4.6 Analysis: During training, relevance and completeness rewards move in opposite directions before reaching equilibrium, while factual reward consistently increases.The authors interpret relevance precision and information-completeness recall as potentially adversarial objectives.

5 Related Work

Related work has used RLHF to improve language-model behavior and has also explored refinement, reward-guided generation, and incorporating feedback during pre-training.

  • RLHF trains language models toward content desired by humans across tasks including summarization, instruction following, question answering, and harmfulness reduction.
  • Refinement-based methods use natural-language feedback to revise outputs, then select refined responses for supervised fine-tuning.
  • Inference-time reward guidance complements this work, which aims to improve language models during training.
  • Prior work has also investigated incorporating human feedback into language-model pre-training.

6 Discussion

The framework offers dense, multi-reward guidance, flexibility, and user-controlled behavior customization, while requiring additional computation, task-specific design effort, and cleaner feedback than deployment may provide.

  • Discussion: For long-form QA, fine-grained and preference-based feedback each take approximately 6 minutes per annotated sample.
  • Broader Impacts: Fine-grained RLHF incorporates multiple reward models that provide dense rewards, optimizing language-model outputs toward selected rewards.
  • Broader Impacts: The framework can incorporate reward models for fact-checking, sentiment classification, toxicity detection, and other feedback types.
  • Broader Impacts: Different reward-model combinations let users control training and customize behaviors for applications such as personalized educational tools.
  • Limitations and Future Work: Fine-grained rewards require more compute than holistic rewards, including multiple toxicity-detector API calls or greater GPU memory and computation.
  • Limitations and Future Work: Feedback types and density levels vary across tasks, so designing suitable feedback and reward models requires non-trivial manual effort.
  • Limitations and Future Work: The study controls annotation quality, leaving effective learning from noisy human feedback in the wild for future investigation.
  • Limitations and Future Work: Future questions include obtaining feedback from language models, combining non-reinforcement-learning approaches, and studying reward- and value-model sizes.

7 Conclusion

The paper introduces FINE-GRAINED RLHF, evaluates it on two text-generation tasks, and reports performance gains over holistic-reward RLHF alongside customizable model behavior.

  • FINE-GRAINED RLHF trains language models with multiple human-feedback reward models, each targeting an error category and providing dense rewards.
  • Experiments on detoxification and long-form question answering show performance gains over holistic-reward RLHF using automatic and human evaluation.
  • Different combinations of fine-grained reward models customize language-model behavior for specific needs.

Appendices

The appendices provide qualitative examples of customization and errors, compare ChatGPT with the proposed system, and document the long-form QA input example.

  • LM customization: Reward-model weight configurations named short, medium, and long produce concise, intermediate, and detailed response behaviors.
  • Tables: The customization examples are presented in Table 6.
  • Tables: Another table presents examples of C1 and C2 errors in outputs from supervised fine-tuning, preference-based RLHF, and FINE-GRAINED RLHF.
  • QA example: The QA example asks where the flat iron cut of meat comes from and supplies an article describing its origin in the animal’s shoulder.
  • ChatGPT comparison: ChatGPT responses are described as relevant and factual but lacking auxiliary information for ambiguous questions.
  • ChatGPT comparison: The ChatGPT comparison uses one in-context example because each input is too long to fit more than one.
  • Tables: The appendix includes a qualitative comparison between ChatGPT outputs and FINE-GRAINED RLHF outputs.

A.3 Examples on Reward Model Ablation

The reward-model ablation examines how language-model behavior changes when one of the three fine-grained reward models is removed during reinforcement-learning training. Table 9 provides qualitative output examples comparing the full reward combination with these ablations.

  • Removing one reward model changes the language model’s behavior during reinforcement-learning training.The paper describes the reward models as competing against each other and studies ablations by removing one model.
  • Table 9 compares outputs trained with all reward models against outputs trained without one reward model.The table marks errors in each qualitative output example.
  • Fine-Grained RLHF uses PPO to update both the policy language model and the value model from K fine-grained reward models.The algorithm computes rewards and advantages before updating the policy and value models.

B.2 Implementation Details

The implementation uses task-specific policy and value-model initializations, separate training settings for detoxification and long-form QA, and human-feedback collection procedures for both fine-grained and preference-based supervision.

  • Model architectures: Detoxification initializes GPT2large as the policy model and GPT2-base as the value model, while long-form QA uses T5-large and T5-base.The larger policy and smaller value-model design follows InstructGPT’s architecture pattern.
  • Training details: Long-form QA uses 1024-token inputs and 200-token outputs, with smaller batches and fewer training episodes than detoxification.The longer output setting motivates reduced batch size and training duration.
  • Training details: The selected long-form QA reward weights are w1 = 0.3, w2 = 0.5, and w3 = 0.3.The authors choose these weights to balance the three reward models and allow all three rewards to increase during training.
  • Fine-grained feedback: Human feedback is collected from sampled outputs after training an initial SFT policy on 1K randomly sampled training examples.Feedback covers the remaining 2,853 training examples and 500 development examples through Amazon Mechanical Turk.
  • Feedback collection: The study collects both fine-grained annotations and pairwise preferences from the same qualified worker group.Workers annotate error spans for fine-grained feedback and compare pairs of model outputs for preference-based feedback.
  • Pay structure: Workers receive an average of roughly $1.65 per example, corresponding to an $16.5 hourly pay rate.A base rate of $1.5 is supplemented by a $0.3 bonus when an example contains at least three passages.

C.3 Analysis of Collected Fine-Grained Feedback

The collected feedback is distributed across several error categories, and workers show substantially higher agreement on category presence than on exact error-span boundaries.

  • 76% of collected error spans are C1 errors, while the remaining 24% are C2 errors.Workers label C2 only where no C1 error is present; 75% of examples are marked incomplete under C3.
  • Workers reach 83% agreement for C1 at the sub-sentence level, 92% for C2 at the sentence level, and 85% for C3 completeness.Exact agreement on error-span boundaries is low, but category-level agreement is reasonably high.

D Long-Form QA Reward Model Training Details

Long-form QA reward models are trained separately for sub-sentence errors, sentence-level factual errors, incomplete information, and pairwise preferences, using labels or comparisons tailored to each feedback type.

  • Reward-model training: The three category-specific reward models are selected using development performance after training on 2,835 feedback examples.Rϕ1 and Rϕ2 use classification training, whereas Rϕ3 uses pairwise comparison training.
  • Rϕ1 for C1: Rϕ1 detects irrelevance, repetition, or incoherence with token-level classification at sub-sentence boundaries.Short error spans under five words are filtered before constructing Rϕ1 training examples.
  • Rϕ2 for C2: Rϕ2 predicts incorrect or unverifiable facts at the sentence level and excludes sentences containing only C1 errors from its loss.The resulting dataset contains 1,600 “has error” and 3,411 “no error” sentences.
  • Rϕ3 for C3: Rϕ3 predicts a scalar completeness reward trained from pairwise comparisons ordered by the fraction of mapped passage sentences retained.The authors construct 6,821 training pairs by comparing sampled responses with different information-completeness scores.
  • Preference-based reward model: The preference-based reward model is trained on 14,981 non-tied response pairs using the same general pairwise-training approach.Tied preference pairs are dropped before training.
  • Completeness-score construction: Over 90% of manually reviewed passage-sentence mappings are correct, indicating frequent extractive behavior from the initial policy.The review covers 50 mapped passage sentences.
Loading 2306.01693v2…