Source-linked AI summary
Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
Patrick Fernandes, Aman Madaan, Emmy Liu, António Farinhas, Pedro Henrique Martins, Amanda Bertsch, José G. C. de Souza, Shuyan Zhou, Tongshuang Wu, Graham Neubig, André F. T. Martins
TL;DR
Internet-trained language models can generate toxic, inaccurate, and unhelpful content, while automatic metrics often fail to capture human-perceived quality. This survey formalizes and organizes research on human feedback for natural language generation, covering feedback formats, uses, datasets, collection concerns, and AI feedback. It concludes that feedback itself may matter more than the specific leveraging method, while comprehensive large-scale comparisons remain lacking.
Problem
Internet-scale training and automatic evaluation can miss desired behavior and human-perceived quality, motivating human feedback for evaluating and improving natural language generation.
Method
The survey formalizes human feedback, develops a taxonomy, reviews direct and feedback-model approaches for training or decoding, examines datasets and collection concerns, and surveys AI feedback.
Results
The survey finds that many feedback formats are underleveraged, natural-language feedback is promising, and simpler feedback-leveraging methods may outperform RLHF, although large-scale comparisons remain lacking.
Takeaways & Limitations
The survey indicates that research should look beyond ranking and numerical feedback and beyond RLHF when leveraging feedback to improve language generation.
Takeaways & Limitations
The survey’s formalization simplifies human disagreement and excludes active learning and other human-in-the-loop approaches; feedback-model optimization can also overoptimize imperfect proxies.
Abstract
from arXiv · showhide
Many recent advances in natural language generation have been fueled by training large language models on internet-scale data. However, this paradigm can lead to models that generate toxic, inaccurate, and unhelpful content, and automatic evaluation metrics often fail to identify these behaviors. As models become more capable, human feedback is an invaluable signal for evaluating and improving models. This survey aims to provide an overview of the recent research that has leveraged human feedback to improve natural language generation. First, we introduce an encompassing formalization of feedback, and identify and organize existing research into a taxonomy following this formalization. Next, we discuss how feedback can be described by its format and objective, and cover the two approaches proposed to use feedback (either for training or decoding): directly using the feedback or training feedback models. We also discuss existing datasets for human-feedback data collection, and concerns surrounding feedback collection. Finally, we provide an overview of the nascent field of AI feedback, which exploits large language models to make judgments based on a set of principles and minimize the need for human intervention.
1 Introduction
Internet-trained language models can produce toxic, inaccurate, or unhelpful text, while automatic metrics often miss human-perceived quality. This survey organizes research using human feedback to evaluate and improve natural language generation.
- 1 Introduction: Internet-scale training can produce toxic, inaccurate, and unhelpful content because scraped data contains noise, biases, and errors and MLE may miss target behavior.Automatic metrics based on surface features often fail to correlate with human-perceived quality, especially when optimized.
- 1 Introduction: Human feedback helps assess generated-text quality, address Goodhart’s law, and steer systems toward desired outcomes.The survey connects this research direction with the emergence of widely used general-purpose language assistants.
- 1 Introduction: The survey formalizes human feedback and presents a taxonomy organized around its format, objective, use during training or decoding, and modeling approach.It distinguishes direct use of feedback from surrogate feedback models that approximate human preferences.
- 1 Introduction: The survey reviews direct feedback optimization, feedback-model approaches, human-feedback datasets and collection concerns, and emerging AI feedback from language models.These topics span reinforcement learning with human reward functions, proxy-model optimization, data collection, and reduced human intervention.
2 A Taxonomy for Leveraging (Human) Feedback for Generation
The survey models generation as producing text from inputs and classifies human feedback by its format, objective, timing, and modeling strategy. It also notes that this formalization simplifies human disagreement and excludes some human-in-the-loop approaches.
- 2.1 Background: Generated-text evaluation is difficult because automatic metrics often miss paraphrasing, stylistic variation, and other nuances captured by human judgment.For many tasks, human feedback is therefore treated as the gold standard for quality assessment.
- 2.1 Background: The formalization treats a generation model as mapping an input x to natural-language output y, with feedback functions taking inputs and one or more outputs to produce feedback.It covers tasks including summarization, machine translation, dialog generation, and image captioning.
- 2.1 Background: The framing simplifies human disagreement and excludes active learning and other human-in-the-loop approaches.Different humans may provide contradictory feedback for the same output, which a single feedback function may not capture.
- 2.2 Taxonomy: The taxonomy classifies feedback by format, objective, when it is used, and how it is modeled.Formats include binary judgments, numerical scores, ordinal rankings, and natural-language explanations; objectives range from performance assessment to preventing harmful behavior.
- 2.2 Taxonomy: Feedback may optimize model parameters during training, guide decoding at inference, or be approximated by surrogate models when direct collection is costly.The formalization includes simple good-or-bad judgments as well as rankings and natural-language feedback.
3 Describing Feedback
The survey characterizes human feedback by its format and objective, emphasizing trade-offs between expressivity, collection difficulty, consistency, and usefulness for aligning generation systems.
- 3.1 Format: Simpler feedback formats ease collection and system use but may omit information needed to improve generation.Format complexity also affects human effort, agreement, consistency, and the rationality of the resulting feedback.
- 3.1 Format: Feedback formats range from numerical scores and rankings to natural-language explanations, multi-aspect judgments, and post-edits.Numerical and ranking feedback are simpler to collect and use, while natural-language and richer formats can convey more detailed information.
- 3.1 Format: Numerical feedback is easy to leverage but can be costly, subjective, variable, and unable to distinguish outputs of similar quality.Reducing complex judgments to a single score may be difficult and ill-defined for humans.
- 3.2 Objective: The survey frames feedback collection as a means of aligning model behavior with often ill-defined goals such as preserving core information or avoiding harmful outputs.Examples include longer but more complete summaries, protection of business-critical translation content, and polite, harmless dialogue.
- 3.2 Objective: Feedback objectives primarily concern helpfulness or harmlessness, including task performance, instruction following, relevance, consistency, accuracy, toxicity, and safety.The survey notes that helpfulness is necessary but not sufficient for a helpful system, while harmlessness targets norm-violating or harmful outputs.
4 Directly Leveraging Human Feedback
The survey describes direct use of human feedback through supervised learning, joint-feedback modeling, and reinforcement learning, with decoding-time feedback offering adaptation when parameter updates or training-time feedback are unavailable.
- 4.1 Optimizing for Human Feedback: Direct feedback optimization uses three training mechanisms: feedback-based imitation learning, joint-feedback modeling, and reinforcement learning.These mechanisms differ in whether they train on preferred outputs, model feedback jointly, or optimize parameters through feedback-driven rewards.
- 4.1 Optimizing for Human Feedback: Feedback-based imitation learning trains on positively labeled generations, such as human-approved dialogue, translations, or preferred rule-compliant conversations.The approach performs supervised learning using inputs paired with generations judged positively by humans.
- 4.1 Optimizing for Human Feedback: Joint-feedback modeling uses collected inputs, generations, and feedback to train models that predict feedback, including natural-language comments or rankings.Predicting rankings of summaries has been reported to improve generation and potentially outperform more complicated feedback-model approaches.
- 4.1 Optimizing for Human Feedback: Reinforcement learning optimizes model parameters using human feedback without requiring differentiable feedback, often training offline from logged data because online collection is expensive.REINFORCE updates policy parameters from feedback-weighted gradients, and RLHF commonly refers to optimizing against human feedback or learned feedback models.
- 4.2 Leveraging Feedback During Decoding: When parameter modification or training-time feedback is impractical, decoding-time feedback can guide generation through feedback memory and iterative output refinement.These techniques let models use prior feedback or intermediate user responses to adapt outputs without changing model parameters, and they can be combined.
5 Improving Generation using Human Feedback Models
Feedback models approximate human judgments so generation systems can use feedback at lower cost, but imperfect proxies create overoptimization risks. The survey covers their training, use during generation training or decoding, and representative findings.
- Learning Models of Human Feedback: Feedback models approximate human feedback, enabling feedback-dependent generation techniques to scale after training despite imperfect accuracy.They can be trained on explicit human judgments or naturally occurring implicit feedback, though the latter is less accurate.
- Learning Models of Human Feedback: Ranking, numerical, and other human-feedback formats are mapped to numerical feedback models through suitable agreement losses.Preference models can learn from rankings, while numerical feedback can use regression losses.
- Using Feedback Models: Feedback models improve generation either by optimizing model parameters during training or by incorporating feedback during decoding.These approaches include reinforcement learning, joint-feedback modeling, and decoding-based use of feedback models.
- Using Feedback Models: Glaese et al. (2022) found reinforcement learning with a feedback model produced better preference rates and lower rule-violation rates than feedback-based imitation learning.The comparison concerns two feedback-based approaches to improving generation.
- Limitations: Feedback-model optimization can overfit imperfect proxies, yielding outputs rated well by the model but poorly by humans.This overoptimization problem motivates regularization, such as the KL term used to limit deviation from the original supervised model.
- Limitations: Gao et al. (2022) found similar overoptimization levels when preference models were used during training or decoding, with generation-model scale helping little.The result applies to both feedback-model use modes studied in that work.
6 Collecting and Using Human Feedback
The survey reviews how human-feedback datasets are collected and emphasizes that feedback quality, annotator variation, bias, and ethics constrain their use. It also outlines dimensions for designing preference-data collection.
- Open Issues: Human-feedback datasets may underrepresent richer feedback and annotator variability, which remain incompletely explored.Most existing datasets use rankings or numerical scores, while people often prefer richer feedback forms.
- Dataset Collection: Human-feedback collection varies by annotator expertise, engagement length, and other design dimensions, so datasets should be matched carefully to use cases.The survey presents these dimensions alongside existing datasets and collection methods.
- Bias and Reliability: Human feedback may be low-quality, contradictory, or adversarial, making annotation guidelines and agreement assessment important.Different expert groups may disagree, and inter-rater reliability can reveal unclear criteria, subjectivity, or multiple plausible interpretations.
- Bias and Reliability: Even unanimous judgments can be mistaken when annotators overlook evidence or interpret underspecified instructions as a different task.Agreement therefore does not guarantee validity or alignment with the intended objective.
- Bias and Reliability: Anchoring and positivity biases can distort judgments, while ranking diverse outputs and specifying evaluation dimensions are proposed mitigations.Annotators may overrate isolated texts or provide disproportionately positive feedback.
- Ethical Considerations: Feedback collection raises ethical concerns because toxic-content annotation can cause psychological harm and inadequate pay can exploit workers.The survey notes that some annotation tasks may exceed minimal-risk assumptions and reports concerns about very low compensation.
7 AI Feedback
AI feedback uses language models to evaluate and improve outputs, reducing reliance on continual human intervention. The survey distinguishes self-feedback from feedback supplied by a separate model, while noting data and consistency challenges.
- Motivation: AI-generated feedback aims to improve models without constant human intervention, but feedback models still depend heavily on human input.Gao et al. (2022) found fewer than 1,000 comparisons produced only minor improvements, approaching chance.
- Challenges: Static feedback can create consistency and accuracy challenges because integrating feedback changes the model’s output distribution.The survey identifies this as an open issue for AI-feedback systems.
- Self AI Feedback: Self AI feedback uses the same model to evaluate and revise its outputs, including harmlessness revisions and rule-based reward-model fine-tuning.The model can generate harmful responses, critique or revise them, and use principles or rules to guide improvement.
- External AI Feedback: External AI feedback pairs the task model with a separate feedback model, which need not be a large general-purpose model.This setup makes training smaller feedback models possible.
8 Conclusion
The survey finds that human-feedback research is still underdeveloped but already reveals important patterns about feedback formats, optimization methods, and the amount of supervision needed. It also identifies open questions about feedback’s role and encourages researchers to explore broader feedback sources and uses.
- Most feedback-based systems rely on ranking or numerical signals, while natural-language feedback appears promising because its expressiveness can provide useful improvement signals.The survey notes that many feedback formats and datasets remain underleveraged.
- The benefits of human feedback may matter more than the specific method used: multiple approaches outperform nonfeedback baselines, and simpler methods may outperform RLHF in some comparisons.A comprehensive, large-scale comparison across methods is still lacking.
- The field still lacks a clear account of how feedback changes model behavior or how much supervision is necessary.AI feedback suggests that human supervision might be substantially reduced, while other work questions whether feedback is needed when high-quality instruction data is available.
- The survey aims to help researchers understand the current state of the art and identify new and existing feedback sources and ways to use them.