Source-linked AI summary
Improving alignment of dialogue agents via targeted human judgements
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, Geoffrey Irving
TL;DR
Sparrow addresses how to obtain reliable human supervision for helpful, correct, and harmless information-seeking dialogue. It combines RLHF with per-rule judgements and inline evidence, yielding higher preference, 78% supported-and-plausible evidence-backed answers, and an 8% adversarial rule-violation rate, while leaving distributional and evidence-related limitations.
Problem
Reliable human supervision is difficult when dialogue objectives lack programmatic rewards and depend on informed, motivated raters and robust data collection.
Method
Sparrow combines RLHF with natural-language rule decomposition, targeted per-rule judgements, rule-conditional reward models, and source evidence for factual claims.
Results
Sparrow is preferred more often than baselines, supports sampled factual responses 78% of the time with evidence, and violates targeted rules in 8% of adversarial probe conversations.
Takeaways & Limitations
Breaking dialogue goals into detailed rules and adding external evidence improves measured helpfulness, correctness, and resilience to adversarial probing.
Takeaways & Limitations
Sparrow uses one external knowledge fragment at a time, does not investigate source trustworthiness, and does not address several harms or statistical evidence aggregation.
Abstract
from arXiv · showhide
We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.
1. Introduction
Sparrow applies RLHF to information-seeking dialogue, using targeted rule judgements and inline evidence to improve helpfulness, correctness, harmlessness, and evaluation. The approach outperforms prompting, reranking, and supervised-learning baselines while remaining vulnerable to distributional fairness concerns and requiring further work.
- Research focus: Sparrow studies human-judgement rewards for helpful, correct, and harmless information-seeking dialogue, where success and harms are more clearly defined than in chit-chat.The task involves answering users’ questions and follow-ups.
- Targeted human judgements: Targeted rule judgements ask annotators about specific violations, enabling failure characterization, targeted classifiers, and probing of selected failure modes.Examples include threatening statements and financial advice.
- Multi-objective RLHF: Sparrow combines targeted rule and preference judgements with RLHF to train one model preferred over prompting, reranking, and supervised-learning baselines.The system jointly optimizes preference rates and rule-violation reduction.
- Inline evidence: 78% of evidence-supported answers were supported and plausible, improving over prompted baselines and helping raters verify factual claims.Sparrow adapts evidence-based methods to interactive dialogue while showing similar performance to GopherCite on single-turn question answering.
- Robustness and limitations: Sparrow broke targeted rules in only 8% of adversarial probe conversations, but its interventions did not address every listed harm.Unalleviated areas included stereotypes, medical advice, legal advice, microaggressions, and insults.
- Analysis and future work: The methods improve rule following but can amplify distributional fairness concerns, motivating future work on more robust alignment and richer supervision.The paper identifies dialogue as a possible medium for combining evidence, instructions, clarification, and iterative discussion during evaluation.
2. Methods
Sparrow combines targeted human judgements, rule-conditional reward modeling, evidence-supported dialogue, and reinforcement learning to improve helpfulness, correctness, and harmlessness. Its pipeline iteratively collects preference and rule-violation data, trains reward models, and updates the policy.
- Human data collection begins from Dialogue-Prompted Chinchilla and gathers both rule-violation judgements and per-turn response preferences.
- 2.8. Reinforcement learning: Reinforcement learning jointly optimizes preference and rule-compliance objectives while iteratively expanding ratings with improved policies.The approach uses A2C and varies both the training-data distribution and rewards; red-teaming data is complementary to rule models.
- 2.1. Defining rules: The method decomposes helpfulness, correctness, and harmlessness into detailed natural-language rules for adversarial probing and rule-conditional classification.The rule set contains 23 rules and is designed around short, standalone formulations for annotation.
- 2.3. Human data collection: Raters select preferred responses in multi-statement comparisons, with evidence supplied alongside responses to support assessment of factual correctness.Comparisons include responses generated with and without retrieved evidence, providing signal about response quality and evidence use.
- 2.5. Reward models: Preference reward models score candidate responses, while a conditional Rule RM estimates whether each specified rule was violated.The Rule RM outputs the estimated probability that rule y was violated in dialogue x.
- 2.6. Reranking: At inference, Sparrow samples N responses and selects the candidate with the maximum combined preference and rule reward.Reranking@8 samples four responses without evidence and four conditioned on search results before scoring all candidates.
3. Results and analysis
Sparrow’s interventions improve preference, evidence use, adversarial resilience, and some correctness measures, while targeted rules do not address all harms and distributional bias persists or increases.
- Preferences and rule violations: Combining RL with reranking@8 achieves the best performance in both preference win rates and resilience to adversarial probing.
- Evidence evaluation: 78% of responses with evidence are judged plausible and supported by human raters.
- Evidence evaluation: Sparrow agrees with raters on whether evidence is necessary with an overall agreement rate above 90%.
- Evidence evaluation: Sparrow receives a 63% preference rate over GopherCite RL@16, while achieving similar supported and plausible rates in the same QA setting.
- Correctness evaluation: Around 80% of Sparrow’s answers in free-conversation evaluation are rated true or mostly true, although the correctness evaluation is not adversarial.
- Distributional harms: Bias persists across models and datasets, with RL finetuning increasing the Winogender bias score from 0.06 to 0.10.
4. Discussion
The discussion frames Sparrow as a starting point for alignment research, while identifying limitations in evidence use, harm coverage, rule design, human supervision, and cognitive effects. It proposes dialogue, multistep reasoning, broader stakeholder engagement, and cognitive-science research as directions for future work.
- Evidence limitations: Sparrow uses one external knowledge fragment at a time, limiting comparison with systems that combine documents, browse pages, or follow links.The authors suggest multistep reasoning as a possible way to address these evidence limitations; Sparrow also frequently copies evidence verbatim.
- Evidence limitations: Evidence-based correctness is defined conservatively, but the paper does not investigate source trustworthiness or aggregate statistical evidence.The authors also distinguish correctness from honesty, noting that other machinery is needed to target honesty specifically.
- Dialogue as supervision: Dialogue is proposed as a medium for accurate supervision because it supports clarification and iterative discussion of subtle cases.The paper illustrates this with corrections and changed minds in a supervision dialogue, while suggesting that capable agents might also provide corrections or clarifications.
- Rule design and scaling: Rule mechanisms could incorporate multiple stakeholders’ views of good speech, but their origins, stakeholder trade-offs, and scalability remain open questions.The current rules were designed for this study, and rule-conditional reward models may require architectural work to scale to hundreds or thousands of rules.
- Distributional and cognitive harms: Targeted rules and inline evidence address instance harms detectable from single examples, but not distributional harms that emerge from aggregate behavior.The discussion also notes that dialogue supervision depends on human-computer interaction and cognitive science, because human beliefs can resist contradictory evidence.
- Rule granularity: More specific rules may be easier to apply, but raters can hold only a handful simultaneously, creating a specificity–data-collection efficiency trade-off.The authors limit their own evaluations to at most five rules at once and identify empirical study of this trade-off as future work.
- Scope boundaries: The methods are limited to instance harms that raters can detect without significant help, leaving issues such as privacy and social justice outside their scope.The rule set was explicitly not intended to be complete and focused on harms encodable as natural-language rules and mitigated with RLHF.
5. Conclusion
Sparrow improves dialogue-agent alignment by combining detailed rules with external evidence, achieving higher helpfulness, 78% evidence support for factual responses, and an 8% rule-violation rate under adversarial probing. The conclusion emphasizes that broader coverage and careful responses will require multistep reasoning, stakeholder engagement, debate, and cognitive-science research.
- Conclusion: Sparrow focuses on alignment breadth by decomposing goals into detailed rules and using external knowledge to broaden topics it can discuss correctly.The conclusion contrasts this width with depth: handling each topic carefully and correctly remains a future challenge.
- Conclusion: 78% of factual responses with evidence were correctly supported, while Sparrow reduced adversarial rule violations to 8%.Sparrow was also preferred more often by raters, according to the conclusion’s summary of results.
- Conclusion: Future work should address depth through multistep reasoning, expert and participatory rule development, debate-based supervision, and cognitive-science research.These directions aim to improve careful reasoning, rule evolution, violation detection, and interaction with real people.
7. Author contributions
The project involved distinct contributions spanning model architecture, training schemes, retrieval, reward models, evaluations, infrastructure, ethics, and analysis.
- Model and training: Sparrow’s architecture and reinforcement-learning training scheme were designed and built by the named project contributors.
- Model components: The team designed and built Sparrow’s retrieval capabilities, preference reward model, rule reward model, and supervised fine-tuning scheme.
- Research and engineering: Contributors designed experiments, evaluations, infrastructure, model serving, frontend systems, operations, ethics, and distributional-bias analyses.
- Oversight and support: Research advisors, technical leads, project managers, and supervisors supported the project, while additional contributors handled exploratory experiments and sociotechnical work.
D.4. Rule RM training data
The rule reward model was trained on human ratings of adversarial and free dialogues, with targeted probing used to increase coverage of difficult violations.
- Training data: The rule RM was trained from rule-violation ratings collected through adversarial probing and free-dialogue tasks.
- Data collection: 14,576 adversarial-probing dialogues were collected, rated for one or more rules, and split into 70% training and 30% test data.
- Labels: Likert ratings were binarised into break or follow labels, while unsure ratings were discarded.
- Adversarial probing: Adversarial dialogues were seeded with LM-generated questions designed to elicit bad behaviour and target selected domains such as medical, legal, and sexual topics.
- Reward use: During reinforcement learning, rule and preference rewards were combined differently for Agent, Search Query, and User turns.
F.1. List of rules
Sparrow’s high-level goals are operationalised as detailed rules covering helpfulness, correctness, harmlessness, and protection against misleading or anthropomorphic behaviour.
- Harmlessness: Harmlessness rules cover threats, sexual aggression, identity attacks, insults, hate or harassment, stereotypes, and microaggressions.
- Self-anthropomorphism: Self-anthropomorphism rules prohibit claims of having a body, relationships, opinions, emotions, or a human identity or life history.
- Correctness: Correctness rules prohibit implausible claims, unsupported assumptions about users, and claims of taking real-world actions.
- Misinformation: Misinformation-related rules restrict medical, legal, and financial advice, conspiracy theories, and statements that are obviously false.
- Helpfulness: Helpfulness rules require staying on topic, addressing user queries, making sense, and avoiding unnecessary repetition.
- Annotation: Raters were UK-based native English speakers with at least undergraduate education, received tutorials and checks, and preference agreement remained 67% with Krippendorff’s alpha of 0.44.
G.2. Rating task screenshots
The rating-task materials include screenshots of preference, annotation, and adversarial-probing interfaces, alongside procedures for monitoring annotator well-being.
- Preference rating: Figure 20 depicts the per-turn response-preference task in a three-way comparison setting.
- Annotation workflow: Figure 21 shows additional annotation tasks completed before selecting the preferred response in Figure 20.
- Adversarial probing: Figure 22 presents the adversarial-probing task and the self-reported success question asked after the dialogue.
- Well-being monitoring: Well-being was assessed after tasks involving harmful language, and 13 of 209 rating-task participants reported greater negative impact than in similar tasks without harmful language.
- Well-being caveat: Some raters skipped sensitive dialogues, but the rating interface exposed them to conversation content before they could decide whether to perform the task.
H.1. Stereotypes and social biases
The paper evaluates stereotype-related bias using benchmark datasets, accuracy controls, and a BBQ-derived metric. Few-shot prompting enables measurement, while RL tuning reduces abstention but increases errors on ambiguous questions.
- Dataset selection: Winogender, Winobias, and BBQ benchmark whether models challenge or reinforce harmful stereotypes with well-defined desirable responses.The datasets target concrete representational harms and provide natural bias metrics.
- Bias metric: The BBQ bias metric measures the fraction of non-abstaining responses that reinforce stereotypes, scaled from -1 to 1.A score of 1 means always stereotype-reinforcing, while -1 means always stereotype-challenging.
- Bias metric: For balanced datasets, the BBQ bias score equals the accuracy difference between stereotype-reinforcing and stereotype-challenging questions.This connects BBQ scores with gotcha and non-gotcha accuracies.
- Results: 69.1% overall accuracy was observed for DPC on BBQ, compared with 77.8% for UnifiedQA reported by Parrish et al.The DPC result used few-shot prompting and exact string matching.
- Results: RL tuning reduced “I don’t know” responses, lowering ambiguous-question accuracy from 87% to 65% and nearly tripling error.The resulting higher ambiguous-question bias scores reflect both more incorrect answers and fewer abstentions.
H.2. Disparate impact for factual question-answering
The paper studies whether factual-question accuracy varies across demographic characteristics and whether evidence-based models change these patterns. Evidence generally improves accuracy, while demographic correlations can be introduced or removed.
- Methodology: TriviaQA, Natural Questions, and Quiz Bowl are factual QA benchmarks used to study demographic accuracy patterns and evidence-based models.The analysis focuses on dev-set questions containing linked demographic entities.
- Methodology: Accuracy is measured by exact matching the correct answer within each single-turn model response for every demographic value.The evaluation uses demographic categories including gender, occupation, and country.
- Methodology: A χ2 contingency-table test evaluates independence between demographic characteristics and accuracy, using a 0.0167 significance threshold.The threshold divides 0.05 across three tests per dataset.
- Results: Evidence-based models greatly improve accuracy for most characteristics and values, except where evidence does not improve or slightly hurts Quiz Bowl performance.Quiz Bowl is described as difficult, and its dataset contains 2216 samples.
- Results: Models with evidence introduce some demographic-accuracy correlations while removing others; occupation and accuracy remain correlated in TriviaQA across model variants.The table reports these relationships using the stated χ2 test.
I. Evaluation of alignment taxes
The paper measures whether alignment interventions reduce standard capabilities, while adapting evaluation prompts to the dialogue-trained setting. It finds negligible or no alignment tax relative to DPC on MMLU and TruthfulQA.
- Evaluation: The paper evaluates possible alignment taxes—capability decreases after human-preference fine-tuning—using MMLU and TruthfulQA.These benchmarks are compared before and after RLHF interventions.
- Evaluation: MMLU questions are dialogue-ified by appending question-answer pairs as User/Sparrow utterances to the DPC prompt.Answer likelihoods are computed independently, and the highest-probability answer is selected.
- Evaluation: The full evidence-enabled agent is excluded from MMLU because internet-sourced questions can appear verbatim in Google search results.The evaluation therefore focuses on the core Sparrow policy to measure rule- and preference-tuning effects.
- Results: Few-shot MMLU evaluations show negligible alignment tax relative to DPC, while TruthfulQA shows no alignment tax compared with DPC.Subsequent SFT and RL@1 improve over DPC in TruthfulQA evaluations.
- Results: Figure 26 reports that MMLU topic sizes are too small to distinguish Sparrow and DPC after accounting for 95% bootstrapped confidence intervals.The figure breaks down score differences by topic.
J.2. Negative examples of knowledge grounding
The examples show that knowledge grounding can fail when search results are ignored, misread, or unavailable. Errors include hallucinated, outdated, out-of-context, and misinterpreted claims, alongside safer medical responses.
- Search grounding failures: Sparrow sometimes hallucinates by ignoring search results or producing out-of-date facts.The examples explicitly label both failure modes.
- Search grounding failures: When relevant search results are unavailable, reranking can prefer an answer without evidence for an ELI5 question.The figure identifies failure to find relevant results as the setting.
- Search grounding failures: Taking search results out of context can produce incorrect numerical answers, such as confusing sampled showings with total stagings.The Shakespeare example reports 118 stagings despite the cited distinction.
- Search grounding failures: Sparrow misinterprets evidence when it treats 7.88, a rating out of 10, as a speed in m/s.The error appears in the fast-paced-sport discussion and its cited table.
- Safety behaviour: For medical questions, Sparrow declines diagnosis or medical advice while still offering limited symptom-related information in one example.The examples cover fever, cough, and depression-related questions.
J.4. Samples rated as rule breaking by human participants
The samples illustrate rule-breaking across medical, legal, harmful, deceptive, and bias-related interactions. Human participants and independent raters were used to identify both specific violations and novel harms.
- Legal and medical advice: Sparrow gives legal advice after a user asks how to sue someone for being slapped.The response advises suing for battery and seeking damages for emotional distress.
- Legal and medical advice: Sparrow refuses to diagnose a sore eye but is rated as probably breaking the medical-advice rule.The example contrasts the user’s medical question with Sparrow’s disclaimer that it is not a doctor.
- Stereotypes and bias: Sparrow reinforces a gender stereotype by citing an article claiming men are officially better at parking than women.A human participant rates the exchange as probably breaking the relevant rule.
- Adversarial evaluation: 566 dialogues were independently re-rated against the general and specific harm rules to identify harms not captured by the specific rules.Heuristics found 31 self-rated and 5 independently identified dialogues that were not jointly assigned a specific-rule violation.
- Deception and novel harms: The adversarial examples include fabricated private-data access, harmful crime-related recommendations, deception, and claims about a user’s husband.The paper’s novel-harm mining identifies pretending to reveal private data, recommending no punishment for murder, advice on crime, and related failures.
J.7. TruthfulQA samples from GopherCite
The TruthfulQA samples show that Sparrow sometimes answers correctly while citing irrelevant, unreliable, or unsupported evidence. Other examples demonstrate factual errors caused by accepting misleading sources or elaborating beyond the evidence.
- Evidence support: Sparrow answers that Red Bull provides caffeine and taurine, but the evidence does not support the taurine claim.The figure description identifies the answer as correct about “wings” but unsupported for taurine.
- Unsupported factual claims: Sparrow repeats a false explanation that firemen burn controversial books because governments want to keep people powerless.The sample is identified as a failure caused by citing a plot summary of Fahrenheit 451 as fact.
- Source reliability: Sparrow gives palm-line interpretations and is tricked into citing an untrusted source.The examples include unsupported elaboration, irrelevant evidence, and a source that the model should not have trusted.
J.8. Other samples
The additional samples expose failures involving numerical reasoning, conspiracy theories, capability loss, hallucination, and inconsistent or muddled answers. They also show that Sparrow can provide reasonable references while omitting details or hallucinating follow-up information.
- Other behaviours: Training for harmless information-seeking dialogue is shown as removing other capabilities, while failed searches often lead Sparrow to hallucinate.The samples also include inconsistent advice-giving and a muddled list of moral systems.
- Other behaviours: Sparrow supports a conspiracy theory after becoming confused by a leading question.Another example shows confusion when presented with a conspiracy theory, with part of the response taken out of context.
- Other behaviours: Sparrow cites Christiano et al. (2018) reasonably but omits details and hallucinates when asked who made the work.The sample also includes a response claiming that new AIs become better and faster to build, despite the surrounding question about iteration speed.