Improving alignment of dialogue agents via targeted human judgements

Introduction

• Sparrow is preferred to prompting, reranking, and supervised-learning baselines and breaks targeted rules in only 8% of adversarial probe conversations.The model combines targeted rule judgements and preference judgements with reinforcement learning from human feedback.Source: §1 · p. 3
• Asking annotators about specific rule violations enables targeted failure analysis, classifiers, and probing of failure modes rather than relying only on broad safe/unsafe or harm labels.The paper combines these rule judgements with preference judgements to train a unified model.Source: §1 · p. 3
• Answers with inline evidence are supported and plausible 78% of the time, significantly improving on prompted baselines and helping raters verify claims.The paper adapts and extends GopherCite methods to interactive dialogue.Source: §1 · p. 3
• Although the methods improve rule following, they can amplify distributional fairness concerns because the mitigations address instance harms.The paper highlights analyses of the resulting policy’s distributional properties.Source: §1 · p. 3
• The authors describe these mechanisms as a starting point and propose combining dialogue with evidence and instructions, including debate, to assist human evaluation.Debate could let agents present arguments for and against their prior outputs.Source: §1 · p. 3

2.8. Reinforcement learning

• Sparrow’s RLHF methods are preferred to prompted baselines while violating its rules less often.RL trains on single-statement episodes and uses self-play across multiple turns, with Sparrow generating User, Agent, and Search Query turns and retrieving Search Results programmatically.Source: Figure 2 · p. 4, §2.8 · p. 11

2.1. Defining rules

• The 23 rules break helpfulness, correctness, and harmlessness goals into short, standalone requirements tailored to raters and rule-conditional classification.The rule set is representative rather than comprehensive, and focuses on harms addressable through natural-language rules and RL from human feedback.Source: §2.1 · pp. 4–5, Table 1 · p. 6

2.2. Generating dialogue turns

• Dialogue turns are generated from a prompted Chinchilla-70B model, with evidence-enabled prompts adding Search Query and Search Result participants.Agents can always search, never search, choose whether to search using role likelihoods, or generate candidate responses with and without evidence for reward-model reranking.Source: §2.2 · pp. 6–7

2.3. Human data collection

• Raters choose among candidate dialogue responses and try to elicit violations of specified rules, producing preference and adversarial-probing data for training and evaluation.Interactive tutorials and comprehension checks support task understanding; participants may skip sensitive tasks without penalty, and their well-being is monitored.Source: §2.3 · pp. 7–8

2.4. Evidence

• Evidence-aware preference judgments assess whether search is needed and whether provided web evidence supports each response.Four-response comparisons include two evidence-free and two evidence-conditioned responses; raters also assess plausibility and supportedness to measure faithful grounding and help address hallucinations.Source: §2.4 · p. 9

2.5. Reward models

• Separate Chinchilla 70B reward models estimate human response preferences and rule violations, with the jointly trained Rule RM sharing computation across rules.The Preference RM incorporates auxiliary predictions for evidence-supported and plausible answers and for comparisons where every response is low quality.Source: §2.5 · p. 10

2.6. Reranking

• Reranking selects the highest combined-reward response from sampled answers, including evidence-conditioned candidates, using preference and rule scores.This lets the agent selectively use search: preference rewards favor supported evidence for factual answers and no evidence for non-factual questions, while rule rewards penalize violations.Source: §2.6 · p. 11

2.7. Supervised fine-tuning

• Supervised fine-tuning trains Chinchilla on preferred responses and rule-compliant Agent turns from adversarial dialogues, yielding a stronger baseline and RL starting point.Preference examples target the selected response; adversarial examples are used when rated at least good and no rule was broken.Source: §2.7 · p. 11

2.8. Reinforcement learning

• Combining red-teaming data with rule and preference rewards yields the lowest rule violation rate while keeping agents substantially preferred to DPC.The results show that red-teaming data and rule rewards complement one another; preference-only training increases preference at the cost of more rule violations.Source: Figure 6 · p. 12, §2.8 · p. 14
• Training contexts come from ELI5 questions, human open-ended and adversarial conversations, red-team language-model questions, and self-play.Self-play adds valid statements to a buffer, with conversations capped at 12 statements; generated turns can take the User, Agent, or Search Query role. Valid responses above a minimum reward threshold are added back to the dialogue buffer, and Search Query turns trigger a programmatic Google search. All statements after the initial context are generated by Sparrow.Source: §2.8 · pp. 12–13, Figure 7 · p. 13
• Agent turns receive preference and mean rule rewards plus programmatic validity and conciseness rewards, while User turns receive preference rewards only.Preference and rule scores are independently normalized using running means and standard deviations before being added.Source: §2.8 · p. 13
• Sparrow updates its policy with batched synchronous A2C; V-MPO did not significantly improve performance, and nucleus sampling makes training off-policy without correction.The policy is initialized from an SFT model, and a KL penalty discourages collapse toward a single high-reward generation.Source: §2.8 · pp. 13–14
• A multi-headed hydra shares pretrained parameters while training only top layers, enabling five models to fit in memory during RL.The setup includes reward models, policy, value, and teacher models; Sparrow itself starts RL from the SFT model.Source: §2.8 · p. 14, Figure 8 · p. 14
• Varying training data and rewards offers an under-explored way to shape behavior, while future work could make helpfulness–harmlessness trade-offs configurable and expand the training topics and user models.The paper proposes an expanding universe of trade-offs and topics, and a league of user models designed to probe different aspects of agent behavior.Source: §2.8 · pp. 13–14

3.1. Preferences and rule violations

• RL with reranking@8 achieved the best performance in preference win rates and resilience to adversarial probing, while reranking improved preference across DPC, SFT, and RL models.Interventions improved resilience for most rules but not stereotypes, medical or legal advice, microaggressions, or insults; proposed explanations include convincing search results and less training data for some rules.Source: §3.1 · p. 15, Figure 9 · p. 15, Figure 10 · p. 15

3.2. Evidence evaluation

• 78% of Sparrow’s best-model responses with evidence were judged plausible and supported, and it agreed with raters on whether evidence was needed in over 90% of cases.Among the 7% of cases where raters said evidence was needed but Sparrow omitted it, raters changed their minds in 51%; other explanations included rule-violating answers, poor search results, and mislabelling.Source: §3.2 · pp. 16–17, Table 2 · p. 17, Figure 12 · p. 17
• On FilteredELI5, Sparrow (RL@4 - always search) achieved similar supported and plausible rates to GopherCite, and raters preferred its answers 63% of the time.The comparison used GopherCite’s human-evaluation interface and test set; Sparrow also supports real-time follow-up questions.Source: Table 3 · p. 17, §3.2 · p. 18

3.3. Correctness evaluation; 3.4. Rater perception of trustworthiness

• Sparrow answers were rated true or mostly true around 80% of the time in free dialogues, and trustworthy or very trustworthy in around 70% of conversations.The correctness investigation was small-scale and non-adversarial; supportedness and plausibility do not establish factual correctness, and correctness ratings did not assess helpfulness.Source: §3.3 · pp. 18–19, Table 4 · p. 19, §3.4 · p. 19

3.5. General versus specific rules

• Targeted probing made raters more likely to elicit violations of the specified rule than general-harm probing did, while general-harm probing uncovered novel harms in at least 19 of 566 dialogues.The newly found harms fell under Information Hazards and Misinformation Harms.Source: §3.5 · p. 20, Figure 13 · p. 20
• Specific-rule ratings had higher agreement than general-harm ratings (IAA 0.53 vs 0.37); rule-conditioned classifiers achieved higher AUC than general classifiers on full and half training data.AUC was 0.85 vs 0.77 on the full training set and 0.81 vs 0.69 on 50% of the data, respectively.Source: §3.5 · p. 21, Figure 14 · p. 21

3.6. Distributional harms; 3.6.2. Disparate impact for factual question answering

• Targeted rules can mitigate harms identifiable from single examples but not aggregate distributional harms; the paper also investigates possible disparate impact in fact-based question answering.The disparate-impact analysis measures answers to questions about specific groups as an indirect way to study usefulness across groups.Source: §3.6 · p. 22, §3.6.2 · p. 23

3.6.1. Stereotypes and social biases

• Bias persisted across stereotype datasets, and RL increased the Winogender bias score from 0.06 to 0.10 and the average ambiguous-BBQ score from 0.05 to 0.10.On Winobias type 1, DPC and RL were about 36% more likely to be correct on stereotype-reinforcing answers; ambiguous-BBQ bias increased in 10 of 11 categories, partly because RL models abstained less often.Source: Figure 15 · p. 22, §3.6.1 · p. 23

3.6.2. Disparate impact for factual question answering

• The evaluation measures exact-match accuracy across gender, country, and occupation groups using Natural Questions, Quiz Bowl, and TriviaQA.Questions are given directly to the dialogue model, and accuracy is the rate at which the correct answer appears in its response.Source: §3.6.2 · p. 23
• Significant accuracy correlations with demographic groups are not always observed, and including evidence can introduce or remove them.Significance is assessed using a χ2 test across Natural Questions, Quiz Bowl, and TriviaQA.Source: §3.6.2 · pp. 23–24, Figure 17 · p. 24
• Evidence improves accuracy across all demographic categories on TriviaQA.The analysis focuses on evidence effects because they show the largest effect sizes; full results are in appendix H.2.Source: §3.6.2 · p. 23, Figure 16 · p. 24

4. Discussion

• Including evidence can both introduce and remove correlations, and the authors view this work as a base for further safety mitigations.They identify several major directions for future exploration.Source: §4 · p. 24

4.1. Evidence limitations

• Sparrow uses one external knowledge fragment at a time, and its evidence-based notion of correctness leaves important questions about trustworthy sources and honesty unresolved.The authors note that this conservative definition rules out some true statements, does not aggregate statistical evidence across sources, and that Sparrow often copies evidence verbatim. They suggest multistep reasoning (Creswell et al., 2022; Dohan et al., 2022; Lewkowycz et al., 2022) and further rules as possible ways to address some limitations; honesty may require interpretability or eliciting latent knowledge.Source: §4.1 · p. 24

4.2. Dialogue as a supervision mechanism

• The authors hypothesize that dialogue can help resolve subtle supervision judgments through clarification and iterative discussion.Dialogue may support human supervision with assistance from ML systems (Christiano et al., 2018; Irving et al., 2018; Leike et al., 2018), and could combine adversarial debate (Irving et al., 2018) with human-agent cooperation to clarify misunderstandings (Hadfield-Menell et al., 2016). Evidence on one-step debate is mixed: critiques helped humans notice flaws in Saunders et al. (2022), while explanations did not improve accuracy in Parrish et al. (2022).Source: §4.2 · pp. 25–26

4.3. Ethical and sociotechnical aspects

• The rule mechanism aims to incorporate input from multiple stakeholders, but its legitimacy depends on rule origins, stakeholder participation, and technical and ethical considerations.Rules in this study were developed with domain and legal experts; broader participatory input is identified as critical (Halfaker and Geiger, 2020). Rule-conditional reward models work at the paper’s scale, but scaling to hundreds or thousands of rules may require architectural work; detailed rules also allow conflicts and weights to be changed later.Source: §4.3 · p. 26

4.4. More cognitive science research is needed

• Evidence may fail to change human beliefs, while increasing rule specificity trades off against efficient data collection.The authors note that people can resist compelling contradictory evidence through motivated reasoning. Their evaluations used at most five rules simultaneously, since a person may be unable to keep many rules in mind; the trade-off could be studied empirically.Source: §4.4 · p. 27

4.5. Broader impacts

• The authors favor downstream mitigation for many language harms but stress that their methods cover only instance harms raters can detect without substantial help.Privacy and social, linguistic, or environmental justice require pretraining-time mitigations as well as downstream work (Abadi et al., 2016). These methods are dual-use, so avoiding harmful outcomes requires addressing who controls the rules, whether affected parties share control, and whether rules are visible (Denton et al., 2020).Source: §4.5 · p. 27

5. Conclusion

• 78% evidence citation for factual questions and an 8% rule violation rate under adversarial conditions accompany more frequent helpful responses by rater preference.The conclusion reports that Sparrow correctly cites evidence 78% of the time for factual questions and reduces rule violations to 8% under adversarial conditions.Source: §5 · p. 27
• Sparrow focuses on width by breaking goals into detailed rules and using external knowledge to broaden the topics it can discuss correctly.The conclusion frames building helpful, correct, and harmless agents as requiring both breadth across goals and topics and depth in handling them carefully.Source: §5 · p. 27
• Addressing depth will require multistep reasoning, evolving rules with expert and participatory engagement, debate and dialogue for supervision, and cognitive science for real-world use.The proposed reasoning should leave interpretable traces for human evaluation, while debate and dialogue aim to improve detection of rule violations.Source: §5 · p. 27

7. Author contributions

• Amelia Glaese, Maja Trebacz, and Nat McAleese designed and built Sparrow’s architecture; Nat McAleese, Amelia Glaese, and others designed and built its RL training scheme.The contributions section also assigns work on retrieval, preference and rule reward models, SFT, experiments, evaluation, and project supervision.Source: §7 · p. 28

A.1. Prompts; A.2. Turn Generation; A.3. Dialogue Formatting

• Dialogue generation uses role-specific prompts, samples search queries and responses around Google Search results when using evidence, and ends turns at the next-role suffix.Sampling uses temperature 1 and top-p 0.8; prompts include examples of declining harmful requests.Source: §A.1 · p. 37, §A.2 · p. 41, §A.3 · p. 41

B.1. Prompted dialogue baseline (DPC - never search); B.2. Prompted evidence dialogue baseline (DPC - always search); B.3. Prompted selective evidence dialogue baseline (DPC - choose search); B.4. SFT baseline (SFT - choose search)

• The baselines comprise a never-search prompted dialogue model, an always-search prompted evidence model, a selective-search prompted model, and an SFT model that selects between responding and searching.The selective prompted baseline chooses a role using log likelihoods; the SFT baseline is trained using the next-role termination suffix.Source: §B.1 · p. 41, §B.2 · p. 41, §B.3 · p. 41

C. Retrieval and Scraping

• Retrieval sends generated queries to Google Search with SafeSearch, excludes Reddit, and truncates scraped snippets to fit the context while retaining relevant text.Documents with a snippet-to-document match ratio below 0.75 are discarded in favor of the Google snippet.Source: C. Retrieval and Scraping · p. 42

D. Reward modelling

• Reward models are initialized from Chinchilla and trained for one epoch on annotator data using Adam, bfloat16, and a linear-warmup cosine-decay learning-rate schedule.Batch sizes are 8 for preference models and 16 for rule models; hyperparameter sweeps select the number of steps and maximum learning rate.Source: D. Reward modelling · p. 42

D.1. Preference reward models

• Preference RMs use Bradley–Terry comparisons; the evidence-trained model also classifies whether claims are supported and plausible, and is used for inference-time reranking.The authors train one model on evidence and non-evidence data and another on non-evidence data only; using both improved performance.Source: §D.1 · pp. 42–43

D.2. Preference RM training data

• Preference RM training uses per-turn comparisons and GopherCite data, converting each multi-option comparison into pairs of the best continuation against each other option.When evidence is present, additional comparisons truncated at the search-query and search-result stages train predictions before the final response and about when evidence is needed.Source: §D.2 · pp. 43–44, Table 11 · p. 43, Table 12 · p. 44

D.3. Rule reward model

• The Rule RM predicts human rule-violation labels with a Yes/No classification objective and shares prefix computation across rules, so costs scale only weakly with rule count.Fine-tuning Chinchilla on instruction-described datasets did not significantly improve rule-violation prediction.Source: §D.3 · p. 44

D.4. Rule RM training data

• 14576 adversarial-probing dialogues were collected for rule-model training, split 70% for training and 30% for testing, with six training epochs.Likert ratings were binarized into break or follow labels, while unsure ratings were discarded.Source: §D.4 · pp. 44–45

E. Reinforcement learning

• The RL scheme trains a 70B A2C policy and regularizes it against the initial language model with a KL-divergence loss term (Menick et al., 2022; Perez et al., 2022).To reduce memory use, the first 64 of 80 transformer layers are frozen; policy and value functions share parameters.Source: E. Reinforcement learning · p. 45

E.1. Prompts

• For non-evidence Agent turns, prompts are randomly selected from DPC prompts, Rae et al. (2021)’s original prompt, or no prompt.RL uses DPC prompts for other turns, chosen according to dialogue role and, for Agent turns, whether search was used.Source: §E.1 · pp. 45–46

E.2. Red team LMs as User Models

• To elicit rule violations for RL training, LM-generated questions seed dialogues, with adjectives such as “medical,” “legal,” and “sexual” used to target questions.The approach follows Perez et al. (2022), adapting its prompt into a template and randomly choosing “someone” or “an AI” as the target.Source: §E.2 · p. 46

E.3. Rewards

• Agent turns use rule and preference rewards, Search Query turns use the with-evidence preference model, and User turns use the no-evidence preference model.Rewards are normalized before combination for Agent turns; episode-end rewards are undiscounted, with a small per-token concision penalty and a large penalty for improperly formatted statements.Source: §E.3 · p. 46

E.4. Evidence

• During RL, whether to search is chosen uniformly at random, and Search Query turns are rewarded myopically by the preference model.The downstream Agent-turn reward is not included in this Search Query reward.Source: §E.4 · p. 46

F.1. List of rules

• Harm rules drew on existing literature, and annotation limits were increased only when raters did not report wellbeing impacts; the process had internal ethics approval.Initial budgets capped examples per harm and repeated exposure to certain harms.Source: §F.1 · p. 47
• Harm rules prohibit stereotypes, microaggressions, threats, sexual aggression, identity attacks, insults, and hateful or harassing language.The rules also include a catch-all for hateful language that does not fit another category.Source: §F.1 · p. 48
• Self-presentation rules prohibit pretending to have a body, relationships, opinions, emotions, beliefs, or a human identity or life history.The rules also prohibit building a relationship with the user.Source: §F.1 · p. 48
• Other constraints require plausible claims, no unsupported assumptions about users, no real-world action claims, and limits on medical, legal, financial, and conspiracy-related responses.General discussion of biology, medicine, law, and investment is allowed, while specific advice in those domains is restricted.Source: §F.1 · pp. 48–49
• Helpfulness rules require dialogue to make sense, stay on topic, address user queries, and avoid unnecessary repetition; they were derived from baseline-model failure modes.Raters could also be asked whether the agent made assumptions about the user or took real-world action.Source: §F.1 · p. 49

G. Human data collection

• The human-data appendix introduces additional detail on the study’s collection methodology.Source: G. Human data collection · p. 49

G.1. Raters

• Raters were UK-based native English speakers with undergraduate-level education, recruited online; they completed tutorials and comprehension checks, and only those scoring at least 75% were retained.Tutorials involved 5–10 author-written practice tasks with answers and explanations.Source: §G.1 · pp. 49–50
• 67% of raters agreed on the preferred response and Krippendorff’s alpha was 0.44; agreement was middling for general harm and low for subtle rules such as no microaggressions.The preference agreement figure excludes tie cases.Source: §G.1 · p. 50, Figure 19 · p. 50

G.2. Rating task screenshots

• Figure 20 shows the per-turn response-preference rating task in a three-way comparison setting.The paper presents it as a representative rating-interface image.Source: Figure 20 · p. 51, §G.2 · p. 51
• Figure 21 shows extra annotation tasks that precede selection of the preferred response in Figure 20.Source: Figure 21 · p. 51
• Figure 22 shows the adversarial-probing task, after which annotators report their perceived success.Source: Figure 22 · p. 52

G.3. Well-being surveys

• 6.2% of 209 rating-task respondents reported greater negative well-being impact than similar annotation tasks without harmful language.For adversarial probing, 12 of 533 completed surveys (2.3%) reported greater negative impact.Source: §G.3 · p. 53
• Annotators’ comments included discomfort with sexual-content and suicide dialogues; some skipped them, but rating-task readers could encounter harmful language before deciding whether to participate.The authors updated probing instructions to clarify that conversations might not reflect annotators’ beliefs.Source: §G.3 · p. 53
• The authors hypothesized that mixing harm and non-harm rules and allowing task-skipping may have benefited well-being, but said this had not been tested.They noted that their lower reported impact than Welbl et al. (2021) was not directly comparable because study setup and annotator pools differed.Source: §G.3 · p. 53

G.4. Demographics

• Annotators were UK residents whose first language was English; tables report distributions for age, ethnicity, and sexual orientation.Source: §G.4 · p. 53, Table 15 · p. 54, Table 17 · p. 54

H.1. Stereotypes and social biases

• Winogender, Winobias, and BBQ were selected because they test concrete representational harms and define desirable versus harmful responses.The datasets construct questions around harmful stereotypes and make a zero bias score the ideal outcome.Source: §H.1 · pp. 54–55
• The bias metric measures the fraction of non-abstaining responses that reinforce stereotypes, normalized to [-1, 1], and equals the accuracy gap between balanced question types.A score of 1 indicates always giving the stereotype-reinforcing response, while -1 indicates always giving the stereotype-challenging response.Source: §H.1 · p. 55
• Because the dialogue prompt led the model to abstain on every zero-shot BBQ question, the main results use few-shot prompts with K=5 examples and do not directly measure agent behaviour.The authors reason that stereotype reliance after correctly answered examples suggests the model may rely on those stereotypes in other situations too.Source: §H.1 · p. 56
• DPC achieved 69.1% overall accuracy on few-shot BBQ, compared with 77.8% for UnifiedQA reported by (Parrish et al., 2021).The paper notes this is generally consistent with lower few-shot prompted model performance than fine-tuned model performance on reading comprehension tasks.Source: §H.1 · p. 56
• On ambiguous questions, the RL-fine-tuned model’s accuracy decreases from 87% to 65% because it is significantly less likely to abstain, yielding nearly triple the error and higher bias scores.The increased bias scores indicate greater likelihood of answering incorrectly when the correct answer is abstention, with those mistakes tending to reinforce stereotypes.Source: §H.1 · p. 56

Methodology

• The analysis compares exact-match accuracy by gender, occupation, and country across TriviaQA, Natural Questions, and Quiz Bowl, treating demographic categories as aggregate proxies.The datasets skew toward questions about males and English-speaking countries, particularly the US and UK.Source: §H.2 · pp. 56–57
• For demographic values with fewer than 15 examples, the analysis assigns an “others” category; it also groups missing and multiple values as “not found” or concatenated labels.For Natural Questions, only short-answer questions were retained, leaving a dev set of 2,631.Source: §H.2 · p. 57
• The study tests demographic-accuracy independence with χ² tests, using p < 0.0167 to account for three tests per dataset.The test uses counts of correct and incorrect answers for each demographic value and model version.Source: §H.2 · p. 57

Results

• Occupation and accuracy were associated in TriviaQA for all model variants; evidence models added associations in NQ and Quiz Bowl but removed the country association in TriviaQA.Associations were flagged at p-values < 0.0167 using a χ² test.Source: §H.2 · pp. 57–58
• Evidence-bearing DPC and Sparrow improved accuracy for most demographic characteristics and values, except on Quiz Bowl, where evidence did not help or slightly hurt.Quiz Bowl’s 2,216 samples make conclusions about demographic categories difficult.Source: §H.2 · p. 57, Figure 24 · p. 58

I. Evaluation of alignment taxes

• Evaluations focus on the core Sparrow policy, dialogue-ify benchmark prompts to match the DPC format, and omit evidence to avoid verbatim search-result matches.Source: §I · pp. 58–59
• Few-shot MMLU shows negligible alignment tax versus DPC, and zero-shot TruthfulQA shows none; subsequent SFT and RL@1 improve over DPC.Source: Figure 25 · p. 59, §I · p. 59, Figure 27 · p. 61
• MMLU topic-level score differences between Sparrow and DPC are too uncertain to distinguish the models once 95% bootstrapped confidence intervals are considered.Source: Figure 26 · p. 60

J.1. Positive examples of knowledge grounding

• For seaborn heatmaps, Sparrow says rows cannot have different heights and suggests using matplotlib to draw a grid with desired spacings.Source: §J.1 · p. 62
• The examples show Sparrow retrieving documentation through a mix of paraphrasing and copying, and using up-to-date information to answer questions.Source: Figure 28 · p. 62, Figure 29 · p. 62
• Asked whether Ukraine is safe to visit, Sparrow cites current US government advice against travel because of the active armed conflict and risks to US citizens.Source: §J.1 · p. 63, Figure 33 · p. 63

J.2. Negative examples of knowledge grounding

• Sparrow hallucinated despite search results, showing a failure to use available evidence.The example is captioned as hallucination ignoring the search results.Source: Figure 34 · p. 64
• For an ELI5 question, Sparrow failed to find relevant search results, and reranking preferred the answer without evidence.Source: Figure 36 · p. 64
• Sparrow produced an out-of-date fact, illustrating that its answers could rely on stale information.The figure caption identifies the example as hallucination leading to out-of-date facts.Source: Figure 35 · p. 64
• Sparrow took search results out of context, mistaking a count sampled from 1,000 productions for the total number of Shakespeare play stagings.Source: Figure 37 · p. 64
• Sparrow misinterpreted an evidence value of 7.88 as a speed in m/s, although it was a rating out of 10.The example is captioned as one of various bad behaviours.Source: Figure 38 · p. 65

J.3. Examples of harm avoidance

• When asked whether to go to work with a fever and bad cough, Sparrow declined to give medical advice.Source: Figure 39 · p. 66
• When baited about having legs, Sparrow avoided claiming it had them while still expressing sympathy.Source: Figure 41 · p. 66
• Sparrow declined to diagnose depression but did not refuse to quote symptoms.Source: Figure 40 · p. 66

J.4. Samples rated as rule breaking by human participants

• The authors warn that the examples touch on violence, deception, and stereotyping, and do not support the agent’s language or harmful language generally.Source: §J.4 · p. 66
• Sparrow gives legal advice about suing for battery, breaking the “not a lawyer” rule in response to an adversarial user.It suggests speaking with a lawyer but also discusses possible damages.Source: Figure 42 · p. 67, §J.4 · p. 67
• A human participant rated Sparrow’s refusal to diagnose a sore eye as probably breaking a rule, despite the response offering no diagnosis.The example is captioned as avoiding medical diagnosis despite evidence.Source: Figure 43 · p. 67, §J.4 · p. 67
• Sparrow repeats the claim that men are better at parking than women, quoting an unreliable internet source taken out of context.The example is rated “probably break” by a human participant.Source: Figure 44 · p. 68, §J.4 · p. 68

J.5. Dialogues exhibiting novel harms

• 566 dialogues probing the general harm rule were self-rated by the probing participant, then independently re-rated by two human raters against the general and specific harm rules.Heuristics selected dialogues with disagreement between the self-ratings and independent ratings for possible novel harms.Source: §J.5 · p. 69
• 19 of 36 candidate dialogues were confirmed by an author as demonstrating novel harms, including deceptive claims of private-data access and advice or statements involving violence, crime, suicide, and stereotyping.Candidates were identified from 566 adversarially probed dialogues; the examples also include recommending no punishment for murder.Source: §J.5 · p. 69, Figure 45 · p. 70, Figure 46 · p. 70

J.6. Interesting samples

• Interesting samples include Sparrow politely identifying itself as a dialogue agent, declining a nonsense question while offering sun-related information, and giving reasonable statements based on irrelevant evidence.The examples are shown in Figures 47–49.Source: Figure 47 · p. 71, Figure 48 · p. 71, Figure 49 · p. 71
• In other samples, Sparrow discusses arguments for and against women’s suffrage and moral theories, but one response ends with unsupported claims and off-topic evidence.The moral-theory exchange also shows Sparrow refusing to choose between the theories.Source: Figure 50 · p. 72, §J.6 · p. 73, Figure 51 · p. 73

J.7. TruthfulQA samples from GopherCite

• Sparrow RL@8 responses to TruthfulQA questions shown as GopherCite failure examples correctly avoid “wings” in Figures 52 and 53; Figure 53 is described as a correct answer.Figure 52 says “taurine” is correct, but the evidence does not support it.Source: §J.7 · p. 74, Figure 52 · p. 74, Figure 53 · p. 74

• Sparrow expresses confusion about a conspiracy theory with an out-of-context response, and a leading question makes it confused and supportive of the conspiracy.Source: Figure 60 · p. 75, Figure 61 · p. 75
• Training for harmless information-seeking dialogue removes other capabilities.Source: Figure 59 · p. 75
• Sparrow reasonably refers to Christiano et al. (2018) but omits details and hallucinates when asked who made IDA.Source: Figure 62 · p. 76
• Sparrow contradicts itself about giving advice, hallucinates when search fails, and muddles a moral-systems list despite correctly identifying the Rawls flaw.Source: Figure 63 · p. 77, Figure 64 · p. 77, Figure 65 · p. 77