Source-linked AI summary
Teaching language models to support answers with verified quotes
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, Nat McAleese
TL;DR
Language models can produce convincing but ungrounded claims, so users must independently verify answers. The paper trains models to answer with verbatim supporting quotes, evaluates GopherCite through human assessment, and finds high-quality responses across two datasets, with abstention improving reliability.
Problem
Language models generate ungrounded claims that users must accept blindly or verify themselves, motivating question answering with supporting evidence.
Method
The paper trains self-supported question-answering models that generate answers alongside verbatim quotes from retrieved sources, evaluated through human assessments on filtered NaturalQuestions and ELI5 subsets.
Results
GopherCite produced high-quality answers 80% of the time on filtered NaturalQuestions and 67% on filtered ELI5; declining fewer than one-third improved these figures to 90% and 80%.
Takeaways & Limitations
Inline evidence can help users and raters verify model outputs, while reward modeling provides a mechanism for reranking, reinforcement learning, and abstaining when uncertain.
Takeaways & Limitations
Evidence alone does not guarantee truth because supporting sources can be unreliable and claims may be literally false despite appearing supported.
Abstract
from arXiv · showhide
Recent large language models often answer factual questions correctly. But users can't trust any given claim a model makes without fact-checking, because language models can hallucinate convincing nonsense. In this work we use reinforcement learning from human preferences (RLHP) to train "open-book" QA models that generate answers whilst also citing specific evidence for their claims, which aids in the appraisal of correctness. Supporting evidence is drawn from multiple documents found via a search engine, or from a single user-provided document. Our 280 billion parameter model, GopherCite, is able to produce answers with high quality supporting evidence and abstain from answering when unsure. We measure the performance of GopherCite by conducting human evaluation of answers to questions in a subset of the NaturalQuestions and ELI5 datasets. The model's response is found to be high-quality 80\% of the time on this Natural Questions subset, and 67\% of the time on the ELI5 subset. Abstaining from the third of questions for which it is most unsure improves performance to 90\% and 80\% respectively, approaching human baselines. However, analysis on the adversarial TruthfulQA dataset shows why citation is only one part of an overall strategy for safety and trustworthiness: not all claims supported by evidence are true.
1. Introduction
GopherCite trains language models to answer questions with inline verbatim evidence, helping users assess whether claims are supported. It achieves high-quality responses on filtered NaturalQuestions and ELI5 subsets, while abstention improves reliability, though supported evidence is not always truthful.
- Motivation and approach: GopherCite generates claims alongside verbatim supporting quotes extracted from retrieved documents or a user-provided document.The system uses evidence from Google Search or another information-retrieval system.
- Motivation and approach: Inline evidence reduces human appraisal effort by pointing users to specific quotes rather than entire web pages.The authors contrast this with systems that provide only URLs or unsupported answers.
- Motivation and approach: GopherCite casts self-supported question answering as conditional language modeling, generating answers and evidence inline as one string.The model is trained using supervised learning and reinforcement learning from human preferences after retrieval supplies a large multi-document context.
- Evaluation and results: 80% of filtered NaturalQuestions responses and 67% of filtered ELI5 responses were judged high quality, meaning plausible and supported.The evaluations used human raters and questions from filtered subsets of the two datasets.
- Evaluation and results: Declining fewer than one-third of questions raises response quality from 80% to 90% on filtered NaturalQuestions and from 67% to 80% on filtered ELI5.The system uses a reward-model mechanism to abstain from a configurable proportion of test-time questions.
- Limitations: Evidence citation alone does not ensure truth, because models can select misleading evidence, including from authoritative sources.The authors identify source trustworthiness and more careful qualification as directions for future work.
2. Methods
The method trains models to generate answers with inline verbatim evidence, using retrieved documents, human ratings, supervised fine-tuning, reward modeling, and reinforcement learning. At inference, multiple document-conditioned samples are reranked before one answer is returned.
- 2.1. Inline evidence syntax: Inline Evidence represents each answer and its supporting quote as one generated string.The format includes a claim, document title, and verbatim quote from the document.
- 2.1. Inline evidence syntax: The context c consists of documents retrieved from Google Search or provided by the user.
- 2.3. Conditioning and retrieval: At inference, the system retrieves the top K documents, samples N > K document-conditioned answers in round-robin order, and reranks them before returning one.Each sampling pass exposes the model to as much context as possible from a single document.
- 2.4. Training pipeline: The training pipeline repeatedly collects model outputs for human rating, then uses the resulting data for supervised fine-tuning and reward-model training.The trained models are used to collect additional training data across successive iterations.
- 2.7. Supervised finetuning: Supervised fine-tuning teaches the model to produce verbatim quotes in the Inline Evidence syntax and establishes baseline Self-Supported Question-Answering ability.
- 2.8. Reward modeling and reinforcement learning: A reward model trained on pairwise answer preferences supplies quality scores for reranking and reinforcement-learning optimization of quoting behavior.The project repeats the training loop until the desired performance is reached.
3. Results
GopherCite produced supported and plausible answers on NaturalQuestionsFiltered and ELI5Filtered, with human-preference training, reranking, and larger models improving performance. Selective abstention further increased quality among attempted questions, but high supportedness did not guarantee truthfulness on TruthfulQA.
- Human evaluation: 80% of NaturalQuestionsFiltered responses and 67% of ELI5Filtered responses were judged both plausible and supported.These evaluations used human raters assessing model samples on held-out question subsets.
- Training and reranking: Human-preference training, reward-model reranking, and reinforcement learning substantially improved GopherCite over purely supervised baselines.The improvements were reported across both evaluation datasets.
- Selective answering: Declining to answer below a reward-model threshold raised performance above 90% on NaturalQuestionsFiltered and above 80% on ELI5Filtered while attempting 70% of questions.This creates a configurable trade-off between question coverage and answer quality.
- Training and reranking: Reranking and reinforcement learning showed dataset-dependent differences: SFT plus top@64 beat RL plus top@64 on NaturalQuestionsFiltered, whereas RL consistently beat SFT on ELI5.Without reranking, RL outperformed SFT on both datasets.
- Model scale: Scaling the supervised fine-tuning generator improved Supported&Plausible and preference scores, with the 280B Gopher model performing best across configurations.The study varied both model parameters and the number of reranking candidates.
- Truthfulness limitation: TruthfulQA revealed that high Supported&Plausible scores can coexist with low Truthful&Informative scores, because models may select misleading evidence.The authors identify source trustworthiness and answer qualification as future challenges.
4. Discussion
Inline evidence can improve appraisal of model answers, but quoting evidence alone does not ensure truthful or complete support. The paper identifies source reliability, cherry-picking, contentious claims, and quotation limits as key boundaries.
- Limitations: Evidence quoting alone cannot ensure truthful outputs because supporting documents may be unreliable or biased.The authors note that even relatively high-quality corpora can contain errors or bias, making source reliability itself a difficult problem.
- Limitations: A single supporting quote may be insufficient when claims require multiple pieces of evidence or an argument linking evidence to the claim.
- Limitations: Inline evidence does not prevent misleading claims based on cherry-picked studies or quotations.Additional adversarial evidence could help identify cases where selected evidence appears convincing despite contrary findings.
- Limitations: Evidence can present contentious views as facts, including majority opinions that lack sufficient consensus.The authors suggest that addressing this failure mode will likely require dedicated sociotechnical research.
- Limitations: Some negative or statistical claims cannot be supported by a concise quotation, even when they follow from information in the corpus.The paper gives exhaustive negative claims and corpus-wide statistical claims as examples requiring stronger protocols.
5. Conclusion
GopherCite combines retrieval with inline evidence to help users verify answers, while the conclusion frames this mechanism as one component of a broader approach to trustworthy language agents.
- 5. Conclusion: GopherCite addresses hallucinated factual claims by generating self-supported answers that help users and raters verify outputs.The system uses mechanically verifiable verbatim quotes and human preferences to assess whether evidence supports an answer.
- 5. Conclusion: The conclusion presents inline evidence as one tool toward trustworthy language agents rather than a complete solution.It states that significant research is needed to address the mechanism’s limitations and combine it with other tools.
- 5. Conclusion: Given a question, the system retrieves relevant documents through Google Search and supplies the language model with context from multiple documents.
- 5. Conclusion: Retrieved web documents are converted to text and limited to snippets because documents often exceed the model’s 2048-token memory.
- 5. Conclusion: Tables 6 and 7 provide randomly chosen NaturalQuestions and ELI5 examples with model answers, claims, and evidence.
C. Human data collection details
Human evaluations presented claims alongside quoted evidence and asked raters to judge plausibility, support, and preference. Quality controls included screened raters, task-specific pools, and majority voting across independent judgments.
- Evaluation interface: Claims and supporting quotes were displayed in a dialogue-like interface, with claims in blue and evidence in grey call-out boxes.The evidence box included the page title and quoted fragment.
- Evaluation interface: Raters judged evidence in isolation without seeing page URLs, using only the app and requiring quotes to validate the claim.A response was supported only when the quoted evidence was sufficient to establish the claim’s correctness.
- Rater quality: Raters were screened through agreement tests and pre-task questions before contributing to the main evaluations.Super-rater filtering required 85% agreement with researchers, while participants failing at least three-quarters of screening questions were discarded.
- Rater quality: Super raters supplied training and Natural Questions validation data, whereas ELI5 and TruthfulQA used a wider non-overlapping pool with attention checks.This separated the evaluation pools used for different stages and datasets.
- Aggregation: Each example received ratings from multiple independent raters, with majority voting used to determine the final label.Super-rater evaluations used three independent raters per example.
- Documentation: The complete rating instructions were provided in Table 8.The table documents the instructions for comparing model outputs.
C.3. ELI5 vs Reddit evaluation
The ELI5-versus-Reddit evaluation reformatted model and human answers into comparable free-text responses and asked raters to compare their plausibility and overall quality. The task used controlled instructions, randomized answer order, and an adapted interface without separate support judgments.
- Reformatting: Model samples were merged into single free-text responses, and the adapted app omitted separate supported ratings.The model claim and evidence were combined inline to resemble the selected ELI5 answers.
- Task procedure: Answers were presented as attempts from an untrustworthy model with justifications, and raters were instructed to use only the app rather than external search.The answer order was randomized, and raters could select an unsure option when no comparison was possible.
- Rating criteria: Raters evaluated whether answers were plausible, meaning reasonable conversational replies that were on-topic and coherent.Off-topic, incoherent, or unclear replies were marked implausible.
- Rating criteria: Raters compared answers using helpfulness, brevity, naturalness, and their overall preference, allowing ties or genuine uncertainty.They were asked to justify choices with at least seven words and to refer to selected rather than left or right answers.
- Subset selection: ELI5 human answers were restricted to directly cited, unedited responses between 316 and 1313 characters.The selection aimed to make human answers more comparable with evidence-bearing model outputs.
C.4. TruthfulQA evaluation
TruthfulQA evaluation tested whether supporting-quote training also produced truthful claims, using an adapted interface with trusted correct and incorrect answers. The evaluation highlighted that evidence support does not by itself guarantee truth.
- Evaluation design: TruthfulQA evaluation assessed model-generated claims using the dataset’s truthful and informative definitions.The evaluation focused on the claim portions of model outputs.
- Evaluation design: The evaluation measured claims alone to match the expected answer format and test whether quote-support training improved claim truthfulness.An adapted app displayed trusted correct and incorrect answers to help raters identify misconceptions.
- Evaluation interface: TruthfulQA was presented through a dedicated evaluation app.The app interface is documented in Figure 10.
D.2. Training data
Training data combined human-rated QA examples with prompts containing target evidence in either single-document or multi-document contexts. The supervised pipeline used templated prompts and trained reward models validated against preference ratings.
- Training data: Training used answers rated both Plausible and Supported, drawn from QA-dataset training splits; 5151 unique question-answer pairs were shown during training.Training stopped after 60 steps, with the dataset distribution reported in Table 11.
- Prompt construction: One-third of prompts used a single target document, while two-thirds used 1 to 5 documents including other top Google results.The target document and quote were always present, and document order was shuffled.
- Prompt construction: Documents were truncated to lengths that preserved the relevant search snippet or target quote.This ensured the evidence needed for the target response remained in context.
- Prompt construction: Training and inference used templated prompts containing documents, the question, and an answer cue, with targets following the inline-evidence syntax.The prompt and target formats are documented in Table 12.
- Reward-model validation: Reward models were validated on held-out mixtures by preference-prediction accuracy and ROC curves for supported-and-plausible predictions.Validation ratings came from both researchers and external raters.
E.2. Training data
The reward-model training data primarily consists of human preference comparisons, augmented with transformed FEVER fact-checking examples to improve evidence verification. The FEVER transformation process creates question-answer comparisons with supported or refuted preferences, and its automated labels achieved 87% agreement with researcher ratings on 50 comparisons.
- The reward models were trained mainly on human-rated comparisons collected from four popular question-answering datasets.The dataset breakdown and counts are provided in Table 13.
- FEVER claims were transformed into question-answer comparisons to add out-of-distribution, non-extractive examples for evidence verification.FEVER contains Wikipedia-derived claims labeled Supported, Refuted, or NotEnough with associated evidence.
- The transformations compared affirmative or claim-based answers against negations or answers paired with fabricated quotes, assigning preferences according to the original claim’s support status.Types A, B, A2, and B2 vary whether the comparison uses direct templates, few-shot-generated questions, or fake quotes.
- 87% agreement was measured between automatically assigned preferences and researcher judgments across 50 manually rated comparisons.
F.1. Training procedure
The training procedure uses the paper’s supervised and reinforcement-learning setup with constrained implementation choices, including frozen layers, syntax penalties, and selected hyperparameters. Training covered 16,640 episodes and 24,371 unique questions across single- and multi-document prompts.
- During reinforcement learning, the 280B A2C policy was trained with Adafactor, learning rate 2 × 10^-6, effective batch size 16, and gradient clipping at norm 1.0.The first 60% of weights, or 48 of 80 transformer layers, were frozen; policy and value functions shared parameters.
- The procedure applied a bad syntax penalty to samples violating quote, title, claim, or minimum-length requirements.The selected bad syntax penalty was 2, while the alternative value 3 was also compared.
- Across 16,640 episodes, the model saw 24,371 unique questions using either one document or randomly up to five documents, with a 1:4 NaturalQuestions-to-ELI5 training split.
G. Decline to answer ablations
The decline-to-answer analysis varies the attempted-answer rate and evaluates Supported & Plausible performance among attempted answers. Prompting differs between NaturalQuestions and ELI5, while constrained sampling helps produce verbatim evidence quotes.
- Figure 11 plots Supported & Plausible answers against the percentage of questions for which an answer is attempted.The curves compare RewardModel_v1 with RewardModel_v2, which adds 4,000 ELI5 training comparison examples.
- NaturalQuestions: NaturalQuestions uses few-shot examples whose targets are written directly in the desired answer-and-evidence syntax.The examples are randomly drawn from five handwritten examples.
- ELI5: ELI5 generation first produces a claim and then mechanistically retrieves evidence from the conditioned document.This two-part process reflects the longer and less extractive nature of ELI5 responses.
- Constrained sampling: Constrained sampling masks output tokens that do not occur contiguously in conditioning documents, making generated quotes verbatim.A finite-state machine applies masking in quote states but allows free-form claim generation.
I. Released model samples from ELI5 And NatQs test sets
Released NaturalQuestions and ELI5 test-set samples include annotator ratings, while the generation system enforces document titles and verbatim quotes through constrained sampling. A finite-state process separates free-form claims from structurally checked evidence.
- Full NaturalQuestionsFiltered and ELI5Filtered test-set samples, together with annotator ratings, were released at two listed URLs.
- Constrained sampling masks tokens that do not occur as contiguous subsequences within documents in the model’s context.This mechanism is used to enforce verbatim quotes through Inline Evidence Syntax.
- A finite-state machine permits unrestricted claim text while applying quote-specific masking after special-token state transitions.
- The generation states require a document title after the claim and then a quote from one of the conditioning documents.The displayed state sequence includes start, claim, document-title, and quote stages.
K. Examples of GopherCite answering questions about the Introduction
The section demonstrates GopherCite answering simple questions about the paper’s Introduction using a long, uncurated context. The examples are shown in Figure 12, with the best answer selected from 16 SFT-model samples for each question.
- The Introduction served as the context for simple questions answered by GopherCite, as shown in Figure 12.After preprocessing to remove whitespace, the Introduction contained 1774 subword tokens.
- 1774 subword tokens made up the preprocessed Introduction context.Whitespace was removed before counting the tokens.
- 16 SFT-model samples were considered for each question, with researchers cherry-picking the best answer.