PaperlayerView exact revision on arXiv

Source-linked AI summary

RARR: Researching and Revising What Language Models Say, Using Language Models

Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Y. Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, Kelvin Guu

arXiv:2210.08726v3cs.CLcs.AIcs.IRcs.LG

TL;DR

Language models can generate unsupported or misleading content without built-in attribution to external evidence. RARR researches evidence after generation and revises the output to improve attribution while preserving the original output. Across diverse tasks, RARR improves attribution while preserving most of the original text, but its evaluation and generality remain limited.

  • Problem

    Language models often produce unsupported or misleading content, while most lack a built-in mechanism for attributing outputs to external evidence.

  • Method

    RARR retrieves relevant evidence after generation and revises the output to agree with that evidence while preserving properties such as style or structure.

  • Results

    RARR significantly improves attribution while preserving most of the original text, performing robustly across three datasets and outperforming prior methods on NQ and SQA.

  • Takeaways & Limitations

    RARR provides a model-agnostic way to add attribution without modifying or retraining existing language models.

  • Takeaways & Limitations

    RARR is not yet fully general, and its metrics do not capture all aspects of attribution or preservation.

Abstract

from arXiv · show

Language models (LMs) now excel at many tasks such as few-shot learning, question answering, reasoning, and dialog. However, they sometimes generate unsupported or misleading content. A user cannot easily determine whether their outputs are trustworthy or not, because most LMs do not have any built-in mechanism for attribution to external evidence. To enable attribution while still preserving all the powerful advantages of recent generation models, we propose RARR (Retrofit Attribution using Research and Revision), a system that 1) automatically finds attribution for the output of any text generation model and 2) post-edits the output to fix unsupported content while preserving the original output as much as possible. When applied to the output of several state-of-the-art LMs on a diverse set of generation tasks, we find that RARR significantly improves attribution while otherwise preserving the original input to a much greater degree than previously explored edit models. Furthermore, the implementation of RARR requires only a handful of training examples, a large language model, and standard web search.

1 Introduction

Language models often produce unsupported or misleading content without built-in attribution. RARR researches evidence after generation and revises outputs to improve attribution while preserving properties such as style and structure.

  • State-of-the-art language models frequently generate biased, misleading, or unsupported content.
  • Most language models lack built-in attribution, and retrieval-augmented models can still add, ignore, or contradict retrieved information.
  • RARR is a model-agnostic approach that retrieves evidence after generation and revises text to agree with that evidence.
  • The revision aims to improve attribution while preserving original qualities such as style, structure, and seamless usability.
  • RARR formalizes Editing for Attribution and introduces metrics that assess both attribution and preservation across diverse language-model outputs.
  • The system uses few-shot prompting in large language models and standard web search to generalize research-and-revision across domains.
  • The paper distinguishes attribution from correctness: evidence support does not guarantee that the cited source is correct.

2 Task formulation

The Editing for Attribution task evaluates revisions by how well they are supported by evidence and how much they preserve the original text. RARR’s metrics avoid requiring gold edits while balancing attribution against unnecessary changes.

  • 2 Task formulation: Editing for Attribution maps an input passage x to revised text y and an attribution report A containing evidence snippets supporting y.
  • 2 Task formulation: The task evaluates revisions along two dimensions: attribution to evidence and preservation of aspects of the original text.
  • 2.1 Measuring attribution: AIS measures whether each sentence in y is fully attributable to evidence, with the reported score averaging sentence-level judgments.
  • 2.1 Measuring attribution: The attribution report is capped at M = 5 evidence snippets, which manual inspection found sufficient for full attribution in 30 benchmark examples.
  • 2.1 Measuring attribution: During development, Attrauto approximates human AIS using NLI probabilities that each evidence snippet entails each sentence.
  • 2.2 Measuring preservation: Preservation includes whether the revision retains the original intent and whether it avoids superfluous changes such as reordering, style changes, or added information.
  • 2.2 Measuring preservation: PresLev uses character-level Levenshtein distance, scoring 1.0 when x and y are identical and 0.0 when y completely overwrites x.
  • 2.2 Measuring preservation: Prescomb(x, y) = Presintent(x, y) · PresLev(x, y), while F1AP combines attribution and preservation to discourage revisions that optimize attribution alone.

3 Approach

RARR solves Editing for Attribution by researching an input passage, revising disagreements against retrieved evidence, and selecting a concise attribution report. Its few-shot prompted stages aim to improve attribution while minimally altering the original text.

  • 3 Approach: RARR researches an input passage by generating queries for potentially unattributed aspects and retrieving relevant evidence snippets.CQGen produces questions covering aspects requiring verification; Google Search retrieves pages, whose snippets are ranked and filtered.
  • 3 Approach: Most RARR components use few-shot prompting with PaLM, including query generation, agreement judgment, and editing.Figure 3 illustrates the few-shot examples used to prompt PaLM.
  • 3 Approach: The revision stage initializes y = x and edits only when the agreement model detects disagreement between the current text and retrieved evidence.When disagreement is found, the edit model revises y to agree with e; otherwise, the current output remains unchanged.
  • 3 Approach: The edit model aims to agree with evidence while otherwise minimally altering the current passage.It first identifies a span needing revision, and edits above 50 characters or 0.5 times the original text length are rejected.
  • 3 Approach: RARR selects at most M = 5 evidence snippets for the attribution report by maximizing query coverage using relevance scores as a proxy.The report may use fewer snippets than the retrieval and revision stages encountered, and the subset is searched exhaustively.

4 Related work

Related work positions RARR at the intersection of evidence-based fact checking, post-hoc factuality editing, attribution measurement, and retrieval-augmented generation. Unlike typical retrieval-augmented systems, RARR retrieves evidence to attribute and revise an existing output.

  • Fact-checking: Fact-checking research identifies whether claims are supported or refuted, while real-world settings may require retrieving the relevant evidence first.RARR builds on both evidence-based checking and retrieval-oriented work.
  • Post-hoc editing for factuality: Post-hoc editing methods correct text to be factually consistent with evidence, with FRUIT and PEER fine-tuning editors on Wikipedia edit history.These approaches focus on updating or correcting existing text rather than only checking claims.
  • Measuring attribution: Attribution is evaluated through human judgments and automated methods, including entailment-based metrics and answer-consistency checks.RARR uses an entailment-based metric for attribution and uses answer agreement in its model.
  • Retrieval-augmented models: Figure 4 presents input passages, with prior dialog turns included as context for QReCC.The figure illustrates the kinds of passages used as inputs in the broader evaluation setup.
  • Retrieval-augmented models: Retrieval-augmented models have succeeded across many knowledge-intensive tasks, but their retrieved material is not necessarily attribution and typically does not revise an existing output.This distinction separates retrieval for generation support from RARR’s post-hoc attribution and revision setting.

5 Experiments

The experiments evaluate RARR and comparison systems on factoid statements, reasoning chains, and knowledge-intensive dialogs. Across these tasks, RARR improves attribution while preserving substantially more of the original text than prior editors.

  • Evaluation tasks: The benchmarks cover factoid answers, reasoning chains, and knowledge-intensive dialogs generated by several language models.Factoid tasks use NQ, reasoning uses StrategyQA, and dialog uses QReCC; the setups test attribution across factual, intermediate reasoning, and context-dependent content.
  • Compared systems and metrics: RARR is compared with research-and-revise systems, including EFEC and LaMDA Research, alongside evaluation using attribution and preservation metrics.The evaluation reports AIS before and after editing, intent and Levenshtein-based preservation, their combination, and F1AP.
  • Results: RARR significantly improves attribution while preserving most of the original text.F1AP summarizes attribution and combined preservation, enabling comparison of both objectives.
  • Results: RARR preserves the original intent over 90% of the time, compared with 6–40% for EFEC and LaMDA.This result measures whether revisions completely preserve the original intent.
  • Results: RARR can increase attribution by up to 13% absolute while changing only 10–20% of the text.The comparison is against retrieving evidence without editing, where attribution ranges from the low 10s to mid 30s.
  • Results: Preservation and attribution involve a trade-off: EFEC obtains strong F1AP on QReCC by making larger changes than RARR.Figure 5 visualizes the different positions of systems on the attribution–preservation curve.

6 Analysis

RARR’s analysis shows strong preservation and attribution behavior, while exposing retrieval and revision failure modes. Query generation improves robustness, agreement checking prevents over-editing, and revisions can affect downstream performance differently across tasks.

  • Qualitative analysis: RARR preserves the input passage structure, unlike EFEC’s frequent one-sentence summaries and LaMDA’s stylistic changes.EFEC’s behavior can increase attribution while reducing preservation and informativeness on multi-sentence datasets.
  • Qualitative analysis: RARR revises unattributed claims involving entities and numbers, but misleading evidence can still produce erroneous edits.It avoids editing 90% of near-perfectly attributed NQ inputs, while some irrelevant retrievals cause incorrect changes.
  • Qualitative analysis: RARR can correct an earlier reasoning claim without revising later dependent steps, leaving the reasoning chain potentially incoherent.The authors identify further logical-coherence editing as a possible improvement.
  • Ablations: Querying with the entire passage performs poorly because retrieved evidence does not focus on potentially unattributed portions.Sentence queries are closer to full CQGen, but their effectiveness depends on whether the sentences resemble Web content.
  • Ablations: Sentence-as-query retrieval is less robust to corpus shifts and can encourage confirmation bias, whereas CQGen is more robust.Excluding Wikipedia causes a significant attribution drop for sentence queries, while CQGen is more robust; sentence retrieval may also echo outdated evidence.
  • Downstream task performance: RARR improves NQ answer accuracy by roughly 5%, while SQA answer accuracy drops modestly for revision models by up to 2.6%.The SQA decline is suspected to reflect noisy or misleading retrievals and unrevised downstream reasoning steps.

7 Conclusion

RARR addresses unsupported factoid claims by revising generated passages against researched evidence while preserving desirable properties and avoiding LM redesign or retraining. The authors report remaining headroom for improving attribution integration.

  • RARR revises unsubstantiated claims to make passages attributable to researched evidence.
  • Experiments across models and domains improved attribution while preserving writing style and structure.
  • RARR operates on existing generation models without redesigning or retraining them.
  • The authors identify substantial remaining headroom for developing approaches that integrate attribution into language models.

8 Limitations

The limitations concern how attribution and preservation are weighted, what the metrics capture, and how well RARR generalizes across inputs and claims. The model can be costly and may preserve unsupported claims rather than delete them.

  • Limitations of our task definition: Attribution and preservation may warrant different weights when multiple outputs are acceptable, especially in dialog systems.
  • Limitations of our task definition: The metrics penalize some self-evident sentences that do not require attribution, such as “I agree.”
  • Limitations of our task definition: Attribution requirements vary with assertion scope, since personal-experience statements may not require external attribution.
  • Limitations of our task definition: Revising deeply flawed passages without significant changes is difficult and heavily penalized by current preservation metrics.
  • Limitations of our model: RARR is not fully general because its few-shot prompts lack examples for inputs such as poetry and long documents.
  • Limitations of our model: RARR tends to preserve unattributed claims, although some are hallucinations that should be removed, and prompting a large model is computationally costly.

9 Ethical considerations

RARR’s attribution reports do not establish source correctness or fully resolve partial and conflicting support. These limitations require users to interpret attribution reports cautiously.

  • Partial attribution can leave unsupported portions of a revision, so readers should not assume the entire text is attributed.
  • Attributing content to a source does not establish that the source is trustworthy, accurate, or unbiased.
  • When sources conflict, RARR uses a permissive attribution definition requiring only one supporting source rather than resolving contradictions.

Contributions

The contributors developed RARR’s prototypes, prompting, retrieval and evaluation methods, experiments, infrastructure, and writing. Their work spans model implementation, benchmarking, human assessment, and project coordination.

  • Luyu Gao designed few-shot prompting, implemented the PaLM prototype, analyzed results, and advised on evaluations.
  • Zhuyun Dai and Ice Pasupat implemented final models, designed experiments, obtained main results and ablations, and contributed to analysis and writing.
  • Anthony Chen developed automatic attribution and preservation evaluation, helped design human evaluation, and released the GPT-3 RARR implementation.
  • Arun Chaganty led human evaluation and co-proposed the two-dimensional attribution-plus-preservation metric with Kelvin Guu.
  • Yicheng Fan developed the first prototype with Kelvin Guu and implemented retrieval strategies and the EFEC baseline.
  • Vincent Zhao proposed downstream-task evaluation and enabled bulk PaLM inference, while collaborators supported mentoring, integration, and research advising.

A Additional experiments and analysis

Additional experiments show that RARR’s performance is sensitive to model scale and remains consistent across generator models, while challenging domains expose clear limitations. It also improves attribution in MMLU and preserves similar trends on GPT-3 passages.

  • Additional ablations: < 1% difference appears when replacing Google Search with Microsoft Bing, indicating near-identical results across the two retrievers.
  • Additional ablations: 540B PaLM outperforms 62B PaLM by a large margin, suggesting that RARR may benefit from further model scaling.Shrinking only query generation to 64B produced a relatively small per-
  • Additional ablations: GPT-3 implementations perform slightly better than PaLM 540B on all three datasets, motivating an open-source GPT-3-backed version.The authors tuned the few-shot prompts slightly for GPT-3.
  • Challenging domains: News summaries and ELI5 explanations are challenging because retrievers struggle with lesser-known events and subjective, non-factual content.Before-edit auto-AIS scores for news summaries are low, while some ELI5 answers are difficult to attribute.
  • Challenging domains: RARR cannot satisfactorily edit outputs based on false premises when revision would require completely rewriting them.This limitation follows from the revision threshold.
  • Challenging domains: RARR improves attribution across all four MMLU categories, although the increases are relatively small.Performance is low on mathematical-reasoning examples because the edit model cannot handle them with the current prompt.

B Details on automated evaluation

Automated evaluation segments passages into sentences, decontextualizes them, and uses entailment scores to measure attribution, while preservation and attribution are also assessed through human judgments. The automated attribution metric correlates strongly with human labels.

  • Automated metrics: Sentence-level attribution treats each sentence as a claim because no linguistic consensus defines claims, supporting simplicity and reproducibility.The same segmentation is used for human evaluation.
  • Automated metrics: Decontextualization uses the passage’s question or dialog context to convert sentences into context-independent forms before scoring attribution.The procedure uses a T5 model, beam search with beam size 8, and discards outputs differing by more than 4 tokens.
  • Automated metrics: The NLI model maps an evidence–claim pair to entailment labels, and the probability of producing “1” becomes the entailment score.The model was trained on MNLI, SNLI, FEVER, PAWS, SciTail, and VitaminC.
  • Human evaluation: Pearson correlation is 0.74 for human versus automated attribution and 0.62 for preservation.On NQ, human AIS correlates with auto-AIS, although auto-AIS sometimes judges non-attributed sentences as attributed.
  • Human evaluation: Human evaluation compares attribution and intent similarity using the context, passage, and evidence, with annotators judging sentence support and passage similarity.Annotators review whether each sentence is fully supported and compare the intent conveyed by two passages.
  • Human evaluation: The evaluation emphasizes interpreting passages with context while ignoring evidence when determining the information conveyed by a sentence.Annotators use surrounding sentences and necessary context, but evidence is reserved for judging support.

E Details on the dataset

The dataset comprises generated English passages spanning six dataset–model combinations, with prompts tailored to the generation and contextual-dialog settings. The included prompts cover passage generation and each RARR component.

  • Dataset construction: 150 development and 150 test passages were generated for each of six dataset–model combinations involving NQ, SQA, QReCC, PaLM, LaMDA, and GPT-3.The combinations are NQ/PaLM, SQA/PaLM, QReCC/LaMDA, NQ/GPT-3, SQA/GPT-3, and QReCC/GPT-3.
  • Dataset construction: All generated passages are in English, and manual inspection found no personal identifiers in most of the data.
  • Prompts: The appendix provides few-shot prompts for query generation, agreement modeling, revision, and long-form answer generation across question-answering and dialog datasets.Contextual prompt versions support QReCC dialog contexts, whose turns are represented as Q1, A1, Q2, A2, …, Qk.
  • Prompts: Long-form answer prompts are separately specified for PaLM and GPT-3 on NQ and SQA, and for LaMDA and GPT-3 on QReCC.
Loading 2210.08726v3…