PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Attribution in Natural Language Generation Models

Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, David Reitter

arXiv:2112.12870v2cs.CL

TL;DR

NLG systems often hallucinate, but existing evaluations lack a unified framework for determining whether generated statements are supported by sources. This paper introduces AIS and finds that trained human annotators achieve moderate-to-high agreement across tasks, supporting AIS as a common attribution measure.

  • Problem

    Existing NLG evaluations lack a unified, formally expressed framework for measuring whether generated statements are supported by underlying sources.

  • Method

    AIS formalizes whether system statements are derivable from identified sources and operationalizes this definition through a two-stage human-annotation pipeline.

  • Results

    Human evaluations achieved moderate-to-high inter-annotator agreement across tasks and revealed generally expected differences in models’ AIS scores.

  • Takeaways & Limitations

    AIS provides a common framework for evaluating attribution to underlying sources in generated text, with verification serving as a necessary but insufficient quality-assessment step.

  • Takeaways & Limitations

    AIS applies only when a system sentence has an explicature that is a standalone proposition, leaving cases such as questions for future work.

Abstract

from arXiv · show

With recent improvements in natural language generation (NLG) models for various applications, it has become imperative to have the means to identify and evaluate whether NLG output is only sharing verifiable information about the external world. In this work, we present a new evaluation framework entitled Attributable to Identified Sources (AIS) for assessing the output of natural language generation models, when such output pertains to the external world. We first define AIS and introduce a two-stage annotation pipeline for allowing annotators to appropriately evaluate model output according to AIS guidelines. We empirically validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset) via human evaluation studies that suggest that AIS could serve as a common framework for measuring whether model-generated statements are supported by underlying sources. We release guidelines for the human evaluation studies.

1. Introduction

The paper introduces Attributable to Identified Sources (AIS), a framework for evaluating whether generated statements are derivable from underlying sources. Human studies across three NLG tasks indicate that AIS evaluations can achieve moderate-to-high inter-annotator agreement and distinguish model outputs’ attribution quality.

  • NLG models often hallucinate and lack faithfulness to underlying sources, making accurate evaluation important.
  • AIS can attribute generated claims to source documents, helping designers and users assess whether provided information is supported.
  • AIS evaluates whether natural-language statements are derivable from a given underlying source.Its definition formalizes sentence meaning in context using explicatures and tests attribution to background information.
  • The framework is instantiated for conversational QA, text summarization, and table-to-text generation.Responses are evaluated against answer documents, summaries against source articles, and descriptions against source tables and associated metadata.
  • Human evaluations achieve moderate-to-high inter-annotator agreement and reveal differences in models’ AIS scores across tasks.Annotators evaluate outputs from multiple models using task-specific AIS instantiations, with generally expected score trends.

2. Background

Prior work studied hallucination and source support across data-to-text, summarization, and dialogue, using task-specific automatic metrics and human evaluations. However, evaluation practices lacked a unified framework, consistent terminology, and detailed reproducible annotation instructions.

  • Hallucination in NLG: Neural data-to-text models can generate content unsupported by source data, motivating information-extraction metrics and human studies of source support.Wiseman, Shieber, and Rush proposed an automatic metric and conducted a small human evaluation examining whether summaries were supported by their sources.
  • Hallucination in NLG: Summarization research characterized hallucinations extensively and introduced additional automatic question-answering methods for detecting them.Maynez et al. studied content drawn from larger corpora beyond the given source, while Wang, Cho, and Lewis and Nan et al. proposed QA-based detection methods.
  • Dialogue grounding: Dialogue response research measured consistency with conversational history or groundedness in external evidence using dialogue-specific natural language inference methods, human studies, and data creation.This work particularly concerns responses about the real world and treats these properties as closely related to hallucination.
  • Evaluation gap: Despite substantial cross-task research, no unified approach evaluated whether generated statements were supported by source documents, and annotation guidance and terminology were inconsistent.Human evaluation studies varied across papers, while detailed, reproducible annotation instructions were unavailable.
  • Evidence and inference: Related evidence-based work retrieved supporting evidence for claims and sometimes classified semantic relations between claims and evidence, overlapping with natural language inference research.Thorne et al. examined several human annotation tasks in this problem family, but the paper identifies key differences from its own approach.

3. A Formal Definition of Attributable to Identified Sources

This section formally defines AIS first for standalone propositions and then extends it to contextual, multi-sentence utterances through explicatures, interpretability checks, and attribution to identified source parts. The framework applies across conversational QA, summarization, and table-to-text interactions, while excluding cases whose explicatures are not standalone propositions.

  • 3. A Formal Definition of Attributable to Identified Sources: AIS is formally defined first for system utterances that are standalone propositions, establishing the basis for the full definition.The section presents the simple case as informative and foundational for broader utterance types.
  • 3.1 An Initial Definition of AIS: Attribution of Standalone Propositions: A standalone proposition is a declarative sentence interpretable once a time t has been specified.Sentences requiring unresolved references or missing historical context are not standalone propositions.
  • 3.1 An Initial Definition of AIS: Attribution of Standalone Propositions: For a pair (s, t), AIS requires the system to provide source parts P from corpus K and make (s, t) attributable to P.Attribution means a generic hearer can affirm, with a chosen confidence level, “According to P, s,” with s interpreted at time t.
  • 3.1 An Initial Definition of AIS: Attribution of Standalone Propositions: Source corpus K may consist of web pages or a knowledge graph, while P identifies relevant paragraphs, sentences, or graph parts.The definition permits multiple source parts and multi-hop reasoning, including arithmetic processes such as reasoning about dates and age.
  • 3.2 Extending AIS: Attribution of Sentences in Context: The extended framework defines an utterance as one or more system- or user-produced sentences, including declaratives, questions, commands, exclamations, and fragments.It covers multi-sentence cases, non-empty linguistic contexts, and uninterpretable cases.
  • 3.2 Extending AIS: Attribution of Sentences in Context: In context, an utterance’s meaning is represented by an explicature, which can then be evaluated for AIS as a standalone proposition.For example, contextual pronouns can be made explicit by incorporating information from preceding user and system utterances.
  • 3.2.1 Definition of Interactions and Linguistic Context.: The interaction definition supports conversational QA, summarization, and table-to-text settings through ordered user and system utterance sequences.Conversational QA typically alternates user and system utterances; summarization and table-to-text use one system utterance with no user utterances.
  • 3.3.1 The Full Definition of AIS.: The full AIS definition adds interpretability checks and applies attribution to explicatures, but requires each explicature to be a standalone proposition.Cases such as question explicatures are left for future work.

4. Human Evaluation Study

The human evaluation study tests AIS assessment across conversational QA, summarization, and table-to-text generation using a two-stage process that first evaluates interpretability without sources and then evaluates source support. Five annotators judge each example, while the framework isolates attribution from other output-quality dimensions and acknowledges ambiguity and coverage limitations.

  • Study design: AIS feasibility is evaluated across conversational question answering, summarization, and table-to-text generation using outputs from different models.The study presents model outputs for each task to quantify the significance of human judgments.
  • Quality control: Five annotators judge each example, with results compared against consensus when available; flagged malformed tasks are excluded from interpretability and AIS scores.Flagged cases include rendering problems, corrupted text, underspecified sources, and sources that are difficult to understand.
  • Two-stage annotation: The annotation pipeline first asks whether output is interpretable without the source, then asks whether interpretable output is fully supported by the attributed source.Conversational QA additionally provides preceding conversation context, while summarization and table-to-text use empty context.
  • Interpretability assessment: Source-free interpretability assessment is crucial because annotators may otherwise be forgiving of ambiguity when the source helps recover missing context.The main ambiguity concern is unclear anaphora, with syntactic ambiguity and disfluency also potentially obstructing attribution.
  • Source attribution: Annotators mark output attributable only when all information can be directly inferred from the source; misleading wording, omitted context, or slight detail changes make it not fully attributable.The instructions include a paraphrase test asking whether the source can be said to make the resulting statement.
  • Scope and limitations: The study evaluates entire outputs with yes/no judgments and focuses on attribution separately from qualities such as relevance and non-redundancy, limiting fine-grained analysis.If one span is not attributable, the whole output is treated as not AIS.

5. Experiments

Experiments apply AIS human evaluation across conversational QA, summarization, and table-to-text generation, showing that evidence use and extractiveness affect attribution outcomes. Annotation studies indicate generally consistent judgments, while reference data itself can impose AIS-quality limits.

  • Experimental scope: AIS was instantiated for four datasets spanning conversational QA, summarization, and table-to-text generation, with human evaluations of outputs from multiple models.The framework targets tasks whose outputs are intended to be attributable to source documents.
  • Conversational QA: Fine-tuning increased interpretability by about 50 points, while the no-evidence model was interpretable and AIS only 15% of the time.The evidence-conditioned fine-tuned model achieved the highest AIS subscore; model-size differences were generally not significant.
  • Conversational QA: Wizard of Wikipedia produced more flagged examples than QReCC, while controllable inputs for faithfulness yielded the highest AIS results.The overall trend resembled prior human evaluations of faithfulness and subjectivity.
  • Summarization: More extractive summarization approaches generally achieved higher AIS subscores, consistent with reduced hallucination from quoting documents verbatim.Gold reference summaries also had surprisingly lower AIS scores than model outputs.
  • Table-to-text: T5-base responses were most interpretable, whereas ByT5 responses were significantly less interpretable and T5 outputs were more often flagged for encoding artefacts.AIS subscore differences were generally not statistically significant, although larger architectures showed a trend toward higher AIS.
  • Annotation analysis: Agreement among crowd annotators was generally moderate to high, and annotation-quality audits reported results in the high nineties across all three tasks.Crowd annotators agreed more with one another than with experts, while expert comparisons still showed reasonably consistent agreement.
  • Annotation analysis: Completion times decreased as annotators internalized instructions, with conversational QA fastest, table-to-text intermediate, and summarization longest.Shorter completion times generally corresponded to higher interannotator agreement.
  • Limitations: Reference annotations revealed an AIS-quality ceiling in the data itself, while uninterpretable outputs commonly contained repetitive language, ambiguous pronouns, or ellipses.The authors attribute non-AIS references partly to data creators not being instructed to preserve faithfulness to underlying documents.

6. Discussion

The discussion presents source verification as a necessary but insufficient component of evaluating generated text, while identifying limitations in AIS’s applicability and contextual assumptions. It also reports that, with appropriate training, instructions, and interfaces, crowd workers can judge attribution to underlying sources, though not without limitations.

  • Contribution: AIS formalizes verification against a known source as a necessary but insufficient step for ensuring generated-text quality.The framework focuses on whether generated text can be checked against underlying source(s).
  • Contribution: With appropriate training, careful instructions, and optimized user interfaces, crowd workers can judge whether generated text is attributable to underlying source(s).The discussion also identifies limitations in delegating this judgment to crowd workers.
  • Limitations: AIS applies only to propositions judgeable through the “according to” framework, excluding questions without presuppositions and imperatives.The discussion also notes that strict attribution can conflict with desirable characteristics in some systems, such as chit–chat systems.
  • Limitations: The AIS definition deliberately limits context to a shared time t, although real interactions may include richer communication channels and multi-session history.Annotators must account for explicature, including resolving explicit references and implications from context.

7. Conclusion

The paper introduces Attributable to Identified Sources (AIS), an evaluation framework for assessing whether generated information is supported by source documents. It formalizes AIS, applies it to three NLG tasks, and quantitatively validates it through human evaluation studies.

  • AIS evaluates whether information in generated text can be supported by source documents.The framework is designed to inspect support for generated information.
  • The paper formally defines AIS and describes its application to conversational QA, summarization, and table-to-text generation.These are the three NLG tasks covered by the framework.
  • The framework is quantitatively validated through human evaluation studies in which annotators rated model outputs according to AIS.

Appendix A: Examples of Annotations … 1. Interpretability

The appendices provide annotation examples and detailed instructions for evaluating interpretability and attribution across conversational question answering and summarization. The guidance emphasizes context-sensitive interpretation, complete source support, and flagging tasks that cannot be reliably judged.

  • Appendix A: Examples of Annotations: Annotation examples distinguish responses that are uninterpretable, interpretable but not AIS, and interpretable and AIS.The appendix includes separate tables for these three annotation outcomes and examples from table-to-text annotations.
  • Overview: The evaluation workflow rates interpretability before attribution, with later categories skipped depending on earlier ratings.Annotators are instructed to evaluate whether system-generated responses or summaries are understandable, then assess source support when applicable.
  • 1. Interpretability: Interpretability asks whether all response information is understandable, marking No for vague or ambiguous meaning and malformed phrases or sentences.For summaries, the source article is hidden so interpretability is judged on its own; conversational QA uses the query and preceding turns as context.
  • 2. Attribution: Attribution asks whether every part of the system response is fully supported and accurately represented by the cited source document.Annotators use the user query, system response, and source document, while judging support rather than the source’s independent correctness or the response’s relevance.
  • A. What is the information provided by the system response?: Context helps determine the information conveyed by terse responses, but the source document must be ignored when interpreting that information.Pronouns, elliptical answers, and phrases such as “the show” are paraphrased using the query and prior conversation; unclear intended meaning receives an interpretability No.
  • B. Definition of “An Accurate representation of Information in the Source Document”: Attribution requires checking all response information against the document, marking No when any content is missing, inaccurate, or unsupported.The instructions recommend asking whether “according to the document” accurately introduces the response and applying a journalist-like standard of accurate representation.
  • Flag: Tasks should be flagged for missing components, severely malformed text, underspecified sources, or documents requiring expertise to evaluate reliably.Examples include ambiguous event or character references, corrupted text, insufficient source identification, and scientific material beyond the annotator’s expertise.
  • Appendix C: Evaluation Instructions for Summarization — 1. Interpretability: Summary interpretability is judged from the summary alone, requiring annotators to mark No when any part is unclear or difficult to interpret.The article is hidden at this stage, and examples identify unclear noun references or pronouns and malformed sentences as sources of diminished intelligibility.

Interpreting the information provided in the system summary. … Interpretability

The framework evaluates summaries in stages: first, whether they are understandable on their own, and then whether all their information is accurately supported by the source article. Interpretability failures include missing references, vague pronouns, and malformed constructions, while attribution requires checking every claim against the source without judging source quality.

  • Interpreting the information provided in the system summary.: A summary is interpretable when its meaning and subject are clear without consulting the original article.Summaries with missing context should be marked “No, the summary is unclear.”
  • Interpreting the information provided in the system summary.: Missing noun-phrase references, unclear pronouns, and malformed phrases or sentences can diminish interpretability.Examples include unclear references for “the project,” “the museum,” “the city,” and “it,” as well as poorly formed sentence connections.
  • Interpreting the information provided in the system summary.: Annotators should use their best judgment about intended meaning and mark “No, the summary is unclear” when uncertain.The guidance explicitly favors the unclear rating when the summary’s intended meaning is difficult to determine.
  • 2. Attribution: Attribution asks whether all information in the system-generated summary is fully supported by the source news article.Annotators compare the summary with the source article, which is introduced at this stage of the task.
  • 2. Attribution: A fully attributable summary accurately represents the source article and contains no unattested information.If any information is missing from the article or inaccurately represented, the summary should receive “No, not fully attributable.”
  • 2. Attribution: Attribution judgments should check every part of the summary and apply a journalist’s standard of accurate representation.Annotators are encouraged to ask whether “the provided news article says...” could accurately precede the summary.
  • Some Final Important Notes.: Attribution evaluates representation of the source article, not whether that underlying article is correct or trustworthy.Source quality is assessed separately from the attribution category.
  • Interpretability: Interpretability examples show that an unidentified “bill,” vague personal references, and missing or poorly connected sentence elements make summaries difficult to understand.These examples respectively illustrate unclear noun phrases, ambiguous pronouns, and malformed constructions.

Attribution … 1. Interpretability

The evaluation instructions define attribution as whether generated content is supported by its source, while interpretability asks whether a caption is understandable on its own. Annotators also receive procedures for flagging missing, malformed, or overly specialized tasks and are warned to inspect fluent captions for subtle inaccuracies.

  • Attribution: Attribution is positive when all summary information is supported by the source article and negative when any information lacks support.Examples distinguish a fully attributable summary from one containing unsupported details such as a location and an alternative team reference.
  • Flag: Annotators should flag tasks with missing summaries or source articles, severely malformed text, or documents requiring expertise they do not possess.If enough of the document remains understandable, annotators may still rate attributability instead of flagging.
  • Appendix D: Evaluation Instructions for Table-to-Text: The table-to-text appendix provides verbatim instructions used by paid crowd annotators, developed after the conversational QA and summarization instructions.The instructions were presented alongside the task interface, which linked rating questions to examples.
  • Overview: The table-caption task evaluates how well a system-generated caption represents information from highlighted parts of a source table.Caption quality is assessed using the categories Interpretability and Attribution in succession, with the interface controlling the task flow.
  • Overview: Annotators are instructed to read captions carefully because fluent, well-formed text may contain subtle inaccuracies.The guidance explicitly compares the review process to proofreading.
  • 1. Interpretability: Interpretability is evaluated before attribution with the source table hidden, asking whether all caption information can be understood independently.Annotators select “No” when any portion is unclear or difficult to interpret and may provide a concise justification.
  • 1. Interpretability: An uninterpretable caption has reduced intelligibility because of vague or ambiguous meaning, malformed language, or sentences that are difficult to understand.Examples include unclear noun references, insufficient context, and malformed phrases or sentences.
  • 1. Interpretability: Captions judged interpretable proceed to the next evaluation category, with additional examples and rating justifications provided in the appendix.The workflow therefore uses interpretability as a gate before subsequent assessment.

Interpreting the information provided in the system caption. … Attribution

The framework separates interpretability from attribution: captions must be understandable, and every claim must be accurately supported by the source table and its descriptions. Annotators should check all caption information, use highlighted cells as guidance, and judge only representation of the source—not source quality.

  • Interpreting the information provided in the system caption.: Interpretability asks whether the caption’s intended meaning is clear and stands on its own without requiring missing context.Annotators should mark unclear captions when context, noun references, or grammatical elements are missing, erring toward “No, the caption is unclear.”
  • Interpretability: Interpretability failures include unexplained numerical claims, unspecified noun references, malformed sentences, and conflicting or missing temporal context.Examples include “Bradman scored 299,” a sentence missing a verb, and two albums both described as the number one album of the year.
  • 2. Attribution: Attribution asks whether all information in a system-generated caption can be verified from the source table and its associated descriptions.A fully attributable caption accurately represents the source table without unattested information.
  • Some Final Important Notes.: Annotators should evaluate the caption against the table, title, section title, table section text, and highlighted cells that indicate likely focal information.Highlighted cells are helpful hints, but captions may also refer to information elsewhere in the table and descriptions.
  • Assessing the accuracy of the information in the caption against the source table.: Accuracy should be judged as whether it is fair to say “the provided table says” before the caption, using a journalist-like standard of faithful representation.Annotators should consider whether they would regard the caption as an accurate representation if they had written the source document.
  • Assessing the accuracy of the information in the caption against the source table.: A caption is not fully attributable when any information is missing from the table or inaccurately represents it, even if other information is supported.The rating covers captions that are partially or not at all supported.
  • Some Final Important Notes.: Source quality is excluded from attribution judgments because the attribution category evaluates only whether the caption accurately represents the underlying source table.Whether the table itself is correct or trustworthy is evaluated separately.
  • Attribution: Attribution examples distinguish accurate paraphrases and simple inferences from swapped values, incorrect entities, and unsupported labels.Supported examples preserve sales figures or infer the largest ethnic group, whereas unsupported examples swap album sales, add an artist, or call unidentified numbers “seats.”

Flag

The interface provides a flag button for reporting ineligible tasks before proceeding. Tasks should be flagged for missing components, severely malformed text, or source tables requiring expertise to evaluate properly, but empty titles or section text marked “None” are acceptable.

  • Flagging tasks: Use the bottom-left flag button to report an ineligible task and then proceed to the next task.The button is intended for reporting tasks that cannot be evaluated under the guidelines.
  • Flagging tasks: Flag tasks with missing summaries or news articles, or when any component contains severely malformed text such as HTML artifacts or unformatted tables.Missing components and unintelligible text make the task unsuitable for evaluation.
  • Flagging tasks: Empty table titles, section titles, or table section text marked “None” are acceptable and should not be flagged.These empty or designated fields do not by themselves make a task ineligible.
  • Flagging tasks: Flag a source table when properly evaluating the response requires expert-level knowledge, but continue rating if the table remains sufficiently understandable.Scientific formulas and obscure terminology may require flagging when they prevent proper evaluation.

Appendix E: Annotator Interface for Conversational QA Tasks

The conversational QA annotator interface uses separate interpretability and attribution stages. The source document is hidden during interpretability and shown during attribution, with stage-specific justification requirements during training and pilot.

  • Interpretability stage: In the interpretability stage, the source document is hidden; non-interpretable tasks skip attribution and proceed to the next queued task.During training and pilot, the justification element is shown only when the task is rated as not interpretable.
  • Attribution stage: In the attribution stage, the source document is shown, and training and pilot require justification for all ratings.

Appendix F: Annotator Interface for Summarization Tasks

The summarization annotator interface uses separate interpretability and attribution stages. The source document is hidden during interpretability and shown during attribution, with justification requirements varying by phase and rating.

  • Interpretability stage: The interpretability stage hides the source document; tasks rated not interpretable skip attribution and proceed to the next queue item.During training and pilot, the justification element appears only when the task is rated not interpretable.
  • Attribution stage: The attribution stage displays the source document, and during training and pilot, annotators must provide justification for every rating.

Appendix G: Annotator Interface for Table-to-Text Tasks

The table-to-text annotator interface uses sequential interpretability and attribution stages. Attribution is assessed only when the source table and description are interpretable, with training and pilot displays controlling when justifications appear.

  • Interpretability stage: The interpretability stage hides the source table and description before rating whether the task is interpretable.If rated not interpretable, the attribution stage is skipped and the annotator proceeds to the next queued task.
  • Interpretability stage: During training and pilot, the justification element appears in the interpretability stage only when the task is rated not interpretable.
  • Attribution stage: The attribution stage reveals the source table and description while preserving highlighted cells from the ToTTo data.During training and pilot, the justification element is shown for all second-stage ratings.
Loading 2112.12870v2…