PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Hallucinated Content in Conditional Neural Sequence Generation

Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, Marjan Ghazvininejad

arXiv:2011.02593v3cs.CLcs.AI

TL;DR

Neural sequence models can produce fluent but unsupported content, creating a need for runtime faithfulness assessment. The paper proposes token-level hallucination detection trained on synthetic data and reports strong performance across MT and summarization, with downstream gains in low-resource MT.

  • Problem

    Fluent neural generation can hallucinate content unsupported by the input, while common evaluation metrics do not reliably assess faithfulness or directly identify hallucinated tokens.

  • Method

    The paper formulates hallucination detection as binary token-level sequence labeling and fine-tunes pretrained language models on automatically generated synthetic hallucinations.

  • Results

    The approach consistently outperforms strong baselines across MT and abstractive summarization benchmarks and improves low-resource MT through fine-grained token-level losses.

  • Takeaways & Limitations

    Token-level hallucination labels support diagnosis of unseen-input risks, word-level MT quality estimation, and improved training with noisy or partially erroneous targets.

  • Takeaways & Limitations

    The RoBERTa-based detector is constrained by a 512-token maximum input length, causing source-information loss for long summarization inputs.

Abstract

from arXiv · show

Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety of fluent but wrong outputs are particularly problematic, as it will not be possible for users to tell they are being presented incorrect content. To detect these errors, we propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually annotated evaluation sets for this task. We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations Experiments on machine translation (MT) and abstractive summarization demonstrate that our proposed approach consistently outperforms strong baselines on all benchmark datasets. We further demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods. We also apply our method to word-level quality estimation for MT and show its effectiveness in both supervised and unsupervised settings. Codes and data available at https://github.com/violet-zct/fairseq-detect-hallucination.

1 Introduction

The paper targets hallucinated content in fluent neural generation, proposing token-level detection that identifies unsupported output without reference texts. It introduces synthetic-data training and uses the labels to improve low-resource MT training.

  • Neural sequence models can generate fluent outputs containing hallucinated content not entailed by the input.
  • Standard metrics such as BLEU, ROUGE, and BERTScore correlate poorly with faithfulness and require reference outputs at runtime.External semantic metrics score faithfulness but do not directly identify hallucinated tokens and correlate weakly with human judgments.
  • Token-level hallucination detection predicts whether each generated token is hallucinated or faithful to the source without using a reference output.The task uses binary labels focused specifically on hallucination errors.
  • The method creates benchmark datasets for abstractive summarization and MT, then fine-tunes pretrained language models on synthetic hallucinated data.
  • Around 0.6 average F1 across benchmark datasets was achieved without human-annotated supervised training data.
  • Token-level labels support fine-grained loss truncation schemes that improve low-resource MT training and reduce hallucinations.The approach is applied to noisy targets from self-training and mined bitext.

2 Token-level Hallucination Prediction

The paper defines hallucination as unsupported generated content and operationalizes detection as binary token-level sequence labeling. It distinguishes intrinsic and extrinsic examples while annotating only whether hallucination exists.

  • A generated span is hallucinated when it is not supported by the source sequence.
  • Hallucination types: Extrinsic hallucinations add content without clear grounding in the input, such as “happily” when no corresponding source content exists.
  • Hallucination types: Intrinsic hallucinations contain incorrect information synthesized from the source, such as replacing “Mike” with “Jerry”.Multiword spans can also be intrinsic hallucinations when they reverse the source meaning.
  • The annotation scheme does not label hallucinations as intrinsic or extrinsic; it records whether hallucination occurs in generated spans.
  • Human assessment: Human assessment used source, reference, and MT output shown to three bilingual annotators, with majority labels forming the benchmark datasets.Annotators marked outputs as incomprehensible, faithful, or containing hallucinations, and tagged unfaithful tokens when needed.
  • Human assessment: Token-level annotation agreement was moderate at FK≈0.56, while sentence-level agreement was substantial at FK≈0.67.The authors attribute MT disagreement partly to distinguishing ungrammatical translations from hallucinations and understanding specialized patent text.

3 Token-level Hallucination Detection

The paper formulates token-level hallucination detection as sequence labeling and creates synthetic labeled data by inserting fluent hallucinations into target sentences. Pretrained language models are fine-tuned on these examples, with auxiliary objectives helping them generalize without reference targets.

  • Task formulation: Token-level hallucination detection predicts a binary hallucination label for every token in a generated sequence.The task uses source input and machine output rather than requiring a reference output at assessment time.
  • Synthetic data creation: Synthetic examples are built by automatically inserting hallucinated target-side tokens into bitext targets while keeping the resulting sentence fluent and close to the original.The target T is transformed into T′ with token-level labels, producing supervised pairs ((S, T′), L_T′).
  • Synthetic data creation: BART generates hallucinated targets by reconstructing noised sentences without source access, encouraging fluent insertion of new content.Words are removed from T, then BART decodes from the noised target using beam search.
  • Label assignment: Edit-distance backtracing labels target positions involved in deletions or substitutions as hallucinated and all remaining positions as faithful.Figure 2 assigns label 1 to hallucinated words.
  • Model training: A pretrained cross-lingual or monolingual language model is fine-tuned with prediction and masked-language-model losses on synthetic data.The final objective is L = Lpred + α · Lmlm, and the model can include the true target during training but omit it at test time.
  • Model training: Token dropout and paraphrase-based training reduce reliance on the true target and help the detector generalize when references are unavailable.Paraphrased targets are generated through knowledge distillation, and pseudo-labels are computed against the paraphrase rather than the original target.

4 Evaluation Tasks and Data

The evaluation covers abstractive summarization and out-of-domain machine translation, including a low-resource setting. It uses human-annotated hallucination benchmarks and compares Transformer-based systems with pretrained sequence-to-sequence models.

  • Tasks and settings: The study evaluates hallucination detection in abstractive summarization and machine translation.The summarization benchmark uses XSUM, while translation evaluation focuses on Chinese-English data in out-of-domain domains.
  • Abstractive summarization: XSUM contains 226,711 BBC article-summary pairs, with 500 test articles previously assessed across four abstractive summarization systems.The systems are PtGen, TConvS2S, TranS2S, and BERTS2S.
  • Machine translation: The translation data cover law, news, patent, and subtitles, with testing on patent and COVID-19 examples to assess out-of-domain hallucination.Training includes 870 patent parallel sentences alongside much larger law, news, and subtitles collections.
  • Models: Figure 4 depicts fine-tuning XLM-Roberta for cross-lingual tasks and Roberta for monolingual tasks on the synthetic training data.This task-specific choice matches machine translation with cross-lingual modeling and summarization with monolingual modeling.
  • Models: Experiments compare TranS2S, a standard six-layer encoder-six-layer decoder Transformer, with MBART, a multilingual denoising autoencoder fine-tuned on the training data.MBART is pretrained on large-scale monolingual corpora in many languages.

5 Experiments

Experiments show strong hallucination-prediction performance across MT and summarization, close agreement with human annotations, and benefits for low-resource MT and word-level quality estimation. Analyses identify nouns as especially frequent hallucinated tokens, while long summarization inputs remain challenging.

  • Hallucination prediction: The proposed detector ranks best among baseline methods on six MT and abstractive-summarization benchmark datasets, although the task remains unsolved.The comparison uses token-level hallucination-label F1.
  • Limitations: Summarization detection is limited by RoBERTa’s 512-token input maximum, which cuts an average of 158 source subwords and loses source information.The authors suggest long-sequence modeling as a direction for future work.
  • Hallucination prediction: Pearson correlation between annotated and predicted hallucinated-token percentages is 0.986 across the benchmark test sets.Table 2 compares human annotations with model predictions.
  • Model analysis: Pretrained-model outputs contain fewer hallucinations: BERTS2S summaries and MBART translations hallucinate less than outputs from the compared systems.The paper reports this trend for both summarization and machine translation.
  • POS analysis: Nouns are the most frequently hallucinated part of speech in both MT and summarization, while verbs also contribute substantially in abstractive summarization.Predicted POS distributions match the gold annotation distributions well.
  • Word-level quality estimation: The supervised word-level QE model outperforms the best baseline by 2 points in F1-Mult, while the unsupervised setting outperforms the third-ranked shared-task system.The supervised model uses only the labeled parallel data, whereas the unsupervised setup uses synthetic data.

6 Case Study I: Improving Self Training in Machine Translation

This case study uses token-level hallucination predictions to reduce the impact of noisy pseudo-parallel data in low-resource MT. Fine-grained loss control improves translation quality and reduces hallucinations relative to strong baselines.

  • Method: Token-level hallucination labels enable fine-grained training objectives for noisy self-training and parallel data.The study contrasts token-level control with methods that discard entire sentence pairs or examples.
  • Self-training: Self-training uses a teacher trained on labeled bitext to generate pseudo-parallel data for student training.Performance can suffer when the teacher is not sufficiently high quality, especially in low-resource settings.
  • Method: The proposed methods either discard losses for predicted hallucinated tokens or mask their decoder hidden states.Both methods retain the rest of partially erroneous outputs for training.
  • Analysis: Removing hallucinated target words before training underperformed, likely because it produced too many ungrammatical target sentences.This alternative differs from loss truncation, which preserves useful non-hallucinated portions of outputs.
  • Results: 1.7 BLEU is the further improvement over self-training achieved by the best result, while also obtaining the best BLEU and BLEURT and the largest hallucination reduction.The comparison is reported on the CWMT2017 test set against strong baseline methods.

7 Case Study II: Improving Corpus Filtering for Low-Resource MT

Token-level hallucination labels are used to improve training on noisy, low-resource parallel data. With loss truncation, the method outperforms the strongest shared-task system on both language pairs, although severe misalignment limits performance for larger Nepali-English datasets.

  • Token-level hallucination labels enable better use of noisy parallel data for neural machine translation training.The approach targets selected parallel corpora that may still contain misaligned segments.
  • Experimental Setup: The WMT19 task evaluates Nepali-English and Sinhala-English systems trained from noisy corpora subsampled at 1M and 5M English words.The shared task also provides clean parallel data that cannot be used to train the final NMT system.
  • Method: The method trains a token-level hallucination predictor on combined Nepali-English, Sinhala-English, and Hindi-English data before labeling target-side hallucinations.Scores are used to select top-scored datasets ranging from 1M to 10M English tokens.
  • Results and Analysis: 7.4 for Ne-En and 8.11 for Si-En are the new best flores test-set BLEU scores achieved with token-level loss truncation.The method further improves the state-of-the-art system across varying training-data sizes for both language pairs.
  • Results and Analysis: 0.14 vs. 5.18 at 10M training data shows the method retaining reasonable Si-En performance where the baseline does not.The 10M corpus is described as very noisy.
  • Results and Analysis: For Ne-En datasets larger than 2M, both systems’ performance drops significantly, possibly because many sentence pairs are misaligned.The suspected misalignment includes sources that are not Nepali and targets that are not English.

8 Conclusions

The paper introduces token-level hallucination detection, human-annotated benchmarks, and unsupervised detector learning. It also shows that hallucination labels support fine-grained losses for machine translation and downstream quality-estimation and noisy-translation tasks.

  • The work proposes token-level hallucination detection, human-annotated benchmark datasets, and unsupervised learning of hallucination detectors.
  • Hallucination labels define fine-grained losses that improve machine-translation training.
  • The method demonstrates downstream effectiveness in word-level quality estimation and noisy neural machine translation.

A Human Evaluations

The human evaluation process combines bilingual annotation, training and practice, blinded model identities, and explicit rules for labeling unsupported content. Annotations use minimal hallucinated spans and operate on raw sentence forms.

  • Setup: Three bilingual speakers annotate Chinese-English evaluation data drawn from a multi-domain dataset and the TICO COVID-19 translation benchmark.The evaluation set includes 150 multi-domain test sentences and samples from TICO domains.
  • Annotation Guidelines and Process: Annotators receive pilot, practice, and education sessions designed to establish a clear workflow for final blind-set evaluation.
  • Annotation Guidelines and Process: Unsupported tokens or spans are colored and the sentence is marked hallucinated; incomprehensible translations receive a separate label.
  • Annotation Guidelines and Process: Translations whose tokens are all entailed by the source are marked faithful.
  • Annotation Guidelines and Process: Annotators minimally mask hallucinated spans so deleting or replacing them can make the generation faithful to the source.They distinguish hallucinations from bad translations and ignore sentence domain when labeling.
  • Annotation Guidelines and Process: Surface-form differences such as capitalization are not labeled hallucinations when the underlying meaning remains faithful to the source.
  • Annotation Guidelines and Process: Annotations are performed on raw sentences, allowing punctuation, prepositions, and stop words to be included in hallucinated spans.Subword predictions are converted to labels on raw sentences at test time.
  • Post-processing: Majority voting aggregates the three annotators’ token labels, while incomprehensible translations are removed before aggregation.The same aggregation procedure is applied to evaluation data from Maynez et al. (2020).

B Training of NMT models

The paper trains Transformer-based systems and hallucination detectors using subword tokenization, synthetic noise, and annotated evaluation sets. It evaluates detection across MT and summarization outputs.

  • Training setup: TranS2S uses separate 32k BPE vocabularies for Chinese sources and tokenized English targets.
  • Training setup: MBART uses its contained sentence-piece dictionary on raw Chinese and English data.
  • Training setup: The generation models use fairseq Transformer implementations, with a base model for TranS2S and a large model for MBART.
  • Hallucination detection: The detector predicts hallucination labels at each generated position and converts subword predictions to word labels when any subword is hallucinated.
  • Hallucination detection: Synthetic data uses random masking, replacement, and mask insertion, while pretrained language models are fine-tuned for hallucination prediction.
  • Evaluation: Evaluation covers MT and summarization outputs, including hallucination-label triplets, partially hallucinated teacher outputs, and POS-based analyses.

D Ablation Studies

Ablations examine reference usage and synthetic-data construction, while baseline comparisons cover abstractive summarization hallucination detection. The analyses also identify strong non-neural alternatives.

  • Reference ablation: For summarization, word dropout on the reference is crucial, whereas removing the reference at training time produces poor performance.
  • Reference ablation: For MT, reference-token dropout does not significantly affect performance because paraphrased targets make synthetic hallucinations less reference-like.
  • Synthetic-data ablation: Distillation data is tested as an alternative target form for generating synthetic data in the MT ablation.
  • Baseline comparisons: Table 10 presents annotations and model predictions, with [0] indicating faithful words and [1] indicating hallucinated words.
  • Baseline comparisons: Synonym-based and string-matching methods are strong and effective baselines for monolingual summarization token-level hallucination prediction.

E.2 Analysis on Part-of-speech tags and with-in Group Hallucination Percentage

The analysis compares hallucination prevalence across part-of-speech groups and finds task-specific patterns. Model predictions align well with gold annotations, while partially hallucinated outputs retain useful faithful content and some prediction inaccuracies remain.

  • POS analysis: In MT, nouns are the most likely hallucinated words, while in summarization cardinal numbers are most likely hallucinated.
  • POS analysis: Model predictions align well with gold annotations for the percentage of hallucinated words within each POS tag.
  • Partially hallucinated outputs: Teacher-model outputs can contain partially hallucinated translations whose faithful portions remain useful for training.
  • Prediction limitations: The detector generally performs well but can be inaccurate when translations contain spelling errors, and some annotation errors were identified.
Loading 2011.02593v3…