Source-linked AI summary
On Hallucination and Predictive Uncertainty in Conditional Language Generation
Yijun Xiao, William Yang Wang
TL;DR
Deep neural NLG models can produce unsupported facts, while existing explanations have largely been examined separately by task. This paper connects hallucination with predictive uncertainty across image captioning and data-to-text generation and introduces uncertainty-aware beam search. Higher uncertainty corresponds to more hallucination, and epistemic uncertainty supports better performance–hallucination trade-offs than aleatoric or total uncertainty.
Problem
Deep neural NLG models can hallucinate unsupported facts, while existing explanations have been examined separately across tasks, leaving no systematic cross-task explanation.
Method
The paper studies hallucination–predictive-uncertainty relationships in image captioning and data-to-text generation and proposes uncertainty-aware beam search for decoding.
Results
Higher predictive uncertainty corresponds to greater hallucination likelihood, and epistemic uncertainty is more indicative than aleatoric or total uncertainty.
Takeaways & Limitations
Uncertainty decomposition helps the proposed beam-search variant achieve better trade-offs between standard generation performance and hallucination reduction.
Takeaways & Limitations
Token-level uncertainty may be restrictive because uncertainty depends on the prediction context, and limiting total uncertainty may shorten generations and lower quality.
Abstract
from arXiv · showhide
Despite improvements in performances on different natural language generation tasks, deep neural models are prone to hallucinating facts that are incorrect or nonexistent. Different hypotheses are proposed and examined separately for different tasks, but no systematic explanations are available across these tasks. In this study, we draw connections between hallucinations and predictive uncertainty in conditional language generation. We investigate their relationship in both image captioning and data-to-text generation and propose a simple extension to beam search to reduce hallucination. Our analysis shows that higher predictive uncertainty corresponds to a higher chance of hallucination. Epistemic uncertainty is more indicative of hallucination than aleatoric or total uncertainties. It helps to achieve better results of trading performance in standard metric for less hallucination with the proposed beam search variant.
1 Introduction
Deep neural NLG models improve standard metrics but can hallucinate unsupported facts, limiting practical use where accuracy matters. The paper connects hallucination to predictive uncertainty and studies uncertainty-aware decoding as a cross-task mitigation approach.
- Hallucination occurs when NLG models generate description tokens unsupported by source inputs, damaging applicability where information accuracy is vital.
- Prior work studies hallucination separately across image captioning, data-to-text generation, summarization, and neural machine translation, with task-specific explanations.Proposed explanations include visual misclassification, language-prior over-reliance, representation-reference misalignment, and domain shift.
- The paper proposes predictive uncertainty as a common theme across hallucination explanations in conditional NLG tasks.Predictive uncertainty is quantified through entropy of token probability distributions, including aleatoric and epistemic components.
- The study empirically investigates hallucination and predictive uncertainty in image captioning and data-to-text generation.It frames the analysis across two conditional NLG tasks rather than a single task-specific explanation.
- The proposed uncertainty-aware beam search reduces hallucination by incorporating predictive uncertainty into decoding.The approach demonstrates that lowering uncertainty can lead to less hallucination.
- Uncertainty decomposition improves the trade-off between hallucination reduction and generation performance.The paper reports better trade-offs when distinguishing uncertainty components.
2 Hallucination and Predictive Uncertainty
This section formalizes hallucination as context-dependent unsupported generation and relates it to predictive uncertainty in conditional language generation. It distinguishes aleatoric and epistemic uncertainty and motivates uncertainty-aware decoding.
- 2.1 Hallucination Probability: Hallucination is generation of false information unsupported by the input, and its probability depends on the current context and next-token distribution.The context includes the source input and previously generated tokens; task-specific heuristics may identify hallucinated tokens.
- 2.2 Relationship with Predictive Uncertainty: Predictive uncertainty is measured with token-level entropy, whose sources include uncertainty about suitable descriptions and probability assigned to unsuitable tokens.The latter source is directly related to hallucination probability, although the relationship need not be monotonic.
- 2.2 Relationship with Predictive Uncertainty: Higher predictive uncertainty intuitively increases the probability mass assigned to unsuitable tokens, motivating an empirical study of uncertainty and hallucination.Near-zero hallucination probability requires a near-zero value of the uncertainty source associated with unsuitable tokens.
- 2.3 Uncertainty Decomposition: Aleatoric uncertainty concerns inherent data or measurement uncertainty, whereas epistemic uncertainty concerns uncertainty in model weights.The study investigates whether these uncertainty types have the same relationship with hallucination.
- 2.3 Uncertainty Decomposition: Using Bayesian notation, aleatoric uncertainty is the expected conditional entropy at fixed weights, while epistemic uncertainty is the difference between total and aleatoric uncertainty.Deep ensembles estimate these components; aleatoric uncertainty reflects prediction spread, while epistemic uncertainty reflects agreement among models.
3 Case Study: Image Captioning
In image captioning, higher predictive uncertainty is associated with more object and action hallucination across models. Epistemic uncertainty is more indicative of hallucination than aleatoric uncertainty.
- 3.1 Hallucination Probability at Different Uncertainty Levels: Higher predictive uncertainty corresponds to higher object hallucination percentages across all four image-captioning models.The Transformer has the highest hallucination chance at high uncertainty, but the lowest overall hallucination percentage.
- 3.1 Hallucination Probability at Different Uncertainty Levels: Higher predictive uncertainty also makes generated action tokens more likely to be hallucinated.Action predictions were binned by uncertainty, with approximately 400 annotated verbs per model.
- 3.2 Results and Discussions: Highly uncertain BUTD predictions often involve unusual textures, visually similar features, or blurred images.The examples are cherry-picked and cover object and action predictions.
- 3.2 Results and Discussions: Both epistemic and aleatoric uncertainty are weakly correlated with hallucination, but epistemic uncertainty is more indicative across all four models.
4 Case Study: Data-to-text Generation
The data-to-text case study examines whether predictive uncertainty tracks generation quality and hallucination on ToTTo. Higher uncertainty is associated with lower fluency and more hallucinations, while medium uncertainty yields the greatest coverage beyond references.
- 4.1 Generation Quality and Average Uncertainty: The ToTTo study evaluates generated sentences by average predictive uncertainty using BLEU, fluency, faithfulness, and coverage.Candidates in the top, bottom, and middle 5% of uncertainty are compared on the validation set.
- 4.2 Results and Discussions: Higher average predictive uncertainty is associated with lower quality and a higher chance of hallucination or unfaithfulness to input tables.The evaluation uses a sequence-to-sequence model trained on highlighted subtables with metadata.
- 4.2 Results and Discussions: 16.9% of medium-uncertainty sentences covered more table facts than the references, compared with 4.7% for high and 7.7% for low uncertainty.Low-uncertainty sentences tend to include fewer but more confident facts.
5 Reducing Hallucination
The section introduces uncertainty-aware beam search, which penalizes predictive uncertainty during decoding to reduce hallucination, and evaluates its trade-offs in image captioning and data-to-text generation.
- 5.1 Uncertainty-Aware Beam Search: UABS subtracts a weighted predictive-uncertainty penalty from candidate log-probability scores before selecting the top-B beam candidates.The penalty balances log probability against aggregated uncertainty during decoding.
- 5.1 Uncertainty-Aware Beam Search: Larger λ selects candidates with smaller predictive uncertainty but can reduce generated-sentence log probabilities and caption quality.The authors therefore expect a trade-off between generation quality and hallucination probability.
- 5.2 Image Captioning Results: Across models, increasing the penalty weight decreases both CHAIRi and CIDEr, indicating less hallucination alongside lower standard caption quality.Lower CHAIRi means fewer hallucinated objects, while higher CIDEr indicates better caption quality.
- 5.2 Image Captioning Results: Penalizing epistemic uncertainty produces higher CIDEr than penalizing aleatoric or total uncertainty, while reducing hallucination rates as λ increases.Epistemic UABS also keeps generic-response rates low, unlike heavily penalized aleatoric UABS.
- 5.2 Image Captioning Results: Aleatoric UABS increases generic responses from 1.0% to 28.4% when λ rises from 1.0 to 4.0, whereas epistemic UABS keeps generic-response rates low.The aleatoric increase partly explains lower CIDEr despite approximately unchanged sentence length.
- 5.3 Data-to-Text Results: On ToTTo, larger epistemic penalties produce shorter, more confident predictions by dropping statements the model considers less certain, including some correct facts.Faithfulness first increases and then returns to regular beam-search levels as λ increases.
6 Related Work
Related work documents hallucination across conditional NLG tasks and develops task-specific explanations, while uncertainty quantification provides tools for decomposing predictive uncertainty.
- Hallucination: Hallucination has been studied in image captioning, data-to-text generation, abstractive summarization, and neural machine translation.Prior explanations include visual misclassification, language-prior reliance, data misalignment, and domain shift.
- Hallucination: Prior work attributes hallucination to visual misclassification and language priors, misalignment between meaning representations and references, or domain shift, depending on the task.These studies generally examine hallucination within specific tasks.
- Uncertainty quantification: Bayes by Backprop, Monte Carlo dropout, and deep ensembles are established approaches for estimating uncertainty in neural models.Related work also studies epistemic and aleatoric uncertainty and prediction entropy across vision, time series, and NLP.
7 Discussion and Conclusions
The paper finds that higher predictive uncertainty is associated with more hallucination and that epistemic uncertainty is especially informative for uncertainty-aware decoding. It also identifies token-level analysis and quality costs as important limitations.
- 7 Discussion and Conclusions: In image captioning and data-to-text generation, predictions with higher uncertainty are more prone to hallucination, with epistemic uncertainty more indicative than aleatoric uncertainty.The conclusion also compares epistemic penalization favorably with aleatoric or total uncertainty.
- 7 Discussion and Conclusions: Uncertainty decomposition helps UABS achieve a better performance–hallucination trade-off, specifically when epistemic uncertainty is penalized.The method incorporates uncertainty into decoding to reduce hallucination.
- Limitations: Token-level uncertainty analysis may be restrictive because uncertainty depends on the current prediction context rather than only the predicted token.The authors note that hallucinated information can still be produced by a highly confident model.
- Limitations: UABS can reduce hallucination by limiting total uncertainty, but this may produce shorter generations and lower generation quality.More sophisticated uncertainty-aware training and decoding methods are identified as future work.