Source-linked AI summary
Object Hallucination in Image Captioning
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, Kate Saenko
TL;DR
Image captioning models can hallucinate objects, while standard sentence metrics may not fully reflect image relevance. The paper introduces CHAIR and consistency measures to evaluate hallucination across MSCOCO models, finding that stronger sentence-metric performance does not always mean less hallucination and that more-hallucinating models tend toward language-prior-consistent errors.
Problem
Standard captioning metrics may not fully capture whether generated captions are relevant to image content, leaving object hallucination insufficiently measured.
Method
The paper proposes CHAIR using ground-truth object and sentence annotations, and uses image- and language-consistency scores to analyze hallucination across diverse MSCOCO models.
Results
Models with the best standard sentence-metric scores do not always hallucinate less, while models that hallucinate more tend to make errors consistent with a language model.
Takeaways & Limitations
Image relevance should be considered alongside cross-entropy and standard sentence metrics when evaluating captioning models.
Abstract
from arXiv · showhide
Despite continuously improving performance, contemporary image captioning models are prone to "hallucinating" objects that are not actually in a scene. One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance. In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination. We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination. We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models. Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors.
1 Introduction
Object hallucination occurs when captioning models describe objects absent from an image, while standard sentence metrics may fail to penalize it. The paper proposes CHAIR to measure image relevance and examines model causes and metric limitations.
- Object hallucination occurs when a captioning model generates an object that is not present in the image.
- Although captions need not mention every object, describing absent objects is less preferable to humans, especially for people who value correctness over coverage.
- The paper proposes CHAIR, an image-relevance metric that evaluates generated captions against ground-truth objects and sentences.
- The authors analyze hallucination across captioning architectures and learning objectives, and find that strong standard-metric performance does not always correspond to lower hallucination.
- The paper investigates visual misclassification and language priors as likely causes, finding that more-hallucinating models make errors more consistent with a language model.
- Standard sentence metrics do not fully capture hallucination, whereas CHAIR uses image object labels to identify discrepancies and complements those metrics in measuring human preference.
2 Caption Hallucination Assessment
CHAIR measures whether generated object words correspond to objects annotated in the image, using both segmentation labels and reference captions. The paper also defines image and language consistency measures to analyze whether hallucination errors align more with visual evidence or language-model predictions.
- 2.1 The CHAIR Metric: CHAIR evaluates the image relevance of captions using ground-truth object segmentations and sentence annotations.
- 2.1 The CHAIR Metric: CHAIRi measures the fraction of mentioned object instances that are hallucinated, while CHAIRs measures the fraction of sentences containing a hallucinated object.
- 2.1 The CHAIR Metric: The study restricts analysis to 80 MSCOCO objects and maps tokenized, singularized words and synonyms to those object categories.
- 2.1 The CHAIR Metric: Using both annotation sources captures naming variation and avoids treating unmentioned scene objects as hallucinations; using only one source produces higher hallucination rates.
- 2.2 Image Consistency: Image consistency averages P(w|I) for MSCOCO objects, with higher values indicating errors more consistent with an image-only model.
- 2.3 Language Consistency: Language consistency uses 1/R(w_t), where R(w_t) is the language model's prediction rank; higher values indicate errors more consistent with preceding words.
3 Evaluation
Across MSCOCO models, hallucination varies with architecture, training objective, and test distribution, while standard caption metrics do not reliably reflect image correctness. Analysis links hallucination to language priors, visual integration, and downstream sentence changes.
- Which Models Are More Prone To Hallucination?: 7.4%–17.7% of Karpathy Test captions contain hallucinated objects, and hallucination consistently increases on harder test distributions.For TopDown, CHAIRs rises from 8.4% to 12.1% and CHAIRi from 6.0% to 9.1% on the Novel Object split.
- Which Models Are More Prone To Hallucination?: Attention and bounding-box features generally reduce hallucination, whereas self-critical training frequently increases it.Deconstructed models suggest spatially local feature maps, rather than attention itself, account for much of the attention-based gain.
- Which Models Are More Prone To Hallucination?: Hallucinated objects often follow contextual trigger words, with furniture—especially dining tables—among the most frequent hallucination categories.Objects can also be hallucinated out of context, such as a bed in a bathroom.
- What Are The Likely Causes Of Hallucination?: Models with less hallucination make errors more consistent with image models, while models with more hallucination align more with language models.On the Robust split, image consistency decreases and language consistency remains similar across models, consistent with bias toward learned language associations.
- How Well Do The Standard Metrics Capture Hallucination?: 47.3% of words following a hallucinated object differ between TopDown and TD-Restrict, showing effects beyond the incorrect object name.TD-Restrict prevents detected absent MSCOCO objects from being generated, enabling direct comparison with TopDown.
- How Well Do The Standard Metrics Capture Hallucination?: SPICE correlates more strongly with absence of hallucination than METEOR or CIDEr for the FC model, at 0.32 versus around 0.25.Metric scores can still favor captions containing hallucinated objects over more image-correct alternatives.
4 Discussion
The analysis finds that object hallucination is substantial and is not reliably reflected by standard sentence metrics. Visual access and image consistency are associated with less hallucination, motivating image relevance as an additional guide for captioning design and training.
- 5.5% to 13.1% of MSCOCO objects are hallucinated by current captioning models.
- Standard sentence metrics can disagree with hallucination rates, including when similar CIDEr, SPICE, or METEOR scores accompany different numbers of hallucinated objects.Self-critical loss increases CIDEr while also increasing hallucination.
- Attention lowers hallucination, but much of its apparent gain may come from access to underlying convolutional features rather than attention itself.
- Models with stronger image consistency frequently hallucinate fewer objects, indicating that visual processing is relevant to avoiding hallucination.
- Captioning design and training should consider image relevance alongside cross-entropy loss and standard sentence metrics.CHAIR evaluates hallucination, while other image relevance metrics could also incorporate missed salient objects.