PaperlayerView exact revision on arXiv

Source-linked AI summary

THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models

Prannay Kaul, Zhizhong Li, Hao Yang, Yonatan Dukler, Ashwin Swaminathan, C. J. Taylor, Stefano Soatto

arXiv:2405.05256v2cs.CVcs.AIcs.LG

TL;DR

LVLM hallucinations appear in both free-form descriptions and specific-concept answers, and existing evaluations do not capture both in the same way. THRONE uses language models to evaluate object hallucinations in open-ended descriptions; the paper reports that Type II progress does not correspond to Type I reductions and that POPE underestimates Type II hallucinations.

  • Problem

    LVLM hallucinations occur in open-ended and specific-concept responses, and reducing one type does not necessarily reduce the other.

    Source: §1 · p. 1

  • Method

    THRONE uses open-source language models to evaluate object hallucinations in free-form image descriptions with respect to a predefined object vocabulary.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    Progress in reducing Type II hallucinations did not translate to corresponding Type I reductions, and POPE underestimates Type II hallucinations.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    Type I and Type II hallucinations require separate evaluation because reducing one does not necessarily reduce the other.

    Source: §1 · p. 1

    THRONE measures false object existence within a closed class set and does not measure other hallucinations, such as wrong reasoning or attributes, without modification.

    Source: F Limitations · p. 17

Abstract

from arXiv · show

Mitigating hallucinations in large vision-language models (LVLMs) remains an open problem. Recent benchmarks do not address hallucinations in open-ended free-form responses, which we term "Type I hallucinations". Instead, they focus on hallucinations responding to very specific question formats -- typically a multiple-choice response regarding a particular object or attribute -- which we term "Type II hallucinations". Additionally, such benchmarks often require external API calls to models which are subject to change. In practice, we observe that a reduction in Type II hallucinations does not lead to a reduction in Type I hallucinations but rather that the two forms of hallucinations are often anti-correlated. To address this, we propose THRONE, a novel object-based automatic framework for quantitatively evaluating Type I hallucinations in LVLM free-form outputs. We use public language models (LMs) to identify hallucinations in LVLM responses and compute informative metrics. By evaluating a large selection of recent LVLMs using public datasets, we show that an improvement in existing metrics do not lead to a reduction in Type I hallucinations, and that established benchmarks for measuring Type I hallucinations are incomplete. Finally, we provide a simple and effective data augmentation method to reduce Type I and Type II hallucinations as a strong baseline. Code is now available at https://github.com/amazon-science/THRONE .

1 Introduction

LVLM hallucinations arise in both open-ended descriptions and specific-concept responses, and improvement on one type does not necessarily reduce the other. THRONE evaluates object hallucinations in free-form descriptions using language models, while the paper also reports findings on POPE and an instruction-data augmentation method.

1 Introduction

  • Type I hallucinations occur in open-ended responses, whereas Type II hallucinations concern specific concepts; reducing one type does not necessarily reduce the other.The paper notes that the same LVLM can give contradicting answers when evaluated for the two types.Source: §1 · p. 1
  • POPE addresses Type II hallucinations, but its negative-category undersampling gives an incomplete picture of hallucinations and underestimates their prevalence.The paper also identifies limitations in CHAIR’s text matching for current LVLM responses.Source: §1 · p. 2
  • THRONE evaluates object hallucinations in free-form image descriptions using language models to judge whether mentioned objects are implied to exist in the image.It uses a predefined object vocabulary and can distinguish implied existence from abstract mention.Source: §1 · pp. 1–2
  • THRONE combines open-source language models and voting to make evaluation accessible and mitigate single-model judging bias.The models can run on common GPUs, unlike approaches relying on closed-source commercial models subject to change.Source: §1 · p. 2
  • The paper reports that Type II progress does not translate to corresponding Type I reductions and presents data augmentation that improves Type I while maintaining or improving Type II performance.It also reports a complete POPE version to address the benchmark’s sampling issue.Source: §1 · p. 2

2 Related Work

Prior hallucination benchmarks target either short captions with text matching or restricted-answer questions, while LVLMs can generate long free-form responses. The paper contrasts these approaches with THRONE’s LM-based evaluation of Type I hallucinations.

2 Related Work

  • CHAIR uses exact text matching against object classes and synonyms, but long, diverse LVLM responses make it prone to mismatches and require manual synonym selection.A generic response can also achieve a perfect score under its metrics.Source: §2 · pp. 2–3
  • THRONE uses pretrained language models to judge object existence in free-form responses and considers both precision and recall without manual synonym curation.Unlike CHAIR’s direct matching, its approach can distinguish abstract mentions from implied object existence.Source: §2 · p. 3
  • POPE evaluates Type II object hallucinations through yes-no questions, but its balanced sampling of only three positive and three negative questions per COCO image misses potential hallucinations.The paper states that POPE dramatically underestimates Type II hallucinations.Source: §2 · p. 3
  • Concept-neutral prompts elicit Type I hallucinations, while prompts specifying a concept elicit Type II hallucinations.These are distinct response settings in the paper’s comparison of hallucination benchmarks.Source: Figure 3 · p. 3
  • LVLM architectures combine pretrained language models with image encoders and connector modules; the cited examples include Flamingo, BLIP-2, and LLaVA.The paper notes that traditional evaluation datasets do not measure the extent of hallucinations.Source: §2 · p. 3

3 THRONE

THRONE evaluates Type I hallucinations in open-ended image descriptions by using language models to judge whether mentioned objects are implied to exist, rather than relying on restricted answer formats or exact text matching. It combines those judgments across models and prompts before computing precision- and recall-based metrics.

3 THRONE

  • Unlike restricted-answer benchmarks, THRONE enumerates all classes for every image rather than subsampling classes, enabling evaluation of free-form generations against the ground-truth vocabulary.POPE and related benchmarks use restricted answer spaces; POPE also undersamples negative classes.Source: §3 · p. 3
  • THRONE uses language models to distinguish objects implied to exist from objects mentioned hypothetically, avoiding CHAIR’s exact-match errors on free-form vocabulary.Figure 4 contrasts concept-specific POPE questions with concept-neutral descriptions and illustrates why the distinction matters.Source: §3 · pp. 3–4, Figure 4 · p. 4

3.1 Evaluating Hallucinations with THRONE

  • For each image, THRONE prompts the LVLM for a detailed description, then asks an LM about every vocabulary class using the description as context.The workflow produces yes/no existence judgments without additional parsing.Source: §3.1 · p. 4
  • THRONE reports overall and class-wise precision and recall; its principal F0.5 metric weights precision twice as much as recall.Precision reflects Type I hallucinations, while recall indicates class coverage; class-wise metrics reduce skew toward common categories.Source: §3.1 · pp. 4–5

3.2 Ensuring Robustness via Ensembling

  • Because different LMs and semantically equivalent prompts can yield opposing answers, THRONE ensembles both sources of judgments.The pipeline combines N models and M question formats, requiring at least k agreeing answers and ignoring combinations without sufficient agreement.Source: §3.2 · p. 5, Figure 5 · p. 5

4 Implementation

The implementation uses public datasets and open-source language models to evaluate free-form LVLM descriptions. It applies multiple prompts and models to generate and combine object-existence judgments.

4.1 Dataset

  • COCO supplies 80 exhaustively labeled image-level categories and complex scenes suited to long descriptions; its validation set contributes 5,000 images.The evaluation runs one AQA query per image-class pair, or 400,000 queries per LVLM response set.Source: §4.1 · p. 5
  • For each image, LVLMs receive the prompt “Describe this image in detail,” and an LM judges every category in the dataset using the generated response.Objects365 is also evaluated because it is rarely used in LVLM training.Source: §4.1 · p. 5

4.2 Language Models

  • THRONE uses FLAN-T5 because its models are open-source, instruction-tuned, locally accessible, and able to follow yes/no instructions.The implementation uses Base, Large, and XL variants with 250M, 780M, and 3B parameters, respectively.Source: §4.2 · pp. 5–6

4.3 Prompt Ensembling

  • Three semantically equivalent questions are paired with the three FLAN-T5 variants, yielding nine judgments combined by unanimous voting.The prompts ask whether the object is present, implied to be present, or explicitly mentioned as present.Source: §4.3 · p. 6, §4.2 · p. 6

5 Evaluation Results

THRONE’s COCO results show that progress on Type II benchmarks does not reliably track Type I hallucinations, while its Type I rankings generalize across datasets. The evaluation also finds that POPE’s sampling misses substantial Type II errors, and reports human-oracle and sampling ablations.

5.1 Models

  • The evaluation selects publicly available LVLMs with approximately 7B-parameter language models and instruction tuning suited to generating free-form descriptions.Image encoder sizes and resolutions vary across the evaluated models.Source: §5.1 · p. 6

5.2 THRONE Results on COCO

  • THRONE and POPE scores have a Spearman rank correlation of 0.2, while THRONE and POPE-C correlate at 0.4; the best models still hallucinate about 20% of objects by class-wise precision.The paper reports that improvements on POPE and other benchmarks may be orthogonal or at odds with THRONE performance.Source: §5.2 · p. 6

5.3 THRONE Results on Objects365

  • The COCO-to-Objects365 comparison shows a Spearman rank correlation of r = 0.900; the Table 2 caption separately reports a rank correlation of 0.912.Objects365 uses a larger vocabulary and was not used to train the evaluated LVLMs; the authors conclude that COCO results indicate intrinsic Type I hallucination levels.Source: §5.3 · pp. 6–7, Figure 6 · p. 7, Table 2 · p. 7

5.4 Completing POPE for Type II Hallucinations

  • POPE-C’s exhaustive evaluation produces large, often extreme, precision and F1-score reductions across evaluated LVLMs compared with POPE.POPE samples at most 12 classes per image and at most 9 negative classes, leaving many potential hallucinations unevaluated.Source: §5.4 · p. 7, Figure 7 · p. 7
  • Follow-up LVLMs generally improve on POPE and POPE-C but show small THRONE performance reductions, suggesting Type I and Type II progress can be orthogonal.The comparison covers three pairs of model generations.Source: §5.2 · p. 6, §5.4 · p. 7

5.5 Ablations

  • Against human judgments on subsampled COCO responses, THRONE achieves 96% precision versus CHAIR’s 91%, reducing false discovery rate by more than 50%.Most estimated THRONE errors arise from COCO class definitions, such as its inclusion of computer monitors in the “tv” class.Source: §5.5 · p. 7
  • Applying the POPE sampling strategy to THRONE overestimates class-wise F0.5 (CLS) by an average of 12.3 points compared with using the complete set of classes in THRONE.Source: §5.5 · p. 8
  • Majority and all-but-one voting correlate strongly with unanimous voting, with metric correlations above 0.99 and rank correlations above 0.94.The paper uses unanimous voting with k = 9 to minimize false discovery rate.Source: §5.5 · p. 8

6 Improved Baselines

The section adds object enumeration to LLaVA visual instruction tuning as a baseline for reducing Type I hallucinations. The augmented models improve on THRONE, with smaller precision gains on POPE and POPE-C.

6.1 Visual Instruction Tuning Data Augmentation

  • Training prepends an object-listing task with approximate locations to LLaVA visual instruction samples, using COCO bounding-box annotations while leaving the remaining data and training pipeline unchanged.Negative examples are biased toward frequently co-occurring class pairs, based on a correlation matrix.Source: §6.1 · p. 8

6.2 Improved Baseline Results

  • At inference, the model first performs object enumeration before generating a response to the benchmark prompt; VisualGenome bounding-box annotations are also evaluated where available.The reported results cover THRONE, POPE, and POPE-C.Source: §6.2 · p. 8
  • Object enumeration increases THRONE classwise precision and F0.5_cls, particularly for LLaVA-v1.5, while POPE and POPE-C show small precision improvements.The table caption characterizes the THRONE gains as large and the POPE and POPE-C reductions in Type II hallucinations as small.Source: Table 3 · p. 8, §6.2 · p. 8

7 Conclusion

THRONE evaluates Type I hallucinations in free-form image descriptions, while POPE-C completes evaluation of Type II hallucinations. The paper reports limited Type I progress and proposes an augmentation baseline that reduces Type I hallucinations while maintaining or improving Type II performance.

7 Conclusion

  • THRONE benchmarks 11 publicly available LVLMs on COCO and Objects365 and finds limited progress in reducing Type I hallucinations.THRONE uses multiple LMs and prompt formats with voting, and runs open-source LMs on a single commercial GPU.Source: §7 · p. 8
  • POPE underestimates Type II hallucinations, and the paper presents POPE-C as a completed version for comparison with THRONE.Source: §7 · p. 8
  • Object-enumeration data augmentation can reduce Type I hallucinations while maintaining or improving Type II performance.Source: §7 · p. 8

A Voting Mechanism Ablation

The ablation compares three voting mechanisms and reports low ignore-label counts in almost all cases, with metrics strongly correlated across mechanisms.

A Voting Mechanism Ablation

  • Across unanimous (k = 9), simple majority (k = 5), and jury majority (k = 8), metrics are strongly correlated (> 0.99) and ignore labels are low in almost all cases.THRONE uses unanimous voting to minimize the likelihood of hallucination-judgment errors.Source: Table 4 · p. 12

B THRONE vs. POPE Sampling

POPE-style sampling underestimates Type I hallucinations relative to THRONE's exhaustive sampling. The CHAIR overview describes an object-matching approach whose instance-level precision focus and sentence-level presence measure have limitations for detailed captions.

B THRONE vs. POPE Sampling

  • POPE-style sampling underestimates mean F0.5_cls by 10.9 points versus THRONE's complete sampling, regardless of LVLM.The table caption likewise states that POPE sampling underestimates Type I hallucination prevalence regardless of LVLM.Source: B THRONE vs. POPE Sampling · p. 13, Table 5 · p. 13
  • Applying POPE-style sampling to THRONE yields a large underestimation of Type I hallucinations relative to its complete sampling method.Source: B THRONE vs. POPE Sampling · p. 13

C CHAIR Overview

  • CHAIR extracts caption objects through hard rules, maps them to MSCOCO classes using a synonym dictionary, and checks for false positives against ground-truth objects.Its ground truth combines instance-segmentation labels with objects extracted from human captions because the two sources are complementary.Source: C CHAIR Overview · p. 14
  • CHAIR_i focuses on hallucinated instances among predicted objects, while CHAIR_s counts sentences containing at least one hallucination; neither captures the full extent of hallucination in long-form text.CHAIR_i omits recall, so short, incomplete captions can receive high scores; CHAIR_s does not distinguish one hallucination from many within a sentence.Source: C CHAIR Overview · p. 14

D.1 Evaluation Method

The evaluation estimates THRONE's error rate at 4.3%, compared with CHAIR's 8.8%, using weighted agreement and disagreement cases. The section also details object-enumeration data construction and negative sampling for the improved baseline.

D.1 Evaluation Method

  • About 97% of false-positive object classes in LVLM responses are true hallucinations rather than plausible object misclassifications.Source: §D.1 · p. 15
  • THRONE's estimated error rate is 4.3%, versus CHAIR's 8.8%, using weighted agreement and disagreement case error rates.The estimated rates are weighted sums of error rates in cases where the methods agree or disagree.Source: §D.1 · p. 15

D.2 Discussion

  • Most THRONE errors reflect mismatches between an LM's class definition and COCO's; handcrafted rules could reduce errors but are deliberately avoided.For example, COCO's tv class includes computer monitors, while an LM may not treat a response implying a monitor as evidence of a tv.Source: §D.2 · p. 15

E Improved Baseline Implementation Details

E.1 Object Enumeration Implementation Details

  • The task limits each class to three instances, adds six negative classes, and selects positive instances in descending bounding-box area.The instance cap keeps sample length from pushing visual-instruction data beyond the LLaVA context length.Source: §E.1 · pp. 15–16

E.2 Negative Sampling Implementation Details

  • Negative classes are sampled using co-occurrence with classes present in the image, training the model to distinguish objects rather than rely on global context.Source: §E.2 · p. 16

E.3 Object Enumeration Data Details

  • For COCO and VisualGenome training, VisualGenome annotations are used when available and COCO annotations otherwise, without combining annotations for an image.About 33,000 of the 81,479 COCO training images are also in VisualGenome.Source: §E.3 · p. 17

E.4 Inference Details

  • The reported Table 3 results train with object enumeration and also perform that task at inference.A subsequent comparison tests omitting enumeration at inference and directly addressing the relevant task.Source: §E.4 · p. 17

E.5 Ablation Results on Improved Baseline

The ablation reports that negatives and inference-time object enumeration improve THRONE and POPE precision and F-score, while inference-time enumeration can lengthen sequences.

E.5 Ablation Results on Improved Baseline

  • Including negatives improves THRONE and POPE precision and F-score; inference-time object enumeration also improves them but can generate long sequences.Source: Table 8 · p. 17

F Limitations

THRONE measures false object-existence claims within a closed set of classes, leaving other hallucination forms and societal bias outside its current scope. The ethical discussion presents the framework as a step toward evaluating LVLMs, while noting that societal bias is not investigated.

F Limitations

  • THRONE measures false object existence within a closed class set, not out-of-vocabulary objects, incorrect scene reasoning, wrong attributes, or other biases.Evaluation also relies on curated datasets with detailed, image-level-complete object annotations because general exhaustive ground-truth lists are unavailable.Source: F Limitations · p. 17

G Ethical Considerations

  • The authors describe THRONE as a step toward measuring and mitigating LVLM hallucinations, but do not investigate societal bias.Source: G Ethical Considerations · p. 17

Related papers

Loading 2405.05256v2…