PaperlayerView exact revision on arXiv

Source-linked AI summary

Analyzing and Mitigating Object Hallucination in Large Vision-Language Models

Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, Huaxiu Yao

arXiv:2310.00754v2cs.LGcs.CLcs.CV

TL;DR

LVLMs often hallucinate objects in image descriptions, undermining reliable downstream use. LURE post-hoc revises potentially hallucinatory descriptions using statistical analysis of co-occurrence, uncertainty, and object position; across six open-source LVLMs, it reduces hallucination and ranks highly in GPT and human evaluations.

  • Problem

    LVLMs generate inaccurate descriptions containing nonexistent objects or omitting essential features, creating risks for downstream applications.

  • Method

    LURE is a lightweight, compatible post-hoc revisor grounded in statistical analysis of co-occurrence, uncertainty, and object position.

  • Results

    LURE significantly reduces object hallucination across multiple open-source LVLMs and outperforms strong baselines in automated, GPT, and human evaluations.

  • Takeaways & Limitations

    Post-hoc revision can mitigate object hallucination without replacing the evaluated LVLMs or relying solely on additional fine-tuning data.

  • Takeaways & Limitations

    The theoretical explanation relies on an assumption that the predictive function has a specified linear form with conditionally Gaussian features.

Abstract

from arXiv · show

Large vision-language models (LVLMs) have shown remarkable abilities in understanding visual information with human languages. However, LVLMs still suffer from object hallucination, which is the problem of generating descriptions that include objects that do not actually exist in the images. This can negatively impact many vision-language tasks, such as visual summarization and reasoning. To address this issue, we propose a simple yet powerful algorithm, LVLM Hallucination Revisor (LURE), to post-hoc rectify object hallucination in LVLMs by reconstructing less hallucinatory descriptions. LURE is grounded in a rigorous statistical analysis of the key factors underlying object hallucination, including co-occurrence (the frequent appearance of certain objects alongside others in images), uncertainty (objects with higher uncertainty during LVLM decoding), and object position (hallucination often appears in the later part of the generated text). LURE can also be seamlessly integrated with any LVLMs. We evaluate LURE on six open-source LVLMs, achieving a 23% improvement in general object hallucination evaluation metrics over the previous best approach. In both GPT and human evaluations, LURE consistently ranks at the top. Our data and code are available at https://github.com/YiyangZhou/LURE.

1 INTRODUCTION

LVLMs can generate object hallucinations—descriptions containing nonexistent objects or omitting essential features—which can mislead downstream applications. LURE addresses this through post-hoc revision grounded in three factors identified by statistical analysis: co-occurrence, uncertainty, and object position.

  • Object hallucination produces inaccurate image descriptions by including nonexistent objects or omitting essential features.
  • Hallucinatory LVLM outputs can mislead users and negatively affect applications in robotics, medical imaging, and human-computer interaction.
  • Existing approaches either align modalities, reduce co-occurrence through augmentation, or improve fine-tuning data, but direct transfer from smaller models is impractical and high-quality data collection is labor-intensive.
  • Statistical analysis identifies co-occurrence, uncertainty, and object position as three key factors underlying object hallucination in LVLMs.
  • LURE creates a hallucination revisor that converts potentially hallucinatory descriptions into more accurate ones using targeted synthetic modifications.
  • LURE is a lightweight, compatible post-hoc method evaluated across multiple open-source LVLMs and object-hallucination, GPT, and human evaluation settings.

2 WHY DO LARGE VISION-LANGUAGE MODELS EXPERIENCE OBJECT HALLUCINATION?

The paper analyzes object hallucination through co-occurrence, uncertainty, and position, combining empirical comparisons with theoretical explanations. Hallucinations are associated with spurious object associations, high uncertainty, and later generated positions, while the analysis links reduced co-occurrence and greater certainty to lower test error.

  • Object hallucination is analyzed through co-occurrence, uncertainty, and position, with theoretical analysis complementing empirical findings.
  • Co-occurrence: Spurious co-occurring patterns in training data can lead models to generate associated objects that are absent from a test image.
  • Co-occurrence: CoScore sums co-occurrence degrees across hallucinatory objects to quantify their association patterns.
  • Co-occurrence: Hallucinatory captions tend to have higher co-occurrence scores than non-hallucinatory captions.
  • Uncertainty: Objects with higher decoding uncertainty are more likely to be inaccurate, and hallucinatory objects are concentrated in the high-uncertainty range.
  • Object Position: Hallucinations predominantly occur later in generated descriptions, with high-density regions of hallucinatory objects appearing toward the sequence end.
  • Theoretical Explanation: The theoretical analysis assumes a predictive function for object presence and models object features conditionally on their image presence or absence.
  • Theoretical Explanation: Theorem 2.1 states that reducing co-occurrence can lower test misclassification error, while Theorem 2.2 indicates that sampling more certain objects can reduce test error.

3 LVLM HALLUCINATION REVISOR

LURE uses a hallucination revisor to transform potentially inaccurate LVLM descriptions into accurate ones. It trains the revisor with artificially corrupted descriptions and masks uncertain or late-position objects during inference for reevaluation.

  • Framework: LURE reconstructs accurate descriptions from potentially hallucinatory LVLM outputs using a revisor inspired by denoising autoencoders.The framework separates revisor training from deployment for rectifying generated descriptions.
  • Training data: The training data pairs images and hallucinatory descriptions with correct descriptions as targets.Hallucinatory descriptions are generated by modifying accurate descriptions with GPT-3.5.
  • Hallucination generation: LURE introduces likely co-occurring objects into accurate descriptions to create naturally occurring co-occurrence errors.GPT-3.5 deduces objects likely to co-occur in the scene instead of relying on dataset-specific co-occurrence counts.
  • Hallucination generation: LURE replaces uncertain objects and objects in later description positions with “[IDK]” placeholders during corruption.Uncertainty is estimated from noun-token entropy, while late position is determined using Index(zi) ≥η ∗Length(s).
  • Inference: During inference, LURE masks high-uncertainty or late-position objects in generated descriptions and passes the masked text to the trained revisor.The inference procedure applies the uncertainty threshold γ and position threshold η before returning the revisor’s output.

4 EXPERIMENTS

LURE is evaluated across six open-source LVLMs using automated, human, and GPT-based hallucination assessments, alongside ablations, robustness tests, and case analysis. Results show that post-hoc revision reduces object hallucination relative to strong baselines and remains effective across training choices.

  • Experimental Setup: LURE is evaluated on 5,000 MSCOCO images, with 5,000 LLaVA-150k image-text pairs used to train the hallucination revisor.The experiments use six open-source LVLMs and generally employ MiniGPT-4 as the revisor backbone.
  • Evaluation Strategies: CHAIR measures hallucination at object-instance and sentence levels, while automated evaluation also includes BLEU and CLIP score.Lower CHAIRS and CHAIRI values indicate less object hallucination.
  • Evaluation Strategies and Results: LURE significantly reduces object hallucination compared with strong baselines in automated, human, and GPT evaluations.The authors attribute the advantage partly to avoiding error propagation found in approaches such as CoT, whose object-listing step can be incorrect.
  • Analysis of LURE: LURE outperforms fine-tuned LVLMs trained with the revisor’s additional data, supporting post-hoc rectification rather than additional-data fine-tuning as the source of improvement.This comparison is reported for MiniGPT-4 in Table 3.
  • Analysis of LURE: Ablations show that co-occurrence, uncertainty, and object position each contribute to training an effective hallucination revisor.Removing any one factor reduces the strength of the revisor.
  • Robustness and Case Analysis: LURE improves over the original descriptions across MiniGPT-4, LLaMA-Adapter, and mPLUG-Owl backbones and performs strongly across uncertainty thresholds.A case study also shows more accurate depiction of primary objects while avoiding hallucinated fork and handbag objects.

5 RELATED WORK

Prior work addressed hallucination in smaller vision-language models through alignment, feature fusion, contrastive learning, and data augmentation, but these methods do not transfer directly to LVLMs because their autoregressive training differs. Recent LVLM research instead studies hallucination evaluation, detection, and higher-quality fine-tuning data.

  • Traditional VLM mitigation methods include fine-grained contrastive learning, ROI feature fusion, and data augmentation to reduce co-occurrence patterns.
  • LVLMs’ autoregressive training paradigm makes direct application of traditional VLM hallucination methods challenging.
  • Recent LVLM work addresses hallucination evaluation, detection, and higher-quality datasets for fine-tuning.
  • Obtaining many high-quality fine-tuning examples can be time-consuming and labor-intensive.

6 CONCLUSION

The paper introduces LURE, a lightweight post-hoc method for rectifying object hallucinations in LVLM-generated descriptions. LURE is grounded in co-occurrence, uncertainty, and object position, and is evaluated on six open-source LVLMs.

  • LURE is a lightweight post-hoc method designed to rectify object hallucination in descriptions generated by LVLMs.
  • LURE targets three factors associated with object hallucination: co-occurrence, uncertainty, and object position.
  • LURE’s effectiveness is evaluated on six open-source LVLMs.

REPRODUCIBILITY STATEMENT

The paper provides complete proofs and discusses theoretical assumptions, while experimental details, metrics, baselines, case demonstrations, and prompt templates are documented in appendices.

  • Complete proofs for the theoretical claims and a discussion of their assumptions are provided in Appendix B.
  • Appendices A and B.3 document the experimental setup, additional metrics, and baseline details.
  • Appendix D provides detailed case demonstrations and comparisons.
  • Template prompts used during the study are included in the appendices.

A.1 EXPERIMENTAL SETTING FOR THE HALLUCINATION ANALYSIS

The hallucination analyses use MiniGPT-4 descriptions from COCO images, with separate settings for co-occurrence, uncertainty, and object-position analysis. The appendix also documents revisor training, baseline prompts, manual annotation, and theorem-proof material.

  • Experimental Setting for Co-occurrence Analysis: Co-occurrence analysis uses 80 COCO object labels and MiniGPT-4 descriptions generated from 5,000 COCO 2014 training images.
  • Experimental Setting for the Uncertainty Analysis: Uncertainty analysis uses MiniGPT-4 descriptions of 200 COCO 2014 test images and computes uncertainty as −log p(z_i|s<i, x).
  • Experimental Setting for the Analysis of Position of Hallucinated Objects: Object-position analysis uses manually annotated MiniGPT-4 descriptions for 200 COCO 2014 test images to determine precise positions.
  • LURE training requires one A100 80G GPU and takes approximately 10 minutes.
  • GPT-3.5 generates hallucination data by proposing co-occurring objects and integrating them with uncertain objects in descriptions.
  • The appendix describes baseline strategies, GPT-3.5 prompts, manual binary annotation of hallucinated objects, and theorem-proof components.

B.2 PROOF OF THEOREM 2.2

The proof analyzes uncertainty and misclassification error under the model’s sampling assumptions, then reports that LURE outperforms baselines on several additional evaluation metrics.

  • Uncertainty analysis: The uncertainty analysis models the generated-token probability using a standard normal variable Z.The passage defines Z as a standard normal random variable.
  • Uncertainty analysis: The predicted probability decreases as the norm of β_k increases, so small predicted probabilities correspond to larger sample sizes for classes with larger feature means.This links uncertainty-based sampling to class-specific sample sizes.
  • Misclassification analysis: When total sample size is fixed, allocating larger n_k to classes with smaller ||μ*_k|| makes the average misclassification error small.The result is stated under a fixed total sample-size condition.
  • Additional evaluation: The additional evaluation considers BLEU, BERTScore, ROUGE-L, CLIP, METEOR, CIDER, and SPICE on the same dataset.These metrics cover text-generation quality and image-text matching.
  • Additional evaluation: LURE outperforms other baselines on the additional metrics, with notable CLIP Score improvement, while BLEU differences remain small and may be poorly suited to hallucination assessment.The authors attribute CLIP’s sensitivity to object-level differences and contrast it with text-style emphasis in BLEU.

B.5 ADDITIONAL RESULTS ON POPE AND MME

Additional experiments evaluate LURE on POPE and MME, using post-hoc correction before inference; both datasets show reduced hallucination after introducing LURE.

  • Benchmark evaluation: LURE is additionally evaluated on the hallucination-oriented POPE and MME benchmarks.The experiments extend evaluation beyond the main setting to two popular benchmark datasets.
  • Evaluation setup: For LURE inference, the input includes the original question, image, and description rectified by LURE.The uncorrected description is used for the “Ori + Cap” condition, whereas LURE receives its corrected description.
  • Evaluation setup: The MME hallucination subset uses binary Yes-or-No questions to assess selected perceptual and cognitive capabilities.MME contains ten perceptual and four cognitive subtasks, with a hallucination-focused subset selected for evaluation.
  • Results: LURE significantly reduces hallucination on both POPE and MME, supporting the conclusions of the main experiments.The reported reduction is qualitative in the supplied passage.

C.1.1 ADDITIONAL ANALYSIS OF OBJECT POSITIONS AND HALLUCINATIONS

Position analyses across datasets, description lengths, and larger samples find that hallucinations concentrate near sequence ends; LURE reduces factor-associated hallucination while preserving usefulness.

  • Object position analysis: Across 5,000 COCO examples, short descriptions, and ImageNet and CC evaluations, hallucinated objects concentrate toward the sequence end.The pattern holds regardless of description length.
  • Object position analysis: Generating shorter descriptions does not lower position hallucination, so combining multiple short descriptions may not necessarily improve quality.The passage explicitly cautions against this strategy.
  • Factor analysis: LURE evaluates uncertainty-, co-occurrence-, and sentence-position-based hallucination ratios against original MiniGPT-4 descriptions.The comparison uses the COCO 2014 test dataset and ratios defined for each factor.
  • Factor analysis: All three hallucination ratios decrease after applying LURE, indicating lower contributions from uncertainty, co-occurrence, and object position.The conclusion is based on Table 16.
  • Sensitivity analysis: LURE is robust within a certain range of uncertainty thresholds, where the threshold controls replacement of an object with [IDK].This is reported in the sensitivity analysis using MiniGPT-4 and LLaVA as revisors.
  • Correctness and usefulness: LURE reduces hallucinatory objects by around 56% on average while reducing correctly identified objects by approximately 1.6%.Description length changes are minor, supporting a balance between correctness and usefulness.
  • Short descriptions: LURE remains effective for concise captions generated with the prompt “Generate a short caption of the image.”The result is reported for the four best-performing LVLMs.

D.2 CASES OF OUR TRAINING DATASET

The appendix presents GPT-3.5-generated hallucination data and qualitative comparisons showing LURE removing or substituting hallucinated objects, while baselines retain distinct errors.

  • Training dataset: GPT-3.5 constructs hallucination captions by adding objects from co-occurring or uncertain object lists to original captions.The added hallucinations are marked in red in the constructed examples.
  • Rectified descriptions: LURE’s rectified examples demonstrate removal or substitution of hallucinated objects.The examples are presented as qualitative evidence of the revisor’s behavior.
  • Baseline comparison: Compared with selected strong baselines, the Revisor produces descriptions judged more accurate and detailed.The comparison is illustrated through detailed dialogue flows in Figure 10.
  • Baseline comparison: In a tennis example, the Original description hallucinates a net and cap, while CoT incorrectly infers two tennis players and propagates that error.The qualitative comparison identifies different failure modes across baselines.
  • Baseline limitations: The Teacher method remains unable to eliminate hallucinated objects reliably, and GPT rewriting can omit hallucinations or introduce errors without visual patterns.This limitation is stated for rewriting-based correction.
Loading 2310.00754v2…