PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation and Analysis of Hallucination in Large Vision-Language Models

Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, Jitao Sang, Haoyu Tang

arXiv:2308.15126v3cs.LGcs.AIcs.CLcs.CV

TL;DR

LVLMs can produce information that does not align with visual input, and object-based evaluation may not reflect hallucinations in real-world descriptions. The paper proposes HaELM, an LLM-based evaluator using reference captions and LVLM responses, and reports performance comparable to ChatGPT. It also analyzes hallucination factors and offers mitigation suggestions.

  • Problem

    LVLMs can generate information that does not align with visual input, and object-based evaluation may be weakly correlated with hallucinations in real-world descriptions.

    Source: §1 · pp. 1–2

  • Method

    HaELM fine-tunes an LLM to evaluate LVLM responses using image reference descriptions, and the paper defines its target setting as image-description requests.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    HaELM achieves performance comparable to ChatGPT and aligns with human annotations.

    Source: §1 · p. 2, §7 · p. 8

  • Takeaways & Limitations

    The authors use HaELM to analyze factors contributing to LVLM hallucinations and offer suggestions to mitigate them.

    Source: §1 · p. 2, §7 · p. 8

    HaELM and ChatGPT fall short of human-level evaluation because language models use reference captions in place of images and cannot truly comprehend image content.

    Source: §8 · p. 9

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) have recently achieved remarkable success. However, LVLMs are still plagued by the hallucination problem, which limits the practicality in many scenarios. Hallucination refers to the information of LVLMs' responses that does not exist in the visual input, which poses potential risks of substantial consequences. There has been limited work studying hallucination evaluation in LVLMs. In this paper, we propose Hallucination Evaluation based on Large Language Models (HaELM), an LLM-based hallucination evaluation framework. HaELM achieves an approximate 95% performance comparable to ChatGPT and has additional advantages including low cost, reproducibility, privacy preservation and local deployment. Leveraging the HaELM, we evaluate the hallucination in current LVLMs. Furthermore, we analyze the factors contributing to hallucination in LVLMs and offer helpful suggestions to mitigate the hallucination problem. Our training data and human annotation hallucination data will be made public soon.

1 Introduction

LVLM hallucinations are incorrect responses that do not align with visual input, while existing evaluation approaches may not reflect hallucinations in real-world descriptions. The authors propose HaELM, an LLM-based evaluator, and use it to analyze hallucination factors and suggest mitigations.

1 Introduction

  • LVLM hallucination is incorrect information that does not align with visual input, a problem with potential substantial consequences.LVLMs inherit hallucination pathways associated with both language models and visual-language pre-trained models.Source: §1 · p. 1
  • Object-based evaluation is weakly correlated with real-world hallucinations: LVLMs answered yes to over 80% of absent-object queries, but fewer than 10% of descriptions included those objects.The authors argue that idealized prompts can influence responses and advocate evaluating real-world scenarios, where responses are too complex for traditional matching methods.Source: §1 · p. 2
  • HaELM achieves performance comparable to ChatGPT and offers low cost, reproducibility, privacy preservation, and local deployment.The framework fine-tunes LLaMA with LoRA and uses image reference descriptions to assess LVLM responses.Source: §1 · p. 2
  • The authors use HaELM to evaluate current LVLMs, analyze factors contributing to hallucinations, and offer suggestions to mitigate them.They describe their work as the first use of an LLM for hallucination evaluation in LVLMs.Source: §1 · p. 2

2 Background

The background introduces language and vision-language models before describing hallucination as a limited-focus problem in LVLM research. It notes that LVLM hallucinations involve mismatches between visual and textual modalities.

2.1 Large Language Model

  • LLMs gained attention for zero-shot capabilities, while instruction fine-tuning such as InstructGPT uses human feedback to improve instruction following.GPT-3 is cited as demonstrating powerful zero-shot capabilities on previously unseen tasks.Source: §2.1 · p. 2

2.2 Large Vision-Language Model

  • LVLM approaches either use visual experts with ChatGPT or align visual tokens with pretrained LLMs and instruction-tune them.Some work seeks to improve LVLM performance through more diverse instructions and larger instruction-tuning datasets.Source: §2.2 · pp. 2–3

2.3 Hallucinations in LLMs and LVLMs

  • Hallucination research has given limited attention to LVLMs; their errors involve visual-textual mismatch, including absent objects common in training data.The section also notes that insufficient visual constraints contribute to LVLM illusions, which may mislead decisions or produce toxic information.Source: §2.3 · p. 3

3 Motivation

The section questions whether object-based prompts measure hallucinations in real-world image descriptions. Its comparison suggests that yes-responses to absent objects often reflect judgment bias rather than hallucination in descriptions.

3 Motivation

  • Object-based evaluation queries whether an image contains a commonly hallucinated item and treats a yes answer for an absent item as hallucination.The feasibility experiment queried mPLUG-Owl on 100 randomly selected MS-COCO 2014 images using the ten most frequently hallucinated absent items.Source: §3 · p. 3
  • The authors speculate that instruction-tuning data may bias LVLMs toward affirming judgment-type queries.They manually examined responses to “Describe this image” to check whether queried items actually appeared as hallucinations.Source: §3 · pp. 3–4
  • Over 80% of absent-object queries received yes answers, while only 10% of image descriptions included hallucinations for those queried items.The authors interpret this discrepancy as evidence that object-based evaluation exploits judgment bias rather than reflecting actual hallucinations.Source: §3 · pp. 3–4

4 Method

HaELM evaluates image-description responses by comparing them with reference captions. Its data collection combines manually annotated real hallucinations with ChatGPT-generated samples, then fine-tunes an LLM-based evaluator.

4.1 Problem Definition

  • Real-world hallucination evaluation checks for discrepancies between image content and LVLM responses under potential human requests; this paper focuses on image descriptions.Source: §4.1 · p. 4

4.2 HaELM

  • The authors manually annotate hallucinations in image-description responses, then use ChatGPT prompts based on reference captions to generate bulk samples.Human similarity assessment and iterative prompt modification align simulated hallucination patterns with realistic ones.Source: §4.2 · p. 4
  • ChatGPT generates non-hallucination descriptions from reference captions, with instructions to include only objects present in those captions.Source: §4.2 · p. 4
  • HaELM is fine-tuned autoregressively on collected data and judges LVLM responses using reference captions and the responses as inference inputs.Source: §4.2 · pp. 4–5
  • After data collection and training, HaELM can be reused at lower cost than ChatGPT and deployed locally to preserve data privacy.The framework is built on an open-source LLM.Source: §4.2 · p. 5

5 Experiments

Experiments compare HaELM with ChatGPT on human-annotated hallucination responses, then use HaELM to evaluate hallucinations across LVLMs and generation settings. They report comparable evaluation performance alongside prompt sensitivity and generation-setting associations with hallucination.

5 Experiments

  • The experiments use MS-COCO 2014 images, including 10,000 training samples for collecting simulated responses and all 5,000 test samples for evaluating LVLM hallucinations.Human-annotated captions serve as reference captions; manually annotated real hallucination data was not used for training.Source: §5 · p. 5

5.1 Evaluation on HaELM

  • 61% accuracy for HaELM versus ChatGPT’s 64% on human-annotated responses; HaELM reaches 95% of ChatGPT’s level.The evaluation covers hallucination, non-hallucination, and overall responses.Source: §5.1 · p. 6, Table 1 · p. 6
  • On human-annotated evaluation data, HaELM’s average F1 scores across the three LVLMs are 88%, 99%, and 88%, respectively, and it outperforms ChatGPT on several metrics.HaELM has lower recall for hallucination responses.Source: §5.1 · p. 6, Table 2 · p. 6
  • HaELM requires 3.8 hours and $4.3 for data collection and training, saving 1.4 hours and $6.6 per evaluation compared with ChatGPT; savings become more significant when multiple evaluations are needed.Source: §5.1 · p. 6

5.2 Evaluation on Hallucination

  • Across prompts, LLaVA has the lowest hallucination and prompt sensitivity among the three LVLMs, while MiniGPT-4 and mPLUG-Owl show severe hallucination and strong prompt dependence.The authors note prior manual annotations found LLaVA performed worst in various aspects, and describe a trade-off between model performance and hallucination.Source: §5.2 · p. 7, Table 4 · p. 7
  • With mPLUG-Owl, increasing maximum generation length strengthened hallucination; hallucinations tended to occur later in responses, though they could also appear earlier.The authors suggest truncating responses to obtain relatively accurate results, while describing the later-occurrence pattern as a trend rather than a rule.Source: §5.2 · p. 7, Table 5 · p. 7
  • As top-K sampling K increases, hallucination becomes more severe, presenting a trade-off between response diversity and hallucination.The authors suggest random sampling may select tokens less aligned with the visual input, with resulting factual errors rationalized under LLMs.Source: §5.2 · pp. 7–8, Table 6 · p. 8

6 Discussion

The discussion uses gradient-based attention visualization to examine how generated tokens attend to prior tokens and the image. In one hallucination example, attention to the image is minimal, motivating a possible future approach of penalizing attention that deviates from it.

6 Discussion

  • Gradient variations, computed using the generated response as a label for backpropagation to input embeddings, are normalized to visualize each generated token’s attention to earlier tokens and the image.Random sampling is disabled to stabilize generation.Source: §6 · p. 8
  • During the hallucinated token sequence “12,” attention to the image is minimal, while the tokens attend mainly to a preceding space token or non-relevant regions.The authors suggest frequent training co-occurrence may have taught a biased false correlation that diverts attention from the image.Source: §6 · p. 8
  • The visualization suggests penalizing attention that deviates from the image as one possible approach to addressing hallucinations, which the authors leave for future work.Source: §6 · p. 8

7 Conclusion

The paper proposes HaELM, an LLM-based framework for evaluating hallucination in real-world LVLM scenarios, and reports performance comparable to ChatGPT. It also analyzes hallucination causes and offers mitigation suggestions.

7 Conclusion

  • The paper proposes HaELM, an LLM-based hallucination evaluation framework designed for real-world scenarios.Source: §7 · p. 8
  • Experiments demonstrate HaELM’s performance is comparable to ChatGPT’s.Source: §7 · p. 8
  • The paper analyzes causes of LVLM hallucinations, offers mitigation suggestions, and reports visualizations with possible implications for future research.Source: §7 · p. 8

8 Limitations

The authors identify limits in both hallucination evaluation and the paper’s analysis of LVLM hallucinations. The appendix also presents parameter settings for evaluated models.

8 Limitations

  • HaELM and ChatGPT fall short of human-level evaluation, in part because reference captions substitute for images and current LVLM evaluators commonly hallucinate.The authors therefore consider language models the optimal choice for hallucination evaluation at this stage.Source: §8 · p. 9
  • The paper does not address the root cause of LVLM hallucinations; substantive methods to reduce learned hallucination patterns remain future work.The authors propose building such methods through analysis of hallucination triggers and describe this as challenging.Source: §8 · p. 9

A Evaluated LVLMs

B Training Details

The training details describe the effective batch-size setup and LoRA choices, alongside parameter settings and detailed object-based evaluation results.

B Training Details

  • The appendix presents detailed LoRA fine-tuning settings in Table 8.Source: B Training Details · p. 10
  • Because 32GB Tesla V100 memory was insufficient to accommodate batch size 64, the authors used batch size 8 and 8 gradient-accumulation steps, achieving equivalent batch size 64.When "train on input" is turned off, the selfregressive loss no longer considers the input.Source: B Training Details · p. 10
  • Tables 9–11 provide detailed object-based validity assessment results for mPLUG-Owl, MiniGPT-4, and LLaVA, respectively.Source: Table 9 · p. 11, Table 10 · p. 11, Table 11 · p. 11
  • The LoRA settings follow Vicuna’s settings, with the Q and V matrix weights replaced.Source: B Training Details · p. 11

C Additional Evaluation on Hallucination

The additional evaluation reports more hallucinations at higher generation temperatures and recommends low temperature when text authenticity matters.

C Additional Evaluation on Hallucination

D Detailed Results

The detailed object-based evaluation results cover multiple LVLMs, and the authors report that prompt induction affects nearly all evaluated models in idealized scenarios.

D Detailed Results

  • Detailed object-based hallucination results are provided for mPLUG-Owl, MiniGPT-4, and LLaVA.Source: D Detailed Results · p. 11
  • The object-based results use QH for queries about absent items, AY for “yes” answers, and CH for hallucinations of those items in generated captions.Source: D Detailed Results · p. 11
  • LVLMs are highly susceptible to prompt induction in artificially constructed ideal hallucination scenarios, a pattern seen in almost all evaluated models.Source: D Detailed Results · p. 11

Related papers

Loading 2308.15126v3…