PaperlayerView exact revision on arXiv

Source-linked AI summary

Visual Hallucinations of Multi-modal Large Language Models

Wen Huang, Hongbin Liu, Minxin Guo, Neil Zhenqiang Gong

arXiv:2402.14683v2cs.CVcs.AIcs.LG

TL;DR

Existing VH benchmarks rely on limited image datasets, constraining the diversity of instances used to assess MLLMs. VHTest generates new VH instances from mode descriptions and evaluates models on the resulting benchmark. The benchmark reveals low accuracy across mainstream MLLMs, while fine-tuning on it reduces hallucination without sacrificing other benchmark performance.

  • Problem

    Existing VH studies draw instances only from existing image datasets, limiting diversity and risking pretraining contamination that can bias MLLM performance estimates.

    Source: §1 · p. 1

  • Method

    VHTest derives mode descriptions from initial VH instances and uses text-to-image generation to create new images and construct benchmark instances.

    Source: §1 · pp. 1–2, §3.1 · p. 4

  • Results

    All seven mainstream MLLMs achieve accuracy close to random guessing on YNQ, with GPT-4V reaching 0.611; the average across three models is 0.229 on OEQ.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Fine-tuning on the benchmark reduces hallucination without sacrificing performance on other benchmarks.

    Source: §7 · p. 9

    Human workers still manually generate question-answer pairs for automatically generated VH images.

    Source: §8 · p. 9

Abstract

from arXiv · show

Visual hallucination (VH) means that a multi-modal LLM (MLLM) imagines incorrect details about an image in visual question answering. Existing studies find VH instances only in existing image datasets, which results in biased understanding of MLLMs' performance under VH due to limited diversity of such VH instances. In this work, we propose a tool called VHTest to generate a diverse set of VH instances. Specifically, VHTest finds some initial VH instances in existing image datasets (e.g., COCO), generates a text description for each VH mode, and uses a text-to-image generative model (e.g., DALL-E-3) to generate VH images based on the text descriptions. We collect a benchmark dataset with 1,200 VH instances in 8 VH modes using VHTest. We find that existing MLLMs such as GPT-4V, LLaVA-1.5, and MiniGPT-v2 hallucinate for a large fraction of the instances in our benchmark. Moreover, we find that fine-tuning an MLLM using our benchmark dataset reduces its likelihood to hallucinate without sacrificing its performance on other benchmarks. Our benchmarks are publicly available: https://github.com/wenhuang2000/VHTest.

1 Introduction

MLLMs can produce factually incorrect image details, and existing VH benchmarks draw only on limited existing image datasets. VHTest uses initial VH examples to generate diverse images and evaluates MLLMs, finding low accuracy across models and reduced hallucination after fine-tuning.

1 Introduction

  • Existing VH studies use images from datasets such as COCO, limiting instance diversity and risking pretraining contamination that can bias performance estimates.Such benchmarks can incorrectly suggest that an MLLM performs well under VH.Source: §1 · p. 1
  • VHTest turns initial VH instances into mode descriptions, then uses text-to-image generation and templated questions and answers to construct new instances.Initial examples are identified from image pairs with high CLIP but low DINO v2 similarity, then manually given questions and reference answers.Source: §1 · p. 2
  • The resulting benchmark contains 1,200 VH instances across existence, shape, color, orientation, OCR, size, position, and counting modes.Shape and size are formulated by this work; the other six modes were also considered in prior or concurrent studies.Source: §1 · p. 2
  • GPT-4V, LLaVA-1.5, and MiniGPT-v2 achieve overall accuracies of 0.383, 0.229, and 0.075, respectively, on the VHTest benchmark.Their most susceptible modes also differ: GPT-4V has 0.153 accuracy on orientation VH, while LLaVA-1.5 and MiniGPT-v2 have 0.127 and 0.000 accuracy on OCR VH, respectively.Source: §1 · pp. 2–3
  • Fine-tuning LLaVA-1.5 on VHTest reduces hallucination and raises its position-VH accuracy from 0.333 to 0.533, while maintaining performance on other benchmarks.The benchmark was split 80%/20% for training and testing.Source: §1 · p. 3

2 Definitions

The paper defines a VH instance by an image, question, and reference answer, and organizes visual errors into eight modes spanning individual-object and group properties.

2 Definitions

  • The eight VH modes concern object existence, shape, color, orientation, OCR, relative size, relative position, and counting; they cover individual-object and group properties.Existence, shape, color, orientation, and OCR are individual properties; relative size, relative position, and counting are group properties.Source: §2 · p. 3
  • Existence VH occurs when an MLLM misses an object present in the image or identifies an object that is absent.Source: §2 · p. 3
  • Shape and color VH occur when the MLLM incorrectly describes an object's shape or fails to identify its color.Source: §2 · p. 3
  • Orientation, OCR, and size VH involve errors in an object's facing direction, image characters, or relative size comparisons.Source: §2 · p. 3
  • Position and counting VH involve errors in identifying spatial relationships between objects or accurately enumerating them.Source: §2 · p. 3
  • A VH instance is an image, a question, and a reference answer; it succeeds when the MLLM response is factually incorrect relative to that answer.For example, the reference answer gives three lamps while the MLLM says two.Source: §2 · p. 3

3 Our VHTest

VHTest turns image pairs flagged by encoder disagreement into broader visual-hallucination cases through generated descriptions and images, then packages them as benchmark instances.

3.1 Step I: Finding Initial VH Instances

  • CLIP similarity ≥0.9 and DINO v2 similarity ≤0.55 select candidate pairs; 200 pairs yield 800 initial instances, with GPT-4V hallucinating on 204 after manual verification.The pairs are selected because CLIP rates them as similar while DINO v2 rates them as dissimilar; initial instances include 100 per VH mode.Source: §3.1 · pp. 3–4

3.2 Step II: Text Description Generation

  • For each mode, an MLLM uses successful hallucinations—or hypothetical ones when successes are scarce—to generate descriptions, which are integrated from 10 examples into a final description.Prompts include the image, question, hallucinated response, and detected error, and ask for potential causes and guidance for generating more images.Source: §3.2 · pp. 4–5

3.3 Step III: Generating More VH Instances

  • A text-to-image model renders mode-specific descriptions; workers create unambiguous object-template questions and correct reference answers, discarding images without suitable questions.The resulting VH instance is an image, question, and reference-answer triple.Source: §3.3 · p. 5

3.4 Benchmark Construction

  • The benchmark contains 1,200 instances, 150 per mode, with open-ended and balanced yes/no versions; construction took about 300 human-hours, and annotator agreement was measured.The open-ended version requires manual response labeling; yes/no questions are constructed for automatic evaluation with a 50/50 split of reference answers.Source: §3.4 · p. 5

4 Experiments

Experiments show that fine-tuning LLaVA-1.5 on VHTest improves accuracy on its VH benchmarks while preserving comparable MME and POPE performance. Evaluations also find substantial hallucination on VHTest and show that its generation choices affect how likely instances are to trigger hallucinations.

4.1 Experimental Setup

  • The evaluation covers GPT-4V, LLaVA-1.5-13b, and MiniGPT-v2, using accuracy as the metric and manual response analysis for OEQ.YNQ accuracy is calculated automatically from its yes/no reference answers; lower accuracy indicates more likely hallucination.Source: §4.1 · pp. 5–6

4.2 Testing VH in MLLMs

  • GPT-4V averages 0.745 accuracy on the 800 initial COCO instances, with only 204 instances successfully inducing hallucination.The authors contrast this with substantial GPT-4V hallucination on VHTest's shape mode.Source: §4.2 · p. 6, Table 1 · p. 6
  • Average accuracy across the three MLLMs is 0.229 on OEQ and 0.561 on YNQ; 925 of 1,200 OEQ instances induce hallucinations on average.Random guessing yields 0.5 accuracy on the balanced YNQ benchmark.Source: §4.2 · p. 6, Table 2 · p. 6, Table 3 · p. 6
  • Across the three MLLMs, average accuracy is poorest on orientation, counting, and OCR VH modes; orientation VH averages only 0.14, making it the most challenging mode.Source: §4.2 · p. 6

4.3 Ablation Study

  • DALL·E-3 yields the lowest average MLLM accuracy, 0.144, among the four tested image generators.The other generators required GPT-4 to turn descriptions into prompts before image generation.Source: §4.3 · p. 7, Table 4 · p. 8
  • VH instances generated from successful initial instances reduce accuracy across all three MLLMs compared with instances generated from unsuccessful ones.The authors conclude that successful initial instances make VHTest more likely to generate successful VH instances.Source: §4.3 · p. 7, Table 5 · p. 8

Fine-tuning LLaVA-1.5 on Our Benchmark

  • Fine-tuned LLaVA-1.5 achieves higher average accuracy on both VH benchmarks and comparable results on MME and POPE.The authors report that this reduces the model's likelihood of hallucinating.Source: Fine-tuning LLaVA-1.5 on Our Benchmark · p. 7
  • Fine-tuning all three model components gives the best overall results, while one epoch retains the best MME performance and extreme learning rates lower performance.The components tested were the vision encoder, vision-language connector, and LLM.Source: Fine-tuning LLaVA-1.5 on Our Benchmark · p. 8, Figure 6 · p. 8

5 Related Work

Prior work examines hallucinations in MLLMs and mitigation approaches, but existing benchmarks draw VH images only from existing datasets. VHTest is presented as a way to generate new, more diverse images, including shape and size modes.

Hallucinations

  • Visual hallucination occurs when an MLLM imagines incorrect image details in visual question answering.The related-work passage also describes hallucination more generally as generative models imagining factually incorrect details.Source: §5 · p. 8

VH Benchmarks in MLLMs

  • Prior VH benchmarks use images from existing datasets, limiting image diversity and potentially introducing pretraining data contamination.VHTest generates new images, and the paper introduces shape and size modes.Source: §5 · p. 8

Mitigating VH in MLLMs

  • Fine-tuning-phase mitigation targets data quality or model structure, while testing-phase methods use visual-evidence prompts or correction tools.The authors report that prior fine-tuning work typically freezes the vision encoder, whereas they fine-tune it as well; testing-phase mitigation is complementary.Source: §5 · pp. 8–9

6 Discussion

The authors report low accuracy on VHTest and attribute its challenge to adversarial construction that moves beyond the image space of existing datasets. This approach generates instances in the text-to-image models' manifold.

6 Discussion

  • The three evaluated MLLMs average 0.229 accuracy on OEQ, while all seven mainstream MLLMs are near random-guessing accuracy on YNQ and GPT-4V reaches 0.611.The authors describe their benchmarks as more challenging than most existing MLLM benchmarks.Source: §6 · p. 9
  • The authors attribute the benchmarks' challenge to deriving text descriptions from initial VH instances and generating new images beyond existing datasets' image space.They characterize this as moving into the manifold of text-to-image generative models.Source: §6 · p. 9

7 Conclusion

The conclusion presents VHTest as a tool for generating VH instances to test MLLMs. The authors report high hallucination rates on its benchmarks and reduced hallucination after fine-tuning without sacrificing other performance.

7 Conclusion

  • VHTest generates VH instances for testing MLLMs, and the collected benchmarks reveal high hallucination rates in state-of-the-art models.The conclusion also reports that fine-tuning on the benchmark reduces hallucination without sacrificing other performance or capability.Source: §7 · p. 9
  • Fine-tuning MLLMs on the benchmark reduces hallucination without sacrificing other performance or capability.Source: §7 · p. 9

8 Limitations and Future Work

VHTest still requires human workers to write question–answer pairs for generated images. The mode descriptions outline conditions associated with several types of visual hallucination.

8 Limitations and Future Work

  • Human workers must still manually generate question–answer pairs for automatically generated VH images; automating this step is proposed as future work.The limitation concerns question–answer generation, not the image-generation step.Source: §8 · p. 9

A Generated Text Descriptions of VH Modes

2. Shape VH: A multi-modal large language

  • Shape errors can arise when crowded or unusual forms are simplified, generalized, or misread as more familiar shapes.Source: §2 · p. 12
  • Color errors include omitted or misidentified transitions, especially with subtle or intertwined colors, lighting, shadows, nearby colors, or small items.Source: §2 · p. 12
  • Orientation errors can involve ambiguous perspective, overlap, busy scenes, or missed directional cues; OCR errors involve irregular spacing, similar letters, vertical text, and autocorrection.Source: §2 · pp. 12–14
  • Size judgments are challenged by perspective, overlap, complex scenes, and competing objects, while position judgments are challenged by non-linear layouts, similar backgrounds, and deceptive viewpoints.Source: §2 · p. 14
  • Counting becomes difficult with packed, overlapping, partially visible, or varied objects, and when counting subtypes within a broader category.Source: §2 · p. 14

B Additional Results

Four annotators showed very high agreement when labeling a random sample of 100 YNQ benchmark instances.

  • κ = 0.958 (0.81 ≤ κ ≤ 1.00) for four annotators labeling 100 randomly selected YNQ instances.The paper reports this as a very high degree of agreement.Source: §B.1 · p. 14

B.2 More MLLMs Evaluation on VHTest

The additional YNQ evaluation reports GPT-4V as the most accurate among seven evaluated MLLMs.

  • GPT-4V has the highest average accuracy, 0.611, among the seven MLLMs evaluated across Tables 3 and 7.Source: §B.2 · p. 14

B.3 Qualitative Comparison Results

Fine-tuning LLaVA-1.5 improves its descriptions of banana shape and chair size in the illustrated examples. The appendices document VHTest prompts, question templates, a special existence-mode procedure, and fine-tuning settings.

B.3 Qualitative Comparison Results

  • After fine-tuning, LLaVA-1.5 correctly describes a banana’s curved shape and identifies the two largest chairs in examples where it previously erred.The authors report similar VQA-quality improvements across the testing set.Source: §B.3 · pp. 14–15, Figure 8 · p. 15

C Prompts Used in VHTest

D Question Templates

E Special Cases in Step III

F.1 Hyperparamters

  • LLaVA-1.5 fine-tuning follows its limited task-specific setting and unfreezes the vision encoder.The hyperparameters are listed in Table 8.Source: §F.1 · p. 18, Table 8 · p. 18

F.2 Pre-processing of the Training Set

The training set is pre-processed for counting and position VH modes. Counting answers are converted into reference sentences, while position questions receive a single-word-or-phrase instruction.

F.2 Pre-processing of the Training Set

  • Counting-mode number reference answers are modified into reference sentences using an LLM prompted as shown in Figure 15.The authors use this pre-processing to address behavioral overfitting to short-form answers, following Liu et al. (2023).Source: §F.2 · p. 18, Figure 15 · p. 18
  • Position-mode instances receive the instruction “Answer the question using a single word or phrase.” after each question.Source: §F.2 · p. 18
  • The authors apply short-form-answer pre-processing to both counting and position VH modes, following Liu et al. (2023).They cite prior work reporting that short-form data behaviorally overfit an MLLM to short-form answers.Source: §F.2 · p. 18

Related papers

Loading 2402.14683v2…