PaperlayerView exact revision on arXiv

Source-linked AI summary

HallE-Control: Controlling Object Hallucination in Large Multimodal Models

Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, Chunyuan Li, Manling Li

arXiv:2310.01779v3cs.CV

TL;DR

Detailed captions are difficult to evaluate for object-existence hallucination, and VQA-based benchmarks may not capture the issue. The paper introduces CCEval, analyzes how vision-language alignment relates to hallucination, and presents HallE-Control to regulate grounded and inferred objects while preserving caption coverage.

  • Problem

    VQA-based evaluations may incompletely assess object-existence hallucination in detailed captions, which require a comprehensive view of visual details.

    Source: §1 · p. 2

  • Method

    The paper introduces CCEval for detailed-caption evaluation and HallE-Control, which uses a lightweight control layer to condition captions on contextual and parametric knowledge.

    Source: §1 · p. 2, §3.2 · p. 7

  • Results

    HallE-Control controls whether captions use contextual knowledge alone or blend it with parametric knowledge while preserving object count, coverage, and sentence length.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors favor controlling generalization rather than removing all imagined objects, distinguishing grounded contextual knowledge from inferred parametric knowledge.

    Source: §1 · p. 2

Abstract

from arXiv · show

Current Large Multimodal Models (LMMs) achieve remarkable progress, yet there remains significant uncertainty regarding their ability to accurately apprehend visual details, that is, in performing detailed captioning. To address this, we introduce $\textit{CCEval}$, a GPT-4 assisted evaluation method for detailed captioning. Interestingly, while LMMs demonstrate minimal object existence hallucination in existing VQA benchmarks, our proposed evaluation reveals continued susceptibility to such hallucinations. In this paper, we make the first attempt to investigate such hallucination from different aspects, including image resolution, the language decoder size, and instruction data amount, quality, granularity. Our findings underscore the unwarranted inference when the language description includes details at a finer object granularity than what the vision module can ground or verify, thus inducing hallucination. To control such hallucinations, we further attribute the reliability of captioning to contextual knowledge (involving only contextually grounded objects) and parametric knowledge (containing inferred objects by the model). Thus, we introduce $\textit{HallE-Control}$, a controllable LMM in terms of $\textbf{Hall}$ucination in object $\textbf{E}$xistence. HallE-Control can condition the captioning to shift between (i) exclusively depicting contextual knowledge for grounded objects and (ii) blending it with parametric knowledge to imagine inferred objects. Our method reduces hallucination by 44% compared to LLaVA$_{7B}$ and maintains the object coverage.

1 Introduction

Detailed captioning can contain object-existence hallucinations, which VQA-based evaluations may not fully capture. The paper introduces CCEval and analyzes hallucination factors, then presents HallE-Control to regulate the use of grounded and inferred objects.

1 Introduction

  • The paper focuses on defining, analyzing, and addressing object-existence hallucination, where captions mention objects absent from the image.It distinguishes this from attribute and relationship hallucinations.Source: §1 · pp. 1–2
  • CCEval evaluates object-existence hallucination in detailed captions while controlling for average sentence length and number of objects.Models that performed well on VQA-based hallucination benchmarks showed substantial hallucinations under CCEval.Source: §1 · p. 2
  • Hallucination analysis examines decoder size, instruction-data quantity, quality and granularity, and vision-encoder input resolution.The authors identify alignment between caption-mentioned objects and objects the vision encoder can perceive as the most crucial factor.Source: §1 · p. 2
  • HallE-Control conditions captions to use grounded objects alone or blend them with inferred objects, while preserving object count, coverage, and sentence length.A lightweight single linear layer controls a frozen LMM using training data that distinguishes contextual knowledge from marked parametric knowledge.Source: §1 · p. 2
  • The paper contributes a component-wise analysis of hallucination factors, a method to control object-existence hallucination, and CCEval metrics for detailed captions.The listed evaluation measures include object count, coverage, and average sentence length alongside hallucination assessment.Source: §1 · p. 3

2 Hallucination Analysis

The analysis compares hallucination across evaluation settings and model factors, finding that detailed-caption assessments can diverge from VQA results and that visual grounding is central to the problem. It examines decoder size, instruction data, and image resolution, then explains hallucination through the distinction between grounded contextual knowledge and inferred parametric knowledge.

2 Hallucination Analysis

  • The analysis examines decoder size, instruction-data quantity, quality, and granularity, and vision-encoder input resolution across LLaVA, InstructBLIP, and Shikra.LLaVA and Shikra share a model structure; Shikra and InstructBLIP use mixed-dataset, multi-task instruction data.Source: §2 · p. 3

2.1 Benchmarks

  • CCEval uses GPT-4 to identify caption objects against ground-truth objects and adds coverage, while balancing caption length and object count.Coverage is the ratio of caption objects matching ground truth to the total ground-truth objects.Source: §2.1 · p. 4
  • Under matched constraints of about 100 words and 9 objects per caption, models remain suboptimal, and Shikra’s VQA advantage reverses on CCEval.This divergence suggests VQA-based evaluation does not consistently capture hallucination in detailed captions.Source: §2.1 · p. 4, Table 2 · p. 4

2.2 Language Decoder

  • Increasing decoder size yielded inconsistent or modest changes: LLaVA improved on POPE but not CCEval, while InstructBLIP’s CCEval CHAIRi and CHAIRs fell by 8 and 5.6.The authors conclude decoder size is not a primary factor in reducing hallucination.Source: §2.2 · p. 4, Table 3 · p. 5

2.3 Data

  • LLaVA trained on 80K instruction examples had fewer hallucinations than models trained on 150K and SVIT; extra data without quality guarantees may increase hallucination.The authors report this pattern for both VQA-based and caption-based evaluations.Source: §2.3 · p. 5, Table 4 · p. 5

2.4 Vision Encoder

  • Larger input resolutions consistently reduced caption hallucination, and the LLaVA7B resolution sweep also increased object coverage.The authors attribute the reduction to improved grounding ability.Source: Table 5 · p. 5, §2.4 · p. 6, Table 6 · p. 6
  • Grounded objects form contextual knowledge, whereas ungrounded training-caption objects can become parametric associations that the model draws on to guess image details.The authors identify alignment between caption objects and what the vision encoder can perceive as a crucial factor.Source: §2.4 · p. 6

3 Hallucination Controlling

HallE-Control controls how much parametric knowledge appears in detailed captions, distinguishing it from objects grounded in the image. Its training data and control parameter encode a shift between contextual-only and contextual-plus-parametric captioning.

3 Hallucination Controlling

  • HallE-Control uses a controller to condition the extent of parametric knowledge expressed in detailed captions.The approach treats contextual and parametric knowledge as distinct sources of caption content.Source: §3 · p. 6

3.1 Data Generation

  • RAM divides ground-truth objects into visually grounded and omitted groups; GPT-4 turns grounded objects into contextual captions, while omitted objects are bracketed in original LLaVA captions.The grounded-only data is paired with parametric-joint data that combines contextual content with marked omitted objects.Source: §3.1 · pp. 6–7

3.2 Hallucination Control

  • A control parameter ε shifts output between restricting and permitting imagination, with the learned projector W governing the intensity of related-object generation.The model is trained with ε = −1 on contextual-only data and ε = +1 on contextual-plus-parametric data; inference allows values from −1 to +1.Source: §3.2 · pp. 7–8

4 Experiment

The experiments test whether marking parametric objects and controlling their expression can reduce hallucination in detailed captions. HallE-Control’s strongest reported setting lowers hallucination without reducing object coverage.

4.1 Finetune on Parametric Joint Data

  • The parametric-joint-data experiment trains LLaVA to mark guessed objects, then evaluates hallucination on indicated objects, unmarked objects, and all objects.These three settings separate indicated-object performance from caption-wide performance.Source: §4.1 · p. 8
  • For indicated objects, CHAIRi is 63.90 for LLaVA7B and 62.31 for LLaVA13B, with coverage of 12.01 and 19.09, respectively, versus 33.58 for the LLaVA7B baseline.The authors interpret indicated objects’ higher hallucination and lower coverage as evidence that they reflect parametric knowledge.Source: §4.1 · p. 8
  • Without indicated objects, CHAIRs falls from 82 to 57, a 30.5% improvement, and CHAIRi falls from 25.3 to 17.10, a 32.4% decrease.The authors also report that the model reproduces the training-time marking behavior at inference.Source: §4.1 · pp. 8–9
  • With indicated objects included, LLaVA7B CHAIRi improves from 25.30 to 14.00, or 44.66%, and LLaVA13B improves from 16 to 9.86, or 38.38%.The reported hallucination decline occurs without a reduction in object coverage.Source: §4.1 · p. 9

4.2 Hallucination Controlling

  • As ε increases, CHAIRi rises from 20.90 to 26.6 while coverage remains at a similar level for HallE-Control7B.The results show that ε provides a way to vary parametric-knowledge output.Source: §4.2 · p. 9
  • CHAIRs = 43 and CHAIRi = 6.37 for HallE-Control13B with control −1 and indication, versus baseline values of 64 and 16, without decreasing coverage.This is the best reported object-existence hallucination result for HallE-Control13B.Source: §4.2 · p. 9

5 Related Work

The related-work discussion argues that existing evaluation approaches do not fully capture object hallucination in detailed captions. It positions CCEval and the paper’s vision–language alignment analysis as addressing those gaps.

Large Multimodal Models (LMMs)

  • The paper places its work among LMM approaches that vary model structure, grounding, and instruction tuning, while using a detector to filter objects for vision–language alignment.It contrasts this detector use with methods that provide region information or add grounding tasks.Source: §5 · p. 9

Evaluation on LMMs

  • VQA-based benchmarks such as POPE (Li et al., 2023e) use paired yes/no questions, while caption metrics can be constrained by reference length or fail to reflect hallucination alongside object coverage.The authors introduce CCEval with CHAIR metrics, object coverage, average sentence length, and object count.Source: §5 · pp. 9–10

Hallucination

  • The paper identifies vision–language misalignment as another cause of hallucination, and interprets Biten et al. (2022)’s label balancing as weakening parametric knowledge associated with that misalignment.The related-work discussion contrasts this account with prior explanations involving label distributions and answer imbalance.Source: §5 · p. 10

6 Conclusion

The paper studies object hallucination in detailed LMM captions through a new evaluation method and component-wise analysis. It identifies vision-encoder and instruction-data alignment as an issue and introduces inference controls for conditioning object generation.

6 Conclusion

  • The study introduces an evaluation method for object-existence hallucination in detailed captions and conducts component-wise analysis of LMMs.It identifies an alignment issue between the vision encoder and instruction data, then introduces controls over object inference.Source: §6 · p. 10

A.1 Why not all Hallucination Bad?

The appendix presents examples framed as both beneficial and potentially harmful hallucination. Table captions identify beneficial examples in content moderation and robotics.

A.1 Why not all Hallucination Bad?

  • The appendix flags a potential harmful example alongside examples labeled as beneficial hallucination.The supplied table captions identify beneficial examples in content moderation and robotic applications, but do not describe their contents.Source: Table 9 · p. 15, §A.1 · p. 16, Table 10 · p. 16

A.2 More Details on Analyzed Models

The appendix describes three analyzed models’ architectures and training setups. Their visual-language components and instruction-tuning data differ.

A.2 More Details on Analyzed Models

  • LLaVA maps visual tokens into LLM input tokens with a linear projector, then instruction-finetunes both the projector and LLM.Its first training stage pretrains only the projector on simple captions; instruction data uses visual ground truth from COCO.Source: §A.2 · p. 16
  • InstructBLIP uses a Q-former to bridge its frozen vision encoder and LLM, and instruction-finetunes on 26 datasets.Source: §A.2 · p. 17
  • Shikra mirrors LLaVA’s model structure, omits pretraining, and introduces a grounding task during finetuning.It is trained on multiple datasets, as is InstructBLIP.Source: §A.2 · p. 17

A.3 Training Data Quality

The appendix illustrates challenges in judging training-data quality and object grounding using three MSCOCO images. The examples include hard-to-see visual details and a ground-truth annotation omission.

A.3 Training Data Quality

  • In one image, small remote controls and a partially visible person are difficult to discern, while the remote type is ambiguous.The image also contains an incomplete bottle at the left edge.Source: §A.3 · p. 17
  • People behind a train window are difficult for detection models and human observers to count; the appendix says this issue recurs in traffic images.Source: §A.3 · p. 17
  • In another image, close inspection is needed to see both seated people, and the ground-truth labels appear to omit plates and other items.The authors report seeing four plates, despite the labels indicating only one bowl.Source: §A.3 · p. 17

A.4 CCEval

The appendix documents CCEval, implementation and experiment details, further results, qualitative examples, and a theoretical account of HallE-Control. It describes a user-adjustable control parameter and reports that LLaVA 1.5 results align with the main-text conclusion.

A.4 CCEval

  • CCEval prompts a model for captions on 100 Visual Genome images; GPT-4 matches extracted caption objects to ground-truth objects, and coverage is matched-object count divided by total ground-truth-object count.It does not strictly enforce consistent coverage, average length, or average object counts across models, which the authors say is unrealistic; these metrics can reflect limitations in producing detailed captions.Source: §A.4 · pp. 17–18

A.5 Prompt

  • The appendix identifies separate prompts for caption-object extraction, hallucination matching, and ground-truth coverage.Source: §A.5 · p. 18

A.6 Sliding Window

  • The sliding-window setup pads images to 672 × 672, divides them into nine 224 × 224 cells, and concatenates their visual tokens.The encoder processes cells from top-left to bottom-right.Source: §A.6 · p. 18

A.7 Experiment Settings

  • The experiments use LLaVA initialized with Vicuna 1.3 and CLIP-Large, while HallE-Control finetunes an added control layer and uses contextual-only and combined-data captions.The appendix specifies 10K contextual-only and 23K parametric-joint caption examples for control-layer training.Source: §A.7 · p. 18

A.8 More Experiments on LLaVA 1.5

  • LLaVA 1.5 experiments are consistent with the main-text conclusion, with less hallucination in the newer version.Source: §A.8 · p. 18

1. Evaluation only on indicated objects

  • The appendix distinguishes evaluations of objects with indications, without indications, and all mentioned objects.Source: §1 · p. 19

A.10 Qualitative Results of HallE-Control

  • Qualitative examples use ε = −1 for no imagination and ε = 1 for maximum imagination; both models can output indications.Source: §A.10 · p. 19

A.11 Theoretical Explanations

  • Under stated HMM and embedding assumptions, HallE-Control adds εW to word embeddings; its output distribution is described as close to a linear interpolation across ε values.The appendix interprets a linear embedding transformation as shifting the language model’s initial condition.Source: §A.11 · pp. 19–20

Related papers

Loading 2310.01779v3…