PaperlayerView exact revision on arXiv

Source-linked AI summary

Unified Hallucination Detection for Multimodal Large Language Models

Xiang Chen, Chenxi Wang, Yida Xue, Ningyu Zhang, Xiaoyan Yang, Qiang Li, Yue Shen, Lei Liang, Jinjie Gu, Huajun Chen

arXiv:2402.03190v4cs.CLcs.AIcs.IRcs.LGcs.MM

TL;DR

MLLM hallucination detection research has focused on limited tasks, categories, and levels of granularity, despite hallucinations that can hinder practical deployment and contribute to misinformation. This paper introduces MHaluBench and UNIHD, which combines claim extraction with auxiliary-tool evidence for verification. UNIHD is reported to perform better across image-to-text and text-to-image generation, while the benchmark covers only those task families.

  • Problem

    Prior MLLM hallucination detection has focused on narrow task settings, limited categories, and coarse granularity, although hallucinations can hinder practical deployment and contribute to misinformation.

    Source: §1 · p. 1

  • Method

    The paper introduces MHaluBench for unified multimodal detector evaluation and UNIHD, which extracts claims and uses auxiliary tools to gather evidence for hallucination verification.

    Source: §1 · p. 2

  • Results

    UNIHD achieves better performance across both image-to-text and text-to-image generation tasks.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The paper presents unified multimodal hallucination detection as a challenging direction and provides a benchmark spanning image-to-text and text-to-image tasks.

    Source: §1 · p. 2

    UNIHD’s stated scope centers on image-to-text and text-to-image tasks and does not yet encompass other multimodal tasks such as video captioning.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Despite significant strides in multimodal tasks, Multimodal Large Language Models (MLLMs) are plagued by the critical issue of hallucination. The reliable detection of such hallucinations in MLLMs has, therefore, become a vital aspect of model evaluation and the safeguarding of practical application deployment. Prior research in this domain has been constrained by a narrow focus on singular tasks, an inadequate range of hallucination categories addressed, and a lack of detailed granularity. In response to these challenges, our work expands the investigative horizons of hallucination detection. We present a novel meta-evaluation benchmark, MHaluBench, meticulously crafted to facilitate the evaluation of advancements in hallucination detection methods. Additionally, we unveil a novel unified multimodal hallucination detection framework, UNIHD, which leverages a suite of auxiliary tools to validate the occurrence of hallucinations robustly. We demonstrate the effectiveness of UNIHD through meticulous evaluation and comprehensive analysis. We also provide strategic insights on the application of specific tools for addressing various categories of hallucinations.

1 Introduction

The paper frames MLLM hallucination detection as a unified task spanning multimodal tasks, hallucination categories, and claim-level granularity. It introduces MHaluBench and UNIHD, a tool-augmented framework, and reports that UNIHD is effective while detection remains challenging.

1 Introduction

  • The paper identifies task singularity, limited hallucination categories, and response-level assessment as gaps in prior MLLM detection research.It calls for a unified perspective spanning multimodal tasks and finer-grained claims.Source: §1 · pp. 1–2
  • MHaluBench covers image-to-text and text-to-image tasks, modality-conflicting and fact-conflicting hallucinations, and fine-grained claim-level detection.Its examples include object-, attribute-, and scene-text-level modality conflicts.Source: §1 · pp. 1–2
  • UNIHD extracts claims, formulates tool queries, executes specialized tools in parallel, and aggregates evidence for hallucination judgments with rationales.The framework uses MLLMs for claim-specific question formulation and verification.Source: §1 · p. 2
  • UNIHD is reported effective in evaluation, while multimodal hallucination detection remains a formidable challenge.The paper presents UNIHD as task-agnostic and tool-enhanced.Source: §1 · p. 2

2 Preliminaries

The paper organizes multimodal hallucinations into modality-conflicting and fact-conflicting types, then formulates unified detection at the claim level. A detector assigns each claim a hallucination judgment and rationale.

Unified View of Multimodal Hallucination Taxonomy

  • Modality-conflicting hallucinations arise when model outputs conflict with another modality, including errors involving objects, attributes, or scene text.Fact-conflicting outputs instead contradict established knowledge.Source: §2 · pp. 2–3

Unified Detection Problem Formulation

  • The unified formulation treats an image-text pair as visual input or output plus text, with text-only detection as the case where the visual component is null.Text may be a generated response or a query for synthesizing the visual output.Source: §2 · p. 3
  • Detectors assess each claim in the pair as hallucinatory or non-hallucinatory and provide a rationale based on the paper’s hallucination definition.A text response may contain multiple claims.Source: §2 · p. 3

3 Construction of MHaluBench

MHaluBench is a meta-evaluation benchmark spanning image-to-text and text-to-image generation, with segmented and claim-level annotations. Its construction draws on existing task datasets and generated model outputs, followed by human annotation.

3 Construction of MHaluBench

  • 200 IC, 200 VQA, and 220 text-to-image exemplars form MHaluBench’s balanced coverage of three tasks.The benchmark is designed to assess progress in multimodal hallucination detectors.Source: §3 · p. 3

3.1 Hallucinatory Example Collection

  • Image-to-text examples use MS-COCO and TextVQA samples with outputs from mPLUG, LLaVA, and MiniGPT-4; text-to-image examples use augmented captions and DALL-E images.The captions draw from DrawBench and T2I-CompBench and are augmented with objects, attributes, and factual details.Source: §3.1 · p. 3

3.2 Segment and Claim Extraction

  • The benchmark segments image-to-text outputs and extracts related claims, while treating fundamental intent concepts in text-to-image queries as claims.These segment- and claim-level units support assessment beyond whole responses.Source: §3.2 · pp. 3–4

3.3 Human Annotation and Agreement.

  • Three graduate-level annotators independently label claims, segments, and responses; uncertain cases are resolved by majority rule, with Fleiss’s κ = 0.822.A segment is hallucinatory if it contains any hallucinatory claim, and a response if it contains any hallucinatory segment.Source: §3.3 · p. 4

4 UNIHD: Unified Hallucination Detection Framework for MLLMs

UNIHD identifies claim-level units, formulates tool-specific queries, runs auxiliary tools concurrently, and aggregates their evidence to classify claims as hallucinatory or non-hallucinatory with explanations.

4.1 Essential Claim Extraction

  • GPT-4V/Gemini derives verifiable claims from image-to-text responses and deconstructs text-to-image queries into distinct claims.Claim extraction is a prerequisite for fine-grained hallucination detection.Source: §4.1 · p. 4

4.2 Autonomous Tool Selection Via Query Formulation

  • For each claim, GPT-4V/Gemini formulates pertinent aspect-oriented queries or returns “none” when a tool is unnecessary.For the example claim, UNIHD selects attribute- and object-oriented questions while bypassing scene-text and fact-oriented tools.Source: §4.2 · p. 4

4.3 Parallel Tool Execution

  • UNIHD runs selected tools concurrently to gather evidence, using specialized tools for objects, attributes, scene text, and factual questions.The framework is tool-agnostic, allowing emerging tools and detection strategies to be integrated.Source: §4.3 · pp. 4–5

4.4 Hallucination Verification with Rationales

  • An MLLM combines tool evidence with the original image and claim list to label each claim hallucinatory or non-hallucinatory and explain its judgment.The final decision is based on the level of evidence support.Source: §4.4 · p. 5

5 Experiment

The experiments evaluate UNIHD and self-check baselines on MHaluBench, where detection remains challenging. UNIHD with GPT-4V outperforms baseline detectors across image-to-text and text-to-image tasks, while results vary by detector base, hallucination type, and task.

5.1 Experimental Settings

  • The evaluation compares UNIHD with Self-Check (2-shot) and Self-Check (0-shot), using GPT-4V and Gemini as detector bases.Performance is assessed with recall, precision, and Micro-F1 for hallucinatory and non-hallucinatory categories, plus Macro-F1 at claim and segment levels; the claim list is batched as a single input.Source: §5.1 · pp. 5–6

5.2 Evaluation Results

  • UNIHD powered by GPT-4V consistently outperforms baseline detectors on image-to-text and text-to-image tasks, including Self-Check (2-shot).The comparison reports Micro-F1 by default and Macro-F1 as Mac.F1.Source: Table 2 · p. 6, §5.2 · p. 7
  • Most detectors achieve overall Macro-F1 between 70% and 80%, indicating subpar performance on MHaluBench.The benchmark's hallucinatory examples come from open-source MLLM outputs, not GPT-4V or Gemini.Source: §5.2 · p. 6

5.3 Analysis

  • Tool enhancement improves scene-text and factual hallucination detection over Self-Check (2-shot), but yields minimal improvement for attribute-level hallucinations.Figure 6 compares object, attribute, scene-text, and fact categories; the authors suggest the attribute result may reflect a lack of specialized direct-detection tools and weak self-reflection.Source: Figure 6 · p. 6, §5.3 · p. 7

5.2 Evaluation Results

  • GPT-4V-powered detectors outperform Gemini counterparts in Macro-F1, while Gemini performs better on non-hallucinatory image-to-text categories.Self-Check (0-shot) with GPT-4V scores claim-level Macro-F1 72.82, versus Gemini's 52.98.Source: §5.2 · pp. 6–7

5.3 Analysis

  • UNIHD synthesizes evidence into more credible rationales, but can fail when tools provide incorrect or unhelpful evidence or the MLLM retains its initial bias.Figure 7 illustrates both failure patterns; the authors identify tool accuracy and MLLMs dedicated to hallucination detection as areas for further research.Source: Figure 7 · p. 7, §5.3 · pp. 7–8
  • Across sampled prompts, GPT-4V has the lowest claim-level hallucination ratio in most tested conditions, and the MLLM ranking generally agrees with leaderboards and human evaluation.Figure 8 compares claim-level ratios using 20 MHaluBench prompts for each of IC, VQA, and T2I.Source: Figure 8 · p. 8, §5.3 · p. 8
  • Both baselines and GPT-4V-enhanced UNIHD detect text-to-image hallucinations better than image-to-text hallucinations.The authors attribute the difference to structured user queries and more uniform images in text-to-image tasks, versus diverse natural images with background noise in image-to-text tasks.Source: §5.3 · p. 8

6 Related Work

Prior MLLM hallucination work has examined evaluation, detection, and mitigation, while related detection efforts have often focused on narrower scopes. This paper broadens the problem and uses automated selection of diverse tools to support multimodal hallucination detection.

6.1 Hallucinations in MLLM

  • Prior MLLM work has evaluated, detected, and mitigated hallucinations; HaELM studies POPE's challenges and proposes training with simulated hallucination samples.The paper distinguishes its broader scope, UNIHD framework, and MHaluBench meta-evaluation benchmark from these efforts.Source: §6.1 · p. 8

6.2 Harnessing Tool Resources for LLMs

  • LLM tool-augmentation research spans external resources and diverse AI models, while this work focuses on automating selection of functionally diverse tools for MLLM hallucination detection.The paper notes that adapting tool augmentation to MLLMs introduces challenges in selecting appropriate tools.Source: §6.2 · p. 8

7 Conclusion

The paper formulates multimodal hallucination detection across diverse tasks and types, introduces MHaluBench and UNIHD, and reports better UNIHD performance across image-to-text and text-to-image generation.

7 Conclusion

  • The paper introduces a unified problem formulation and fine-grained MHaluBench for multimodal hallucination detection.MHaluBench is proposed to promote this challenging direction.Source: §7 · p. 8
  • UNIHD achieves better performance across both image-to-text and text-to-image generation tasks.The authors describe this as confirming the framework’s universality and efficacy.Source: §7 · p. 9
  • UNIHD autonomously selects external tools to capture pertinent knowledge for hallucination verification with rationales.Source: §7 · p. 9

Limitations

The framework focuses on image-to-text and text-to-image tasks, covers several but not all hallucination categories, relies mainly on closed-source models and external tools, and uses preliminary tool choices that may not be optimal.

The Scope of Multimodal Tasks

  • The framework does not yet cover other multimodal tasks such as video captioning.Its stated task focus is image-to-text, including image captioning and VQA, and text-to-image generation.Source: Limitations · p. 9

Limitations of Closed-Source MLLM Pricing and Inference Speed

  • UNIHD relies primarily on closed-source foundation models and external tools, adding operational expenses and inference time.Source: Limitations · p. 9

The Scope of Hallucination Categories

  • MHaluBench and UNIHD include object, attribute, scene-text, and factual hallucinations, but other categories remain uncovered.Source: Limitations · p. 9

Preliminary Attempts at Tool Utilization

  • The current tool choices may not be optimal; the authors call for extensive evaluation of state-of-the-art object detectors for this task.Grounded DINO is given as an example of the configured object-detection tool.Source: Limitations · p. 9

A Prompt Templates

The appendix outlines prompt templates for autonomous query formulation and hallucination verification, with query templates organized by hallucination level and verification templates covering two generation directions.

A Prompt Templates

Related papers

Loading 2402.03190v4…