PaperlayerView exact revision on arXiv

Source-linked AI summary

HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data

Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, Yueting Zhuang

arXiv:2311.13614v2cs.CVcs.AI

TL;DR

Machine-generated visual instruction data can contain object, relation, and attribute hallucinations that propagate inaccurate visual outputs, motivating systematic mitigation. HalluciDoctor detects and removes these errors through question-based cross-checking and counterfactual expansion, with experiments reporting reduced hallucinations and retained or improved MLLM performance.

  • Problem

    Machine-generated visual instruction data contains object, relation, and attribute hallucinations that can lead MLLMs to produce inaccurate visual outputs.

  • Method

    HalluciDoctor uses question-based cross-checking to detect and remove hallucinatory answer chunks, then expands instructions counterfactually to address long-tail object co-occurrences.

  • Results

    HalluciDoctor reduces hallucinations across MLLM backbones and reports higher MME performance for LLaVA+ than LLaVA, 1207.18 v.s. 1148.93.

  • Takeaways & Limitations

    The experiments support using HalluciDoctor to reduce hallucinations while retaining MLLMs’ open-ended capabilities.

Abstract

from arXiv · show

Multi-modal Large Language Models (MLLMs) tuned on machine-generated instruction-following data have demonstrated remarkable performance in various multi-modal understanding and generation tasks. However, the hallucinations inherent in machine-generated data, which could lead to hallucinatory outputs in MLLMs, remain under-explored. This work aims to investigate various hallucinations (i.e., object, relation, attribute hallucinations) and mitigate those hallucinatory toxicities in large-scale machine-generated visual instruction datasets. Drawing on the human ability to identify factual errors, we present a novel hallucination detection and elimination framework, HalluciDoctor, based on the cross-checking paradigm. We use our framework to identify and eliminate hallucinations in the training data automatically. Interestingly, HalluciDoctor also indicates that spurious correlations arising from long-tail object co-occurrences contribute to hallucinations. Based on that, we execute counterfactual visual instruction expansion to balance data distribution, thereby enhancing MLLMs' resistance to hallucinations. Comprehensive experiments on hallucination evaluation benchmarks show that our method successfully mitigates 44.6% hallucinations relatively and maintains competitive performance compared to LLaVA. The data and code for this paper are publicly available. \url{https://github.com/Yuqifan1117/HalluciDoctor}.

1. Introduction

Machine-generated visual instruction data can contain object, relation, and attribute hallucinations that increase inaccurate MLLM outputs. HalluciDoctor detects and removes these errors through cross-checking, while counterfactual expansion addresses long-tail object co-occurrences.

  • Motivation: Machine-generated visual instructions may contain responses incongruent with image content because text-only LLMs generate visual descriptions.An example response mentions traffic lights absent from the image.
  • Motivation: Instruction-tuning on LLaVA data improves MLLM performance but significantly increases hallucination probability, including 32.6% hallucinations in LLaVA.The paper motivates mitigating this hallucinatory toxicity in training data.
  • HalluciDoctor: HalluciDoctor automatically detects hallucinations in arbitrary positions and removes them from massive visual instruction datasets using consistency cross-checking.Its goal is to avoid manual annotations while handling multiple hallucination types.
  • HalluciDoctor: HalluciDoctor identifies spurious correlations from long-tail object co-occurrences, which can cause MLLMs to infer nonexistent objects.The method uses counterfactual visual instruction expansion to balance these co-occurrences and create LLaVA++.
  • Contributions: The paper reports that counterfactual expansion strengthens MLLMs’ resistance to hallucinations and improves overall performance.Figure 1 contrasts LLaVA+ and LLaVA++ after HalluciDoctor refinement.
  • Contributions: The paper contributes a first comprehensive investigation of hallucination toxicity, a low-resource detection-and-elimination method, and automatically generated counterfactual instruction data.Its empirical study evaluates effectiveness in eliminating hallucinations and improving MLLM robustness.

2. Related works

MLLMs have achieved strong vision-language performance but still produce text that conflicts with visual content. Existing hallucination methods often focus narrowly on object errors or require manual ground truth, specialized classifiers, extra training data, or inference-time modules.

  • MLLM hallucinations: MLLMs demonstrate strong performance across vision-language tasks but remain vulnerable to hallucinations that conflict with visual content.The related-work discussion frames hallucination as an ongoing MLLM problem.
  • Existing detection methods: Prior detection methods have limited scope because CHAIR-based work focuses on object hallucinations, while other approaches treat detection as binary classification.These choices limit evaluation for open-ended responses.
  • Existing detection methods: Some methods require manual ground-truth answers or specialized classifiers to identify hallucinations.The paper positions its approach against these additional requirements.

3. The Toxicity of Visual Instruction Data

Self-generated visual instructions address the limited scale of multimodal instruction data but may transmit hallucinations into MLLMs. This section defines object, relation, and attribute hallucinations and extends CHAIR to measure them across machine-generated datasets.

  • The Toxicity of Visual Instruction Data: Self-generated instructions are increasingly used because multimodal instruction-following data remains limited, but GPT-4-generated visual instructions may contain numerous hallucinations.The paper presents this as the first systematic analysis of visual instruction dataset toxicity.
  • 3.1. Hallucination Metrics: Object hallucination describes an object mentioned in a description but absent from the image.This is one of the three hallucination categories used throughout the analysis.
  • 3.1. Hallucination Metrics: Relation hallucination describes an inconsistent relation between corresponding objects in the description and image.The category evaluates whether described object relationships match visual content.
  • 3.1. Hallucination Metrics: Attribute hallucination describes inaccurate object properties such as size, color, or state.Attributes are evaluated separately from object presence and object relations.
  • 3.1. Hallucination Metrics: The extended CHAIR metric incorporates synonym lists, splits descriptions into sentences, and extracts objects, relations, and attributes for evaluation.It measures the ratio of sentences containing hallucinatory elements absent from the image.
  • 3.1. Hallucination Metrics: Table 1 compares statistics for three hallucination types in visual instruction datasets with corresponding HalluciDoctor-rectified versions.The caption identifies the rectified rows as bolded.
  • 3.1. Hallucination Metrics: Higher CHAIR scores indicate more hallucinations, while CHAIRrel and CHAIRattri are computed only among existent objects.This avoids counting compositional errors caused by object hallucinations as additional errors.
  • 3.2. Hallucinatory Toxicity Statistics: The toxicity analysis examines LLaVA’s 158K GPT-4-generated samples and MiniGPT4-Instruction’s approximately 3.5K ChatGPT-refined instances.The datasets are used to study hallucination frequency in machine-generated visual instructions.

4. HalluciDoctor Framework

HalluciDoctor combines consistency cross-checking to detect and remove hallucinated answer chunks with seesaw-based counterfactual expansion to reduce spurious object correlations. Its pipeline converts generated descriptions into questions, compares image-grounded answers from multiple MLLMs, rectifies inconsistent text, and adds targeted counterfactual instructions.

  • Hallucination Cross-Checking: HalluciDoctor decomposes hallucination detection into answer-chunk generation, question generation, and consistency cross-checking subtasks.Answer chunks are extracted from generated visual instructions, converted into questions, and checked against image-oriented candidate answers.
  • Hallucination Cross-Checking: Multiple MLLM experts answer generated questions about the reference image, and BEM-based voting produces a consistency score for each answer chunk.The framework uses image-oriented candidate answers from models including BLIP2, InstructBLIP, and MiniGPT-4, then compares their consistency with the original chunk.
  • Hallucination Cross-Checking: Answer chunks with ConScore < 0.5 are treated as hallucinations and automatically removed with contextual rewriting to preserve sentence coherence.ChatGPT performs the rectification after hallucinated chunks are located, producing more accurate visual instruction data.
  • Seesaw-based Visual Instruction Expansion: HalluciDoctor identifies long-tail object co-occurrence correlations as a source of hallucinations and introduces counterfactual interventions by placing hallucinatory objects into tail scenes.The expansion targets scenes where an object is rare but contextually plausible, aiming to reduce spurious correlations among strongly associated objects.
  • Seesaw-based Visual Instruction Expansion: The seesaw strategy combines enhancement and inhibiting factors into a Seesaw-Score for selecting target scenes that balance co-occurrence rarity with contextual plausibility.The enhancement factor favors objects that rarely co-occur with the hallucinatory object, while the inhibiting factor suppresses contextually implausible combinations.
  • Seesaw-based Visual Instruction Expansion: Selected scenes receive counterfactual descriptions that integrate the hallucinatory object into suitable image locations, expanding visual instruction data.The method uses bounding-box guidance and templates such as “There is also a/an {object} in the image.”

5. Experiments

Experiments evaluate HalluciDoctor across hallucination, general performance, GPT-4, human, robustness, and ablation settings. The method reduces hallucinations while preserving or improving MLLM performance and descriptive quality.

  • Hallucination Evaluation: HalluciDoctor consistently reduces various hallucinations across MiniGPT-4 and mPLUG-Owl, with average reductions of 4.6% / 11.4% and 2.7% / 8.7% at two metric levels.The framework is evaluated after hallucination elimination on the LLaVA+ dataset.
  • Hallucination Evaluation: HalluciDoctor outperforms other model-agnostic methods, including LRV-Instruction, especially for MiniGPT-4 attribute hallucinations at the instance level, 8.5% versus 13.6%.The comparison covers object, relation, and attribute hallucinations.
  • Hallucination Evaluation: Counterfactual instruction expansion further reduces object hallucinations, lowering sentence-level CHAIRrel from 20.5% with LLaVA+ to 17.1% with LLaVA++.The expanded instructions balance long-tail object co-occurrences and are associated with the fewest hallucinations across metrics.
  • MLLM Performance: On MME, LLaVA+ exceeds LLaVA in overall performance, scoring 1207.18 versus 1148.93, while LLaVA++ provides further improvement.The experiments use MiniGPT4-7B and mPLUG-Owl-7B and report summed subtask scores.
  • GPT-4 Evaluation: GPT-4 evaluation shows HalluciDoctor achieves the highest total score by reducing hallucinations while preserving diverse descriptions.LLaVA enriches descriptions but increases hallucinations, whereas LRV-Instruction reduces hallucinations at the cost of descriptive diversity.
  • Human Evaluation: Human evaluation on OwlEval finds LLaVA++ achieves the highest image-content accuracy while maintaining response quality across open-domain questions.OwlEval also measures response stability through score variance.
  • Robustness and Ablation: Robustness analysis shows HalluciDoctor reduces hallucination frequency and improves model performance when applied to the MiniGPT4-Instruction dataset.An ablation study separately examines factors in counterfactual instruction expansion.

6. Conclusions

The paper investigates hallucinations in machine-generated visual instruction data and introduces HalluciDoctor to detect and eliminate them automatically. It also uses counterfactual instruction expansion to improve MLLMs’ resistance to hallucinations.

  • Contributions: HalluciDoctor uses question-based cross-checking to automatically detect and eliminate potential hallucinations in machine-generated visual instruction data.The framework targets various hallucinations and operates without requiring manual annotations.
  • Contributions: The paper identifies a co-occurrence issue associated with hallucinations and addresses it through counterfactual instruction expansion.The expansion is intended to strengthen MLLMs’ resistance to hallucination errors.
  • Contributions: Automatic metrics and human evaluations demonstrate that the approach dispels various hallucinations while retaining MLLMs’ open-ended capabilities.

Supplementary Material

The supplementary material expands the analysis of HalluciDoctor, adds experimental results, and provides additional examples. It also details answer-based question generation using ChatGPT across semantic chunks and question types.

  • Supplementary Contents: The supplementary material contains a more detailed HalluciDoctor analysis, additional experimental analysis, and additional examples.
  • Answer-Based Question Generation: ChatGPT generates questions from context descriptions and answer blocks using a prompt template.The generated questions cover broad semantic chunks and various question types.
  • Answer-Based Question Generation: The generated questions are designed to reflect meaningful semantic information in the descriptions.

B.2. Consistency Cross-checking Analysis

The consistency cross-checking analysis examines threshold effects on hallucination elimination and model performance. It also describes refinement and counterfactual expansion procedures that preserve useful semantics while reducing hallucinations.

  • Consistency Threshold: The analysis evaluates different consistency thresholds using average performance on CHAIR and MME benchmarks.The threshold controls consistency cross-checking for identifying hallucinatory chunks.
  • Consistency Threshold: As lower thresholds increase, more hallucinatory descriptions are detected and eliminated, providing higher-quality instruction data for fine-tuning.
  • Consistency Threshold: When the consistency threshold exceeds 0.5, model performance declines significantly, possibly because nearly all answers are eliminated.
  • Hallucination Elimination: ChatGPT refines descriptions by removing hallucinatory phrases while preserving the original sentence structure.The refinement uses both the hallucinatory phrases and original descriptions as inputs.
  • Counterfactual Instruction Expansion: Counterfactual expansion selects scenes where a target object is absent, then prepares candidate objects and masks for image synthesis.The procedure uses object detection, text-to-image models, and object detection tools.
  • Counterfactual Instruction Expansion: 13.8% →12.0% hallucinations in MiniGPT-4 accompany counterfactual instruction expansion, which also improves perception of specific information.The passage gives a shop’s name as an example and states that the approach alleviates long-tail distribution effects.

C.1. Experiment Details

The experiment details describe MiniGPT-4 implementation, zero-shot object-hallucination evaluation, and validation-image construction using object, relation, and attribute annotations. The evaluation follows multiple POPE sampling strategies.

  • Implementation Details: MiniGPT-4 is initialized from its first-pretraining-stage checkpoint and fine-tuned only through its linear projection layer for 10000 steps.
  • Zero-Shot Evaluation: Table 6 reports MiniGPT-4 zero-shot object hallucination results for visual-instruction variants on POPE.The reported metrics are accuracy and F1 scores.
  • Zero-Shot Evaluation: POPE samples objects absent from images using random, popular, and adversarial strategies.
  • Validation Data: Validation images are constructed from overlapping MSCOCO and Visual Genome images to cover object, relationship, and attribute annotations.Visual foundation models identify objects, relations, and attributes in the validation images.

C.2. POPE Results

HalluciDoctor’s rectified and expanded instruction datasets improve hallucination-related evaluation while preserving or improving performance on conventional vision-language tasks.

  • POPE evaluation: LLaVA+ and LLaVA++ achieve consistent gains in all reported POPE accuracy and F1 scores over the baseline dataset.POPE evaluates close-ended object hallucinations, so it does not cover every hallucination type in the visual instruction data.
  • Zero-shot vision-language tasks: HalluciDoctor achieves better generalization performance than other hallucination-elimination methods on zero-shot captioning and visual question-answering tasks.The evaluation covers NoCaps captioning and GQA and AOK-VQA question answering.
  • Dataset evaluation: LLaVA+ receives higher manual accuracy scores than LLaVA, 451 versus 371, while maintaining comparable quality, 405 versus 412, across 200 sampled instructions.The comparison evaluates generated-data accuracy and quality after hallucination elimination.

D.2. MLLMs’ Inference Analysis

The inference analysis compares models across varied visual-perception cases and uses prompt-based question generation and GPT-4 evaluation to assess hallucination reduction and descriptive detail.

  • Visualized inference comparison: LLaVA+ eliminates hallucinatory descriptions, while LLaVA++ further adds reliable detailed descriptions across varied image types.The comparison includes models fine-tuned on LLaVA, LRV-Instruction, LLaVA+, and LLaVA++.
  • Question generation: Answer-based question generation produces concise, answerable questions from answer chunks and context descriptions for inference analysis.The prompt explicitly excludes “How many” questions and requires each question to match its corresponding answer.
  • HalluciDoctor prompt design: HalluciDoctor prompts use injectable description, answer, passage, and hallucination-phrase slots that are replaced with corresponding text before LLM input.The prompt design supports flexible insertion of the relevant content for hallucination detection and elimination.
  • GPT-4 evaluation: GPT-4 evaluation scores description diversity from 1 to 5 according to the number of objects, relations, and attributes.Longer captions with more content receive higher diversity scores under the stated evaluation prompt.
  • Evaluation cases: The analysis visualizes MLLM comparisons across multi-round conversation, single-round conversation, and detailed-description cases.These cases are presented as varied visual-perception settings for comparing model outputs.
Loading 2311.13614v2…