PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting and Preventing Hallucinations in Large Vision Language Models

Anisha Gunjal, Jihan Yin, Erhan Bas

arXiv:2308.06394v3cs.CVcs.LG

TL;DR

LVLMs often produce visually ungrounded hallucinations, creating a reliability problem for detailed VQA. The paper introduces M-HalDetect, fine-grained FDPO, and reward-model rejection sampling, reducing hallucinations in InstructBLIP and generalizing to other LVLMs. Rejection sampling is effective but prohibitively slow for real-world inference.

  • Problem

    LVLMs generate detailed VQA responses containing non-existent objects, unfaithful descriptions, and inaccurate relationships, while hallucination detection often requires costly human supervision.

  • Method

    The paper builds M-HalDetect with fine-grained multimodal annotations, optimizes InstructBLIP using FDPO, and trains reward models for best-of-n rejection sampling.

  • Results

    The methods reduce hallucination rates by 41% with FDPO and 55% with rejection sampling in InstructBLIP, while reward models reduce rates by 15% in LLaVA and 57% in mPLUG-OWL.

  • Takeaways & Limitations

    M-HalDetect supports hallucination detection and prevention across multiple LVLMs, and reward-model scores align with human ratings.

  • Takeaways & Limitations

    Rejection sampling is prohibitively slow for real-world inference, and the study performs only one cycle of feedback collection and model training.

Abstract

from arXiv · show

Instruction tuned Large Vision Language Models (LVLMs) have significantly advanced in generalizing across a diverse set of multi-modal tasks, especially for Visual Question Answering (VQA). However, generating detailed responses that are visually grounded is still a challenging task for these models. We find that even the current state-of-the-art LVLMs (InstructBLIP) still contain a staggering 30 percent of the hallucinatory text in the form of non-existent objects, unfaithful descriptions, and inaccurate relationships. To address this, we introduce M-HalDetect, a (M)ultimodal (Hal)lucination (Detect)ion Dataset that can be used to train and benchmark models for hallucination detection and prevention. M-HalDetect consists of 16k fine-grained annotations on VQA examples, making it the first comprehensive multi-modal hallucination detection dataset for detailed image descriptions. Unlike previous work that only consider object hallucination, we additionally annotate both entity descriptions and relationships that are unfaithful. To demonstrate the potential of this dataset for hallucination prevention, we optimize InstructBLIP through our novel Fine-grained Direct Preference Optimization (FDPO). We also train fine-grained multi-modal reward models from InstructBLIP and evaluate their effectiveness with best-of-n rejection sampling. We perform human evaluation on both FDPO and rejection sampling, and find that they reduce hallucination rates in InstructBLIP by 41% and 55% respectively. We also find that our reward model generalizes to other multi-modal models, reducing hallucinations in LLaVA and mPLUG-OWL by 15% and 57% respectively, and has strong correlation with human evaluated accuracy scores.

Introduction

LVLMs improve multimodal VQA but still generate visually ungrounded hallucinations that are difficult and costly to detect. The paper introduces M-HalDetect and methods for detecting and reducing these errors.

  • Introduction: LVLMs improve VQA performance but retain a significant propensity to generate hallucinations.
  • Introduction: LVLM hallucinations include incorrect references or descriptions of the input image, reducing multimodal reliability and accuracy.Detection often requires costly human supervision.
  • Introduction: M-HalDetect provides fine-grained sub-sentence annotations for accurate, inaccurate, and analysis segments in detailed image descriptions.The annotation example is illustrated in Figure 1.
  • Introduction: 41% reduction in hallucination rates is achieved by optimizing InstructBLIP with Fine-grained Direct Preference Optimization.
  • Introduction: 55% reduction in InstructBLIP hallucination rates is achieved with reward-model best-of-64 rejection sampling.
  • Introduction: 15% and 57% reductions are obtained for LLaVA and mPLUG-OWL respectively, while reward-model scores align with human hallucination ratings.

Related Work

Prior LVLM work introduced hallucination metrics, datasets, and evaluation methods, but largely emphasized object presence. This paper extends fine-grained feedback and hallucination analysis to descriptions, relationships, and complex reasoning.

  • Hallucination Analysis in LVLMs: Prior LVLM research introduced POPE, LRV, and GAVIE for evaluating or mitigating hallucinations and inconsistent text generation.
  • Hallucination Analysis in LVLMs: Previous approaches collectively provide metrics, datasets, and evaluation methods for improving multimodal text reliability and consistency.
  • Hallucination Analysis in LVLMs: The paper extends prior work beyond object-presence hallucinations to inaccurate attributes, relative positioning, and complex object reasoning.
  • Aligning to Human Preferences: RLHF has shown different hallucination effects across language-model sizes, while this work focuses on relatively smaller 7B multimodal models.
  • Aligning to Human Preferences: DPO directly optimizes a policy without training a reward model or sampling reinforcement-learning rewards, while avoiding unstable RL training.
  • Fine-grained Preferences: M-HalDetect adapts fine-grained feedback to multimodal hallucination detection using 16,000 sub-sentence-annotated image-description pairs and modified DPO loss.

Detection Dataset

M-HalDetect contains fine-grained annotations of InstructBLIP descriptions generated from COCO validation images. Responses are segmented and labeled for factual accuracy, hallucination, and subjective analysis.

  • Dataset Description: M-HalDetect samples image-description pairs from 4,000 COCO validation images, split into 3,200 training and 800 development images.
  • Dataset Description: The dataset avoids COCO training images to reduce potential training-data regurgitation by LVLMs.
  • Dataset Collection: InstructBLIP generates descriptions from images and randomly selected image-description questions, with additional prompts added to increase response diversity.
  • Dataset Collection: 16k image-prompt-response triplets are produced through four nucleus-sampled responses per prompt and divided into 12,800 training and 3,200 validation samples.
  • Dataset Categories: Responses are categorized as Accurate, Inaccurate, Analysis, or Unsure, with the first three serving as the principal annotation categories.
  • Dataset Categories: Annotations operate at the sub-sentence level, allowing one sentence to contain multiple segments from different classes.
  • Dataset Categories: Annotators may leave punctuation, connectors, and introductory phrases unmarked; these words default to Accurate.

Method

The paper trains fine-grained multimodal reward models and adapts DPO to sub-sentence annotations for detecting and reducing hallucinations in LVLM generations.

  • Reward Model Training: Reward models reuse InstructBLIP with a classification head and are trained at sentence and sub-sentence granularities using masked cross-entropy targets.The models classify segment-ending tokens while masking other token positions.
  • Label Granularity: Sentence-level labels condense any inaccurate segment into an inaccurate sentence, while unsure segments are also merged into that class.The dataset supports binary labels by merging Accurate and Analysis, or ternary labels that retain Analysis separately.
  • Rejection Sampling: Rejection sampling scores complete generations by averaging each sentence’s non-hallucination negative log probabilities and selects the best candidates.This produces a normalized score for the entire passage and is applied to InstructBLIP and other LVLMs.
  • Fine-Grained Direct Preference Optimization: FDPO converts fine-grained segment labels into preferred, dispreferred, and ignored neutral signals, increasing preferred-chunk likelihoods and decreasing dispreferred-chunk likelihoods relative to the base model.Segments are non-overlapping, allowing one forward pass to calculate all segment losses in a sample.
  • Fine-Grained Direct Preference Optimization: FDPO experiments compare ignoring Analysis with dispreferring it, while fine-tuning only the QFormer and language head and freezing the remaining model.The experiments use β = 0.5, a maximum of 5 epochs, lr = 10^-6, warmup ratio .03, and cosine scheduling.
  • Evaluation: A hallucination rate is measured as inaccurate words divided by total words, excluding Analysis segments, to separate factuality from helpfulness.This avoids rewarding empty outputs that do not hallucinate but are not useful.

Results

The reward models identify hallucinations and support rejection sampling and FDPO, which reduce hallucination rates across InstructBLIP and other LVLMs. Their scores also correlate strongly with human accuracy evaluations, while increasing rejection-sampling candidates yields diminishing returns.

  • Reward Models: The ternary reward model is selected for rejection sampling because it separates Accurate, Inaccurate, and Analysis classes and functionally subsumes the binary model.
  • Rejection Sampling: Reward-score means and variances exhibit diminishing returns as the number of rejection-sampling candidates increases.
  • Human Evaluation: Reward model scores show a clear and strong correlation with human accuracy scores across all human-evaluated results.
  • Fine-Grained DPO: 41% reduction in hallucination rates is achieved by FDPO when optimizing InstructBLIP with M-HalDetect.

Conclusion

M-HalDetect supports both direct optimization and reward-model rejection sampling to reduce hallucinations in LVLMs, with effectiveness extending across multiple models. However, rejection sampling is prohibitively slow for real-world inference, and one-cycle feedback training may cause overfitting that reduces descriptiveness and precision.

  • Conclusion: M-HalDetect enables fine-grained reward models and FDPO to reduce hallucinations while extending effectiveness across multiple LVLMs.FDPO optimizes InstructBLIP directly without preference pairs, while reward-model rejection sampling generalizes to LLaVA and mPLUG-OWL.
  • Conclusion: Rejection sampling is prohibitively slow for inference in real-world use-cases.The authors identify reinforcement-learning optimization of a generative model as a next step for producing a higher-quality instruction-aware VQA system.
  • Conclusion: The paper performs only one cycle of response-feedback collection and training, despite typical fine-grained-feedback training using multiple iterations.The authors report hints of overfitting to the training objective in some responses.
  • Conclusion: Image descriptions became slightly more generic, and their preciseness may have declined after optimization.Future work could extend the dataset and methods to account for descriptiveness and informativeness.
Loading 2308.06394v3…