PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, Conghui He

arXiv:2311.16839v2cs.CVcs.CL

TL;DR

LVLMs still generate image descriptions containing fabricated or inaccurate content, motivating better hallucination mitigation and evaluation. The paper introduces HA-DPO, which trains models to prefer non-hallucinatory responses using style-consistent sample pairs, and SHR for sentence-level measurement. Across prevalent models, HA-DPO reduces hallucinations and improves general performance, although POPE recall can decrease because it evaluates only “Yes” responses.

  • Problem

    LVLMs generate fictitious information, while existing evaluation systems restrict hallucinations to predefined categories or narrow object-focused scopes.

  • Method

    HA-DPO reframes hallucination elimination as preference optimization, amplifying non-hallucinatory responses over hallucinatory ones with style-consistent training data.

  • Results

    HA-DPO significantly reduces hallucinations and enhances general performance across prevalent multimodal models.

  • Takeaways & Limitations

    The improved models tend to generate non-hallucinatory outputs with enhanced generalization capabilities, while SHR directly measures hallucination in model outputs.

  • Takeaways & Limitations

    POPE recall considers only correct “Yes” responses and overlooks “No” answers, so recall changes do not alone indicate model degradation.

Abstract

from arXiv · show

Multimodal large language models have made significant advancements in recent years, yet they still suffer from a common issue known as the "hallucination problem", in which the models generate textual descriptions that inaccurately depict or entirely fabricate content from associated images. This paper introduces a novel solution, Hallucination-Aware Direct Preference Optimization (HA-DPO), which reframes the hallucination problem as a preference selection task. The model is trained to favor the non-hallucinating response when presented with two responses of the same image (one accurate and one hallucinatory). Furthermore, this paper proposes an efficient pipeline for constructing positive~(non-hallucinatory) and negative~(hallucinatory) sample pairs, ensuring a high-quality, style-consistent dataset for robust preference learning. When applied to three mainstream multimodal models, HA-DPO significantly reduced hallucination issues and amplified the models' generalization capabilities. Notably, the MiniGPT-4 model, when enhanced with HA-DPO, demonstrated a substantial improvement: POPE accuracy rose from 51.13% to 86.13% (an absolute improvement of 35%), and the MME score surged from 932.00 to 1326.46 (a relative improvement of 42.32%). The codes, models, and datasets are made accessible at https://opendatalab.github.io/HA-DPO.

1. Introduction

LVLMs can generate hallucinated image descriptions that mislead users, while existing mitigation approaches trade off training resources, flexibility, or dependence on external tools. The paper reframes hallucination elimination as preference optimization, introduces HA-DPO and SHR, and reports reduced hallucinations with improved general performance.

  • LVLM hallucinations fabricate objects, categories, attributes, or relationships, potentially causing serious harms such as medical misdiagnosis.
  • SFT-based mitigation requires abundant high-quality data, extensive annotation, and substantial training overhead.
  • Post-hoc mitigation avoids additional training unless customized models are needed, but its effectiveness depends on available tools and its time cost grows with added processing.
  • HA-DPO treats hallucination elimination as a preference optimization problem that biases multimodal models toward hallucination-free outputs.
  • SHR provides a broad, fine-grained, quantitative measure of multimodal hallucinations without restricting evaluation to predefined categories or scopes.
  • Experiments on prevalent multimodal models show marked hallucination reduction and notable enhancement of general model performance.

2. Related Work

Prior work addresses hallucinations through data cleaning, SFT, RLHF, DPO, and evaluation benchmarks, but existing methods and benchmarks have important coverage or data-quality limitations. HA-DPO instead combines preference learning with style-consistent hallucination data.

  • 2.1. Hallucination in LLMs: LLM hallucinations are responses that conflict with known facts, and prior work links them to pre-training noise and insufficient prior knowledge during SFT.
  • 2.2. Hallucination in LVLMs: Existing LVLM benchmarks overlook broad categories, attributes, emotions, and other hallucination types beyond object existence.
  • 2.2. Hallucination in LVLMs: Most hallucination-mitigation methods improve SFT data, while LRV and InstructBLIP construct substantial and diverse SFT datasets.
  • 2.2. Hallucination in LVLMs: Prior DPO-based hallucination reduction used only dis-preferred data, risking over-exploitation and complicating effective strategy learning.
  • 2.2. Hallucination in LVLMs: The paper integrates high-quality, human-free, style-consistent data into DPO to improve data quality and reinforcement-learning strategy.
  • 2.3. Human-preference Learning: RLHF learns a reward model from human preferences to guide policy optimization, whereas DPO bypasses explicit reward-model learning.

3. Our Method

HA-DPO reframes multimodal hallucination elimination as preference optimization, training models to favor non-hallucinatory responses through style-consistent positive and negative pairs. Its training combines direct preference optimization with auxiliary supervised learning to improve stability and preserve fluency.

  • Multimodal Hallucination-Aware DPO: HA-DPO extends Direct Preference Optimization to favor non-hallucinatory responses and reject hallucinatory alternatives.The method avoids explicit reward-model learning by directly optimizing the policy.
  • Hallucination Dataset Construction: The dataset pipeline generates descriptions, detects and corrects hallucinations, and augments pairs to maintain stylistic consistency.Visual Genome annotations provide detailed image information for constructing positive and negative samples, while GPT-4 performs detection and correction.
  • Multimodal Hallucination-Aware DPO: Maximizing the reward margin increases the positive sample’s likelihood while decreasing the negative sample’s likelihood.The positive response is non-hallucinatory, whereas the negative response is hallucinatory.
  • Multimodal Hallucination-Aware DPO: The auxiliary causal language-modeling loss integrates supervised fine-tuning gradients to mitigate performance regression and stabilize training.The total objective balances preference-learning and auxiliary language-modeling losses through λ.
  • Style Consistency Analysis: Without style alignment, sample-distribution differences coincide with declining sentence fluency and eventual loss of question-answering ability.After alignment, positive and negative samples occupy the same feature space, fluency remains stable, and the trained model eliminates hallucinations.
  • Style Consistency Analysis: Gradient analysis indicates that misaligned positive and negative sample distributions can induce training instability.The optimization gradient contains a reward-based component and a component that raises positive-response likelihood while lowering negative-response likelihood.

4. Sentencel-level Hallucination Ratio (SHR)

The Sentence-level Hallucination Ratio (SHR) broadens LVLM hallucination evaluation beyond fixed object categories and attributes. It measures textual descriptions that conflict with image content and supports broad, sentence-level coverage of hallucination types.

  • Benchmark Details: SHR addresses POPE’s restricted evaluation scope by measuring hallucinations across a broader range of image-grounded content.The benchmark is designed to avoid treating only predefined categories as relevant hallucinations.
  • Benchmark Details: SHR checks every textual description against image content rather than limiting evaluation to categories or attributes.This produces a quantifiable sentence-level hallucination measure.
  • Benchmark Advantages: SHR reports approximately 95% accuracy using manually annotated factual information to improve GPT-4’s hallucination judgments.The benchmark’s evaluation is grounded in factual annotations from Visual Genome images.
  • Benchmark Advantages: SHR supports unlimited object types and detects hallucinations involving nonexistent objects, emotions, attributes, movements, and other image-inconsistent descriptions.Its scope is broader than benchmarks limited to selected categories.

5. Experiments

Experiments evaluate HA-DPO across hallucination benchmarks, general capabilities, and qualitative responses. The method reduces hallucinations across models and improves broader performance, while β selection affects training stability.

  • Hallucination Mitigation: 35.0%, 28.04%, and 24.4% accuracy improvements were achieved by MiniGPT-4 on POPE’s Random, Popular, and Adversarial sets, respectively.F1-Score also improved by 17.83%, 11.53%, and 9.13% on these sets.
  • Hallucination Mitigation: 90.25% F1 score was achieved by LLaVA-1.5 after HA-DPO fine-tuning, establishing a new SOTA result on POPE.The improvement extended to the more challenging Popular and Adversarial sets.
  • Hallucination Mitigation: About 3% fewer hallucinatory sentences were produced by MiniGPT-4 on SHR after HA-DPO, while InstructBLIP and LLaVA-1.5 improved by 2.1% and 2.7%.The paper notes that hallucination remains a severe problem for LVLMs despite these reductions.
  • General Performance Enhancement: MiniGPT-4’s MME Perception increased from 726.72 to 1051.41, while Cognition increased from 169.64 to 233.57 after HA-DPO.InstructBLIP also improved by 71.32 on Perception.
  • Hallucination Mitigation: HA-DPO qualitatively reduced object existence, object attribute, movement, and physical hallucinations in model responses.The comparison examines responses before and after applying the hallucination elimination strategy.

6. Conclusion

The paper presents HA-DPO with style-consistent hallucination data to improve LVLM outputs and generalization. It also introduces SHR as a direct measure of hallucination in generated responses.

  • Conclusion: HA-DPO and style-consistent hallucination data encourage LVLMs to generate non-hallucinatory outputs while improving generalization capabilities.The conclusion describes the strategy as a method for enhancing LVLM performance.
  • Conclusion: SHR provides a direct measure of hallucination in model outputs.The paper identifies adaptation to real-world hallucination identification and reduction as future work.

7. Dataset

The dataset uses Visual Genome images and GPT-4-assisted processing to create hallucination preferences and evaluate sentence-level hallucination. Style-consistency rewriting aligns positive and negative samples.

  • Dataset: Visual Genome provides densely annotated images used for both hallucination training data and the SHR evaluation set.The dataset contains over 100,000 images averaging 21 objects, 18 attributes, and 18 relationships.
  • Dataset Construction: The data-generation pipeline has three stages: description generation, hallucination detection and correction, and style-consistency augmentation.The stages are illustrated through concrete examples in the dataset construction process.
  • Description Generation: LVLMs receive Visual Genome images with the instruction “Describe the image in detail.”Generation parameters are configured for both hallucination dataset construction and SHR evaluation.
  • Detection and Correction: GPT-4 detects and corrects hallucinated sentences, preserving the original as a negative sample and treating the corrected version as a positive sample.The process uses detailed Visual Genome annotations as image information.
  • Style-Consistency Augmentation: GPT rewrites both positive and negative samples to enforce style consistency between them.Separate examples illustrate negative and positive augmentation.

9. Details of SHR evaluation

SHR evaluates hallucination at the sentence level using GPT-4 judgments grounded in image facts. Additional analyses assess judgment reliability and whether style consistency stabilizes preference training.

  • SHR Evaluation: SHR is computed as the proportion of hallucinated sentences to total sentences in a model response.GPT-4 classifies each sentence as hallucination or correct during evaluation.
  • Judgment Reliability: Manual factual supplements correct GPT-4 judgments affected by insufficient Visual Genome annotations.The evaluation checks GPT-4 judgments and responses from MiniGPT-4, LLaVA-1.5, and InstructBLIP on 20 images.
  • Benchmark Comparison: HA-DPO outperforms competing hallucination-mitigation methods and achieves SOTA POPE accuracy and F1 score.The comparison is reported in Table 7.
  • Style-Consistency Analysis: Style-consistent control eases training instability and reduces degeneration, measured using n-gram fluency.Higher n-gram fluency indicates smoother sentences, while lower values indicate repetition and degeneration.
  • Style-Consistency Analysis: Style-consistent datasets produce more stable optimization gradients than datasets without style consistency.The gradient comparison is presented in Figure 7.

11. Comparison with other hallucination mitigation methods

HA-DPO outperforms other hallucination-mitigation methods on POPE accuracy while using substantially less training data than LRV. Its evaluation emphasizes holistic measures because POPE recall alone can reward excessive “Yes” responses.

  • HA-DPO achieves state-of-the-art POPE accuracy while using 2,000 images and 6,000 positive-negative reply pairs.LRV uses 400,000 training data, whereas HA-DPO uses the smaller image-and-pair dataset.
  • POPE recall can be misleading because it evaluates only the correctness of “Yes” responses.MiniGPT-4 can approach 100% recall while reaching only 51% accuracy when most “No” responses are incorrect.
  • Accuracy and F1 score provide holistic measures for evaluating hallucination on POPE.

12. Quality Examples

The quality examples illustrate hallucination-reduced outputs after HA-DPO optimization across three multimodal models. The surrounding materials cover hallucination detection, correction, style-consistent positive and negative samples, and SHR evaluation.

  • Quality Examples: HA-DPO-optimized MiniGPT4-LLaMA2-7B, InstructBLIP-13B, and LLaVA-1.5-7B produce less hallucinated content in visual question answering and image description tasks.
  • Quality Examples: The data-augmentation figures distinguish negative hallucinated samples from positive non-hallucinated samples for style-consistent training data.
  • Quality Examples: SHR is presented as an evaluation measure for sentence-level hallucination.
  • Quality Examples: The examples include hallucination-eliminated outputs from MiniGPT4-LLaMA2-7B, InstructBLIP-13B, and LLaVA-1.5.
Loading 2311.16839v2…