PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, Dongsheng Li

arXiv:2501.09695v2cs.CV

TL;DR

LVLM hallucinations remain a major challenge, and DPO-based mitigation is hindered when preference data is off-policy relative to its reference policy. OPA-DPO uses expert corrections and on-policy alignment before DPO training, achieving stronger hallucination mitigation with 4.8k samples than prior methods using larger datasets.

  • Problem

    DPO-based hallucination mitigation varies with data construction, while existing methods often neglect preference-data alignment with the reference policy.

  • Method

    OPA-DPO uses GPT4V to correct hallucinated responses, fine-tunes the initial policy on original and revised responses, and applies DPO afterward.

  • Results

    With 4.8k training samples, OPA-DPO achieves SOTA performance on over half of hallucination-related benchmarks, surpassing other DPO-based algorithms generally requiring over 10k data.

  • Takeaways & Limitations

    On-policy alignment enables DPO to incorporate expert corrections while reducing hallucinations with substantially less training data than prior DPO-based methods.

  • Takeaways & Limitations

    Hallucination Recognition methods can leave preferred responses off-policy, while Hallucination Injection does not explicitly identify or substantially reduce model-intrinsic hallucinations.

Abstract

from arXiv · show

Hallucination remains a major challenge for Large Vision-Language Models (LVLMs). Direct Preference Optimization (DPO) has gained increasing attention as a simple solution to hallucination issues. It directly learns from constructed preference pairs that reflect the severity of hallucinations in responses to the same prompt and image. Nonetheless, different data construction methods in existing works bring notable performance variations. We identify a crucial factor here: outcomes are largely contingent on whether the constructed data aligns on-policy w.r.t the initial (reference) policy of DPO. Theoretical analysis suggests that learning from off-policy data is impeded by the presence of KL-divergence between the updated policy and the reference policy. From the perspective of dataset distribution, we systematically summarize the inherent flaws in existing algorithms that employ DPO to address hallucination issues. To alleviate the problems, we propose On-Policy Alignment (OPA)-DPO framework, which uniquely leverages expert feedback to correct hallucinated responses and aligns both the original and expert-revised responses in an on-policy manner. Notably, with only 4.8k data, OPA-DPO achieves an additional reduction in the hallucination rate of LLaVA-1.5-7B: 13.26% on the AMBER benchmark and 5.39% on the Object-Hal benchmark, compared to the previous SOTA algorithm trained with 16k samples. Our implementation is available at https://github.com/zhyang2226/OPA-DPO.

1. Introduction

LVLM hallucinations remain a major obstacle, while DPO-based mitigation depends critically on whether preference data is on-policy relative to the reference policy. OPA-DPO addresses this alignment problem by using expert corrections and limited data.

  • Motivation: 45.9% of GPT-4V responses hallucinate on certain tasks, underscoring hallucination as a practical LVLM barrier.Hallucinations are discrepancies between image content and generated text.
  • Existing methods: Existing methods fall into hallucination injection, hallucination recognition, and self-evolution, each with distinct data-distribution weaknesses.Injection errors are not model-originated, recognition revisions are off-policy, and self-evolution may retain hallucinations in both responses.
  • DPO and on-policy data: DPO learns from offline preference pairs, but alignment with the initial policy is rarely considered in LVLM applications.Unlike PPO, DPO does not require online rollout data and may use datasets collected by other policies.
  • DPO and on-policy data: Strictly off-policy preferred responses cannot be learned by naive DPO because support mismatches create substantially large reverse KL-divergence.Even assigning a small positive probability to such responses can produce this KL constraint.
  • OPA-DPO: OPA-DPO uses GPT4V to revise model-generated responses, fine-tunes the initial policy on those corrections, and then applies DPO to aligned data.This operation is intended to circumvent KL-divergence constraints during subsequent DPO training.
  • OPA-DPO: 4.8k data enables OPA-DPO to achieve SOTA hallucination-benchmark performance while surpassing methods relying on larger datasets.The paper identifies on-policy data reliance as an intrinsic property of DPO.

2. Preliminary

The preliminary section introduces LVLMs, SFT, DPO, and the paper’s operational definition of on-policy data. It presents how preference optimization uses a reward formulation and preference pairs, while SFT trains on standard responses.

  • LVLMs combine a visual encoder, modality connection module, and LLM to generate natural-language outputs from images and prompts.
  • DPO analytically expresses the reward through the updated and reference policies, with β controlling KL-regularization strength.
  • DPO directly optimizes preference pairs comparing winning and losing responses for the same prompt and image through a sigmoid objective.
  • SFT aligns pretrained language and multimodal models with downstream tasks using prompts, images, and corresponding standard responses.
  • A response is on-policy when its reference-policy probability exceeds the small positive threshold ϵ.

3. Problem Analysis

The analysis examines how data distribution relative to DPO’s reference policy affects learning and identifies limitations in existing hallucination-mitigation strategies. It motivates aligning expert corrections with the policy before DPO training.

  • On-policy data: DPO performance depends on whether preference data aligns with the initial/reference policy.The paper frames this as Question 1 and links the issue to DPO’s offline training setup.
  • On-policy data: If a preferred response has positive probability under the updated policy but near-zero probability under the reference policy, KL-divergence tends to infinity.This support mismatch prevents the updated policy from assigning probability to strictly off-policy responses.
  • On-policy data: Off-policy preferred responses receive diminishing DPO updating weight, making effective learning nearly impossible.A single log-likelihood update is insufficient to make such responses likely during autoregressive generation.
  • Existing methods: Hallucination Injection raises the probability of ground-truth responses but does not explicitly identify or substantially reduce model-intrinsic hallucinations.The method treats a ground-truth response as on-policy after incorporating it into the SFT dataset.
  • Existing methods: Hallucination Recognition often leaves preferred responses off-policy, while Self Evolution cannot effectively correct hallucinations shared by both generated responses.Self Evolution consequently requires substantial data and multiple iterative updates.
  • Proposed direction: OPA-DPO combines expert-revised responses with on-policy alignment to address the shortcomings of off-policy expert corrections and unreliable self-generated preferences.Its four steps collect original-policy responses, correct hallucinations with GPT-4V, apply LoRA-SFT, and begin OPA-DPO training.

4. On-Policy Alignment DPO

OPA-DPO first collects responses from the training policy, uses GPT-4V to revise hallucinations, and fine-tunes that policy before preference optimization. Its training combines language corrections with image-focus and anchored-preference mechanisms.

  • Framework: OPA-DPO uses four stages: response collection, GPT-4V revision, LoRA-SFT alignment, and OPA-DPO training.The final OPA-DPO stage starts from the policy produced by LoRA-SFT.
  • Data collection: GPT-4V evaluates generated responses sentence by sentence to identify hallucinations before revised responses enter alignment training.The generated responses are paired with prompts, images, and ground-truth responses for evaluation.
  • On-policy alignment: LoRA-SFT trains on ground-truth and GPT-4V-revised responses, producing πOPA as the reference policy for subsequent OPA-DPO training.This alignment step makes the corrected responses part of the policy used for preference optimization.
  • Language corrections: OPA-DPO forms language-correction preference pairs between ground-truth or revised responses and the generated response.The loss includes two language-level pairs rather than the single preference pair used by classical DPO.
  • Combined objective: OPA-DPO combines language-correction, image-focus, and anchored-preference losses, with each component retained in the full objective.The paper states that omitting any component is inappropriate and refers to ablations for further analysis.
  • Language corrections: Hallucination-weighted log-policy scores assign the same hallucination score to tokens within a sentence while allowing scores to vary across sentences.This weighting supports the construction of language-correction preference pairs.
  • Image focus: The image-focus mechanism compares original and distorted images with the same prompts and ground-truth or revised responses.It targets the model’s tendency to ignore visual input during optimization.

5. Experiments

Experiments evaluate OPA-DPO on two LLaVA models, multiple hallucination benchmarks, policy-distribution diagnostics, data-scaling tests, and ablations. Results show that on-policy alignment improves hallucination metrics, while conservative responses and omitted details create coverage trade-offs.

  • Experimental Setup: OPA-DPO is evaluated on LLaVA-v1.5-7B and 13B using 4.8K samples from RLAIF-V, across AMBER, MMHalBench, and POPE.The models use CLIP ViT-L-336px vision encoders; POPE evaluation uses its 3,000-case Adversarial set.
  • Policy Diagnostics: Response-averaged log probabilities barely change after naive DPO but increase substantially after OPA-DPO on 200 significantly revised responses.The result supports improved learning of off-policy preferred responses after on-policy alignment.
  • Policy Diagnostics: Naive DPO leaves the reverse KL-divergence gap largely unchanged, with the divergence between OPA-DPO and the base policy nearly an order of magnitude larger.These measurements indicate that naive DPO is insufficient to bridge the policy-support gap.
  • Benchmark Evaluation Results: OPA-DPO achieves SOTA performance on 50% of hallucination metrics for LLaVA-7B and 70% for LLaVA-13B, especially on CHAIR and HalRate.The improvement comes with a slight compromise in the coverage metric Cover.
  • Benchmark Evaluation Results: OPA-DPO surpasses most baselines on hallucination-related metrics with 600 samples, and its performance improves markedly as the training set grows.By contrast, increasing data volume produces no significant improvement for πOPA, the policy after LoRA-SFT.
  • Ablation Studies: OPA-DPO’s on-policy alignment reduces AMBER HalRate and Object-hal CHAIRs by nearly 50% versus training without OPA.The no-OPA policy performs nearly identically to RLHF-V and mDPO, which do not account for on-policy data.
  • Ablation Studies: Using all OPA-DPO loss components resolves repetition and missing-EOS failures observed when Image Focus or Anchored Preference is omitted.The issue is particularly pronounced in long-form AMBER generation tasks.
  • Qualitative Analysis: Qualitative results show that OPA-DPO nearly eliminates hallucinations after OPA on 4.8K samples, while omitting some details from the original description.Naive DPO without OPA leaves hallucinations and flawed reasoning largely intact.

6. Related Works

Related work covers RLHF/RLAIF, DPO alternatives, and hallucination-reduction methods for LVLMs. The paper positions OPA-DPO as the only compared DPO-based algorithm combining expert correction, on-policy data, and image focus.

  • RLHF and DPO: RLHF aligns language and vision-language models using human preference data and reinforcement learning, with PPO as the original RLHF algorithm.PPO is stable but requires a reward model and many hyperparameters.
  • RLHF and DPO: DPO removes the need for a separate reward model but has not yet matched PPO’s performance, motivating alternatives such as ORPO, CPO, TPO, and SimPO.These methods seek stronger alignment with preference data through different objective designs.
  • Hallucination Reduction for LVLMs: LVLM hallucination-reduction methods divide into RL-free decoding or representation approaches and RL-based methods using human or AI feedback.RL-based methods generally perform better on hallucination benchmarks, while most such studies use DPO rather than PPO.
  • Hallucination Reduction for LVLMs: OPA-DPO is the only compared algorithm that combines expert correction, on-policy data, and image focus, achieving SOTA performance across multiple metrics.The comparison is framed around three dimensions of DPO-based hallucination algorithms.

7. Conclusions

The study identifies on-policy data as crucial for effective DPO and introduces OPA-DPO to address flaws in existing hallucination-mitigation methods. OPA-DPO uses expert feedback and achieves strong benchmark performance with only 4.8k training samples.

  • DPO’s effectiveness depends heavily on the on-policy property of its training data.
  • OPA-DPO corrects hallucinated responses with expert feedback and aligns both original and revised responses on-policy.
  • 4.8k training samples enable OPA-DPO to achieve SOTA performance on over half of hallucination-related benchmarks for two LLaVA models.The paper states that competing DPO-based algorithms generally require over 10k data.
  • The appendix provides implementation details, additional benchmark results, and further ablation studies.

A.1. Training Details.

OPA-DPO collects on-policy generations, obtains sentence-level GPT-4V feedback and revisions, then applies two training phases combining alignment and preference optimization.

  • OPA-DPO uses consistent settings across LLaVA-v1.5-7B and LLaVA-v1.5-13B, with LoRA applied across the backbone model.The implementation also specifies cosine learning-rate schedules and fixed coefficients for the training equations.
  • The training procedure begins by sampling model responses from pre-collected images and prompts using top-k and top-p sampling.The settings are topk = 30, topp = 0.95, and temperature 1.0.
  • Phase 2 applies OPA-DPO using generated, ground-truth, revised, and scored responses, together with language- and image-weighted losses.The corresponding mappings are described in Tables 6 and 7.
  • GPT-4V assigns sentence-level hallucination severity scores and produces minimal revisions for erroneous sentences.Hallucination scores range from one to four, while revised sentences are aggregated as yRev.
  • Phase 1 performs On-Policy Alignment by LoRA-SFT on ground-truth and GPT-4V-revised responses before preference training.Algorithm 1 denotes the resulting policy as πOPA.
  • The GPT-4V prompt evaluates each generated sentence, preserves accurate content, and minimally modifies or replaces inaccurate content.It requests structured outputs containing copied content, scores, error types, object corrections, and revisions.

B.1. Helpfulness Benchmark Evaluations.

Helpfulness evaluations on LLaVA-Bench show that OPA-DPO maintains an upper-middle performance level while reducing hallucinations.

  • OPA-DPO performs at an upper-middle level on the LLaVA-Bench helpfulness evaluation.
  • Except for LLaVA-RLHF, the evaluated algorithms show minimal performance variation on LLaVA-Bench.The comparison is reported in Table 8.

B.2. Additional Ablation Studies.

Ablations show that the image-focus coefficient has a larger effect than the distorted-image mask ratio, motivating a compromise setting for γ1.

  • γ1 = 0.2 and randomly masking 30% of pixels are optimal for both evaluated LLaVA models.
  • The mask ratio has a slight performance impact, whereas the image-focus coefficient γ1 has a more significant effect.
  • Excessively large γ1 improves hallucination-rate metrics but makes the model overly conservative and reduces explanatory detail.Greedy sampling can also induce sentence or word repetition and prevent EOS generation.
  • The authors select γ1 = 0.2 as a compromise between hallucination-related performance and response quality.

C. Additional Qualitative Examples

Additional qualitative examples show OPA-DPO reducing hallucinated content through conservative responses, while false-premise queries expose persistent hallucinations in LVLMs.

  • Image Descriptions: OPA-DPO adopts a conservative strategy that avoids uncertain assertions, enhancing response credibility but sometimes omitting ambiguous image details.This trade-off is illustrated through image-description examples in Figures 8–11.
  • False Premise Queries: False-premise queries consistently trigger hallucinations because they ask about objects or details absent from, or irrelevant to, the image.Figures 12–14 provide examples, including a request about a girl’s outfit accompanying a basketball image.
  • Image Descriptions: Figures 8–11 mark hallucinated parts in red and missing details in yellow across qualitative model comparisons.Figure 9 also highlights flawed reasoning in blue.
Loading 2501.09695v2…