PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Multimodal Hallucination via Phase-wise Self-reward

Yu Zhang, Chuyang Sun, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang

arXiv:2604.17982v1cs.CVcs.CL

TL;DR

LVLM vision hallucination remains a reliability problem, while existing approaches require costly supervision and fine-tuning or overlook the dynamic timing of hallucination emergence. The paper introduces PSRD, which uses phase-wise self-reward signals distilled into a lightweight reward model for targeted inference-time correction, reducing hallucinations by 50.0% in LLaVA-1.5-7B and outperforming post-hoc methods across benchmarks and LVLMs.

  • Problem

    Vision hallucinations make LVLM outputs inconsistent with visual inputs, while existing mitigation methods depend on costly supervision and fine-tuning or static interventions that miss dynamic emergence.

  • Method

    PSRD analyzes phase-wise hallucination dynamics, distills LVLM uncertainty signals into a lightweight reward model, and uses online rewards for targeted decoding intervention.

  • Results

    50.0%: PSRD reduces hallucinations in LLaVA-1.5-7B and consistently outperforms existing post-hoc methods across five benchmarks and four LVLMs.

  • Takeaways & Limitations

    PSRD mitigates hallucination propagation while offering a controllable trade-off between hallucination reduction and inference efficiency.

  • Takeaways & Limitations

    Compared with MRGD, PSRD avoids extensive external annotation data, whereas MRGD incurs significant labeling costs that limit its generalizability.

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) still struggle with vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on large-scale annotated data for fine-tuning, which incurs massive computational overhead, or employ static post-hoc strategies that overlook the dynamic nature of hallucination emergence. To address these, we introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision. On the empirical side, we reveal that visual hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase. Drawing on these insights, we propose \textbf{PSRD} (\textbf{Phase-wise \textbf{S}elf-\textbf{R}eward \textbf{D}ecoding) for online hallucination correction guided by phase-wise self-reward signals. To reduce the cost of repeated self-evaluation during decoding, we distill the hallucination guidance signal from LVLMs into a lightweight reward model. The reward model subsequently provides on-the-fly guidance for targeted intervention during the decoding process, enabling precise hallucination suppression. The proposed PSRD significantly reduces the hallucination rate of LLaVA-1.5-7B by 50.0% and consistently outperforms existing post-hoc methods across five hallucination evaluation benchmarks for four LVLMs. Further analysis confirms that PSRD effectively mitigates hallucination propagation and achieves a highly controllable trade-off between strong performance and inference efficiency.

1 Introduction

LVLM vision hallucinations undermine reliability, while existing mitigation methods incur supervision and fine-tuning costs or miss when hallucinations emerge. PSRD addresses this with phase-aware self-reward decoding, lightweight reward guidance, and targeted inference-time intervention.

  • Motivation: Existing methods rely on annotated or preference data and full-model fine-tuning, while post-hoc and contrastive decoding overlook dynamic hallucination emergence.These limitations create high costs or imprecise intervention at critical junctures.
  • Motivation: Hallucination severity peaks at the onset of each semantic generation phase, identifying phase transitions as critical intervention points.PSRD uses this phase-wise pattern to target correction where hallucinations are most likely to emerge.
  • Method: PSRD distills LVLM uncertainty-based hallucination discrimination into a lightweight reward model that monitors decoding and triggers iterative, targeted interventions.This enables online mitigation without external supervision or full-model fine-tuning.
  • Results: 50.0%: PSRD reduces hallucinations in LLaVA-1.5-7B and consistently outperforms existing post-hoc methods across five benchmarks and four LVLMs.The framework also mitigates hallucination propagation and provides a controllable effectiveness–efficiency trade-off.

2 Analysis of Visual Hallucination Dynamics

The analysis measures hallucination dynamics across semantically coherent generation phases and reveals a recurring vulnerability at phase transitions. Hallucination severity consistently peaks when a new semantic segment begins.

  • Experimental setup: The study samples 500 COCO2014 images and generates captions with LLaVA-1.5-7B to analyze multimodal hallucination during decoding.It evaluates both phase-level and word-level hallucination rates.
  • Experimental setup: A phase is defined as a fine-grained, semantically coherent caption unit segmented using predefined textual delimiters.This definition supports localized analysis along the generation trajectory.
  • Findings: Hallucination severity consistently peaks at the onset of each semantic segment, making transitions vulnerable critical junctures.At each transition, the model initiates a new descriptive element while realigning linguistic context with visual evidence.

3 Phase-wise Self-Reward Decoding

PSRD constructs a lightweight reward model from self-elicited, phase-wise hallucination signals and uses it to guide targeted decoding interventions. Its constrained search selects intervention settings that improve reward while limiting evaluation overhead.

  • Reward model construction: PSRD distills LVLM hallucination discrimination into a lightweight reward model for real-time, phase-wise guidance during decoding.The framework uses self-elicited hallucination behaviors and uncertainty-guided weighting rather than repeated direct LVLM assessment.
  • Hallucination behavior elicitation: Hallucination behaviors are elicited from clean and noise-corrupted images paired with standard and hallucination-inducing prompts.These four input configurations systematically broaden the behavior signals used for reward-model supervision.
  • Phase-wise uncertainty signals: Responses are segmented into fine-grained semantic phases, and the LVLM is re-prompted on each phrase to obtain grounded-versus-hallucinated uncertainty supervision.The phase-wise signals are then used to calibrate the reward model.
  • Reward model construction: The reward model uses CLIP image-text cosine similarity and combines discriminative alignment, margin enforcement, and hallucination consistency objectives.The margin objective enforces a minimum separation δ between positive and negative alignment scores.
  • Reward-guided targeted intervention: Decoding intervention searches over an alternative initial-token rank k and contrastive penalty α, seeking R(x*) > τ with few reward evaluations.The rank is discrete, while α is refined for a selected seed trajectory.
  • Reward-guided targeted intervention: The bounded projection heuristic falls back to the best probed candidate when local reward trends are unstable, degenerate, or implausible.This preserves robustness instead of continuing unconstrained secant iterations.

4 Experiments

Experiments evaluate PSRD across generative, discriminative, cross-model, phase-level, reward-model, and efficiency settings. The results show broad hallucination reduction, reduced propagation, and a controllable effectiveness–efficiency trade-off.

  • Main Results: PSRD reduces LLaVA-1.5-7B hallucination by 50.0% on AMBER CHAIR and outperforms post-hoc methods across MMHal-Bench and Object HalBench metrics.On AMBER, it also achieves lower CHAIR, Hal, and Cog while maintaining high object coverage.
  • Main Results: PSRD improves LLaVA-1.5-7B F1-score by 13.9 and 5.4 percentage points on AMBER and the full POPE benchmark, respectively.It surpasses other post-hoc baselines on nearly all discriminative evaluation metrics.
  • Phase-wise Mitigation: The phase-level hallucination accumulation rate R_acc is 0.07% for PSRD, versus 0.35% for LLaVA-1.5-7B and 0.40% for M3ID.PSRD also reduces average hallucination rate by 13.2% across generation phases compared with LLaVA-1.5-7B.
  • Reward Model Analysis: The reward model’s L_HC loss reduces positive–negative score overlap to 0.348 by increasing intra-class compactness and inter-class separation.Removing uncertainty weighting raises AMBER CHAIR by 0.7 and lowers MHal-detect F1 by 1.1; removing LDA lowers F1 by 5.2.
  • Efficiency Analysis: At τ=30%, PSRD achieves a 67.2% CHAIR reduction over M3ID with a 4.0× inference-time increase, demonstrating a controllable efficiency–effectiveness trade-off.Lower thresholds reduce inference time but increase hallucination severity.
  • Human Preference: PSRD is preferred over M3ID in 48.5% of cases, while M3ID is preferred in 37.5% and both are rated equally good in 14.0%.This comparison indicates competitive generation quality alongside hallucination mitigation.

5 Related Work

LVLM hallucination mitigation has centered on training-based alignment and inference-time post-hoc correction, but existing approaches often overlook when hallucinations emerge during decoding.

  • Training-based Alignment: Training-based alignment fine-tunes LVLMs with preference datasets derived from human labeling or knowledge distillation.Methods such as HSA-DPO and HDPO construct preference pairs, while RLAIF-V and EOS use open-source model feedback.
  • Post-hoc Mitigation Strategies: Generate-then-revise methods use multi-step pipelines to correct responses after initial generation.Volcano uses natural-language feedback, while Woodpecker adds concept extraction and visual verification.
  • Motivation: Existing approaches often overlook the dynamic emergence of hallucinations, limiting precise intervention at critical decoding junctures.PSRD instead targets the onsets of semantic segments, where hallucination risk is most pronounced.
  • Reward-guided Decoding: Reward-guided decoding steers generation with auxiliary reward signals, but MRGD requires extensive external annotation data for its dual reward models.This requirement incurs significant labeling costs and limits MRGD’s generalizability.

6 Discussion

The discussion frames self-rewarding as a process-oriented route to improving LVLM reliability, using specialized reward models to guide frozen generators without parameter updates.

  • Process-oriented Perspective: Self-reward modeling favors process-oriented hallucination mitigation by discriminating unreliable intermediate outputs during generation.The paper characterizes discrimination as structurally more tractable than faithful long-horizon grounded generation.
  • Lightweight Model Self-evolution: Specialized reward models can provide fine-grained inference guidance through confidence calibration, process-level consistency, and grounding-oriented objectives.This approach avoids continual retraining of the base LVLM.
  • Implications: Self-rewarding is presented as a lightweight paradigm for model self-evolution and a scalable path toward more reliable, controllable, and self-improving LVLMs.The paper distinguishes this direction from improving capabilities solely by scaling the generator.

7 Conclusion

The paper concludes by introducing PSRD for phase-wise inference-time hallucination correction and by defining metrics for quantifying hallucination severity and propagation.

  • Conclusion: PSRD uses phase-wise self-reward signals for online hallucination correction without external supervision.A lightweight reward model distills LVLM guidance to enable targeted intervention during decoding.
  • Hallucination Metrics: Hallucination severity is quantified with phase-level and word-level hallucination rates during decoding.The supplied passages define these rates as measures used to quantify hallucination severity.
  • Hallucination Metrics: The word-level hallucination rate measures the empirical probability that a word is hallucinatory at a phase position and normalized within-phase position.The normalized word position j lies in [0, 1] across m evaluated samples.
  • Hallucination Metrics: The phase-level hallucination rate measures the probability that a phase contains at least one hallucinated word at a given phase position.It is calculated over m samples using a phase hallucination indicator.

B Details for Uncertainty-Guided Data Generation

The reward-model data pipeline elicits hallucinations through image or instruction corruption, segments generated text into semantic phases, and uses LVLM self-evaluation for labels.

  • Hallucination Elicitation: Image corruption adds zero-mean Gaussian noise with σ∈[0.2, 0.6] while preserving overall image semantics.This procedure is used to degrade the visual signal and elicit hallucination behavior.
  • Hallucination Elicitation: Instruction corruption anchors responses in observable content before prompting plausible but nonexistent additions.The procedure encourages fluent responses that blend factual and fabricated details.
  • Phase Segmentation: Punctuation, conjunctions, and spaCy syntactic parsing identify candidate boundaries and verify genuine semantic segmentation.These segments provide the phase structure used for reward-model data generation.
  • Self-annotation: The LVLM self-annotation pipeline checks object presence, quantities, spatial relationships, and attributes to produce image–phrase–label triplets.Labels require every phrase aspect to match the image for a positive judgment.

C.1 Details of Reward Model Training

The reward model uses weighted multi-objective training and guides PSRD through a two-stage Scout-and-Project search. The search exploits local reward trends for bounded early-phase intervention while terminating unreliable refinements.

  • Reward-model training: The reward model uses loss weights λ1 = 1.0, λ2 = 2.4, and λ3 = 0.1, with margin δ = 0.3.The Discriminative Alignment Loss is intended to dominate, while regularization maintains representation consistency.
  • Scout-and-Project search: PSRD identifies a satisficing intervention configuration (k, α) whose reward exceeds threshold τ using few reward evaluations.The lightweight reward model R evaluates candidate configurations during decoding.
  • Scout-and-Project search: Scout-and-Project first greedily evaluates top-K candidate branches at α = 0, motivated by hallucinations emerging near the initial token of each semantic phase.Candidates are ranked by their initial rewards before intervention-strength refinement.
  • Scout-and-Project search: The projection stage performs bounded local refinement of α using a secant-style update rather than globally convergent optimization.The method begins with a probe step and estimates a local slope from observed reward changes.
  • Local reward trends: PSRD bounds the search within [0, αmax] and switches branches when the local trend cannot support further projection.This combines local exploitation with safeguarded behavior across heterogeneous decoding trajectories.
  • Local reward trends: Across 145 sentence groups, 115 showed increasing reward within the probed interval, supporting local projection for many branches.PSRD does not require the trend to hold universally.

D.3 Hyperparameter Sensitivity

The appendix evaluates reward-model hyperparameter sensitivity and intervention timing. Default weights balance the training objectives, while early-phase intervention is more effective than delayed intervention.

  • Hyperparameter sensitivity: The reward-model hyperparameters are fixed at (λ1, λ2, λ3, δ) = (1.0, 2.4, 0.1, 0.3) across LVLMs and benchmarks.These values are used only during reward-model training, not as test-time decoding variables.
  • Hyperparameter sensitivity: The default hyperparameter setting lies in a reasonably stable region rather than a narrow isolated optimum on AMBER generative with LLaVA-1.5-7B.The analysis varies the trained reward model while keeping the PSRD framework unchanged.
  • Loss-weight rationale: The default weights balance the three reward-model objectives because the unweighted consistency loss has a substantially larger numerical scale.Balanced magnitudes yield more comparable gradient contributions during early training.
  • Intervention timing: Delayed intervention is clearly less effective than default early-phase intervention on AMBER generative.The delayed control begins at a random position between token 1 and the phase midpoint while preserving the generated prefix.

D.5 Reliability of Self-Evaluation Signals

The self-evaluation signals provide reasonably reliable weak supervision, and several controls examine phase segmentation, reward calibration, and the representativeness of induced hallucinations.

  • Self-evaluation reliability: On 3,407 samples, pseudo-labels achieve accuracy 0.8739, precision 0.9012, recall 0.8944, and F1 score 0.8978 against AMBER references.These results support using self-evaluation signals as weak supervision.
  • Phase-boundary definition: Punctuation-based phase segmentation performs comparably to an entropy-based alternative on AMBER generative.The comparison suggests that effective intervention depends on identifying vulnerable transition points rather than using a more elaborate boundary detector.
  • Reward calibration: Raw CLIP similarity performs worse than PSRD on AMBER generative, suggesting that distilled reward calibration is better aligned with hallucination mitigation.CLIP-raw computes image–text alignment directly from pretrained CLIP representations.
  • Hallucination construction: Induced hallucinations remain broadly similar to natural hallucinations in object-grounding behavior rather than becoming purely random or noisy text.The comparison uses natural, induced-corruption, and image-Gaussian-noise settings.

E.1 Details of the Datasets.

The evaluation covers generative, discriminative, and hallucination-classification tasks using established benchmarks and metrics. It compares PSRD with standard, fine-tuned, self-improved, and post-hoc mitigation methods under defined decoding settings.

  • Generative tasks: Generative mitigation uses AMBER, Object HalBench, and MMHalBench with CHAIR, Cover, Hal, Cog, and Overall quality metrics.Object HalBench reports CHAIR_i and CHAIR_s, while MMHalBench includes GPT-4-assessed Overall quality on a 0–6 scale.
  • Discriminative tasks: Discriminative mitigation is evaluated with Accuracy and F1-score across POPE’s Random, Popular, and Adversarial settings.POPE’s ALL metric is the arithmetic mean across those three settings.
  • Classification tasks: Reward-model classification is evaluated on AMBER HalDet and MHal-detect, which contains 16k fine-grained VQA annotations.MHal-detect supports sentence-level binary hallucination classification for detailed image descriptions.
  • Baselines: The mitigation comparison includes standard LVLMs, externally supervised fine-tuned models, self-improvement methods, and post-hoc approaches.The post-hoc baselines include VCD, ICD, AVISC, M3ID, OPERA, DeCo, MoD, ConVis, EAZY, ALGA, ONLY, MRGD, and Octopus.
  • Evaluation settings: The default Scout-and-Project setting uses k = 5, δ = 0.5, αmax = 3.0, and acceptance threshold τ = 30.The method searches top-k branches and refines α up to αmax with a secant-style update.
  • Cross-model evaluation: A reward model distilled from LLaVA-1.5-7B is applied to LLaVA-Next-7B and InstructBlip-7B for dynamic mitigation evaluation.The cross-model application is motivated in part by InstructBlip-7B’s poor intrinsic self-hallucination detection capacity.
Loading 2604.17982v1…