PaperlayerView exact revision on arXiv

Source-linked AI summary

Mitigating Multimodal Hallucination via Phase-wise Self-reward

Yu Zhang, Chuyang Sun, Kehai Chen, Xuefeng Bai, Yang Xiang, Min Zhang

arXiv:2604.17982v1cs.CVcs.CL

TL;DR

LVLM vision hallucination remains a reliability problem, while existing approaches require costly supervision and fine-tuning or overlook the dynamic timing of hallucination emergence. The paper introduces PSRD, which uses phase-wise self-reward signals distilled into a lightweight reward model for targeted inference-time correction, reducing hallucinations by 50.0% in LLaVA-1.5-7B and outperforming post-hoc methods across benchmarks and LVLMs.

  • Problem

    Vision hallucinations make LVLM outputs inconsistent with visual inputs, while existing mitigation methods depend on costly supervision and fine-tuning or static interventions that miss dynamic emergence.

    Source: §1 · p. 1

  • Method

    PSRD analyzes phase-wise hallucination dynamics, distills LVLM uncertainty signals into a lightweight reward model, and uses online rewards for targeted decoding intervention.

    Source: §1 · p. 1, §7 · p. 9

  • Results

    50.0%: PSRD reduces hallucinations in LLaVA-1.5-7B and consistently outperforms existing post-hoc methods across five benchmarks and four LVLMs.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    PSRD mitigates hallucination propagation while offering a controllable trade-off between hallucination reduction and inference efficiency.

    Source: §1 · p. 2, §4.6 · p. 8

    Compared with MRGD, PSRD avoids extensive external annotation data, whereas MRGD incurs significant labeling costs that limit its generalizability.

    Source: §5.2 · p. 9

Abstract

from arXiv · show

Large Vision-Language Models (LVLMs) still struggle with vision hallucination, where generated responses are inconsistent with the visual input. Existing methods either rely on large-scale annotated data for fine-tuning, which incurs massive computational overhead, or employ static post-hoc strategies that overlook the dynamic nature of hallucination emergence. To address these, we introduce a new self-rewarding framework, enabling dynamic hallucination mitigation at inference time without external supervision. On the empirical side, we reveal that visual hallucination exhibits phase-wise dynamic patterns, peaking at the onset of each semantic phase. Drawing on these insights, we propose \textbf{PSRD} (\textbf{Phase-wise \textbf{S}elf-\textbf{R}eward \textbf{D}ecoding) for online hallucination correction guided by phase-wise self-reward signals. To reduce the cost of repeated self-evaluation during decoding, we distill the hallucination guidance signal from LVLMs into a lightweight reward model. The reward model subsequently provides on-the-fly guidance for targeted intervention during the decoding process, enabling precise hallucination suppression. The proposed PSRD significantly reduces the hallucination rate of LLaVA-1.5-7B by 50.0% and consistently outperforms existing post-hoc methods across five hallucination evaluation benchmarks for four LVLMs. Further analysis confirms that PSRD effectively mitigates hallucination propagation and achieves a highly controllable trade-off between strong performance and inference efficiency.

1 Introduction

LVLM vision hallucinations undermine reliability, while existing mitigation methods incur supervision and fine-tuning costs or miss when hallucinations emerge. PSRD addresses this with phase-aware self-reward decoding, lightweight reward guidance, and targeted inference-time intervention.

Motivation

  • Existing methods rely on annotated or preference data and full-model fine-tuning, while post-hoc and contrastive decoding overlook dynamic hallucination emergence.These limitations create high costs or imprecise intervention at critical junctures.Source: §1 · p. 1
  • Hallucination severity peaks at the onset of each semantic generation phase, identifying phase transitions as critical intervention points.PSRD uses this phase-wise pattern to target correction where hallucinations are most likely to emerge.Source: §1 · p. 1, Figure 2 · p. 2

Method

  • PSRD distills LVLM uncertainty-based hallucination discrimination into a lightweight reward model that monitors decoding and triggers iterative, targeted interventions.This enables online mitigation without external supervision or full-model fine-tuning.Source: §1 · pp. 1–2

Results

  • 50.0%: PSRD reduces hallucinations in LLaVA-1.5-7B and consistently outperforms existing post-hoc methods across five benchmarks and four LVLMs.The framework also mitigates hallucination propagation and provides a controllable effectiveness–efficiency trade-off.Source: §1 · p. 2

2 Analysis of Visual Hallucination Dynamics

The analysis measures hallucination dynamics across semantically coherent generation phases and reveals a recurring vulnerability at phase transitions. Hallucination severity consistently peaks when a new semantic segment begins.

Experimental setup

  • The study samples 500 COCO2014 images and generates captions with LLaVA-1.5-7B to analyze multimodal hallucination during decoding.It evaluates both phase-level and word-level hallucination rates.Source: §2 · p. 2
  • A phase is defined as a fine-grained, semantically coherent caption unit segmented using predefined textual delimiters.This definition supports localized analysis along the generation trajectory.Source: §2 · p. 2

Findings

  • Hallucination severity consistently peaks at the onset of each semantic segment, making transitions vulnerable critical junctures.At each transition, the model initiates a new descriptive element while realigning linguistic context with visual evidence.Source: §2 · p. 2

3 Phase-wise Self-Reward Decoding

PSRD constructs a lightweight reward model from self-elicited, phase-wise hallucination signals and uses it to guide targeted decoding interventions. Its constrained search selects intervention settings that improve reward while limiting evaluation overhead.

Reward model construction

  • PSRD distills LVLM hallucination discrimination into a lightweight reward model for real-time, phase-wise guidance during decoding.The framework uses self-elicited hallucination behaviors and uncertainty-guided weighting rather than repeated direct LVLM assessment.Source: §3 · p. 2, §3.1 · pp. 2–3

Hallucination behavior elicitation

  • Hallucination behaviors are elicited from clean and noise-corrupted images paired with standard and hallucination-inducing prompts.These four input configurations systematically broaden the behavior signals used for reward-model supervision.Source: §3.1 · p. 2

Phase-wise uncertainty signals

  • Responses are segmented into fine-grained semantic phases, and the LVLM is re-prompted on each phrase to obtain grounded-versus-hallucinated uncertainty supervision.The phase-wise signals are then used to calibrate the reward model.Source: §3.1 · p. 3

Reward model construction

  • The reward model uses CLIP image-text cosine similarity and combines discriminative alignment, margin enforcement, and hallucination consistency objectives.The margin objective enforces a minimum separation δ between positive and negative alignment scores.Source: §3.1 · p. 3

Reward-guided targeted intervention

  • Decoding intervention searches over an alternative initial-token rank k and contrastive penalty α, seeking R(x*) > τ with few reward evaluations.The rank is discrete, while α is refined for a selected seed trajectory.Source: §3.2 · p. 3
  • The bounded projection heuristic falls back to the best probed candidate when local reward trends are unstable, degenerate, or implausible.This preserves robustness instead of continuing unconstrained secant iterations.Source: §3.2 · pp. 3–4

4 Experiments

Experiments evaluate PSRD across generative, discriminative, cross-model, phase-level, reward-model, and efficiency settings. The results show broad hallucination reduction, reduced propagation, and a controllable effectiveness–efficiency trade-off.

Main Results

  • PSRD reduces LLaVA-1.5-7B hallucination by 50.0% on AMBER CHAIR and outperforms post-hoc methods across MMHal-Bench and Object HalBench metrics.On AMBER, it also achieves lower CHAIR, Hal, and Cog while maintaining high object coverage.Source: §4.2 · p. 4
  • PSRD improves LLaVA-1.5-7B F1-score by 13.9 and 5.4 percentage points on AMBER and the full POPE benchmark, respectively.It surpasses other post-hoc baselines on nearly all discriminative evaluation metrics.Source: §4.2 · p. 5

Phase-wise Mitigation

  • The phase-level hallucination accumulation rate R_acc is 0.07% for PSRD, versus 0.35% for LLaVA-1.5-7B and 0.40% for M3ID.PSRD also reduces average hallucination rate by 13.2% across generation phases compared with LLaVA-1.5-7B.Source: §4.4 · p. 6

Reward Model Analysis

  • The reward model’s L_HC loss reduces positive–negative score overlap to 0.348 by increasing intra-class compactness and inter-class separation.Removing uncertainty weighting raises AMBER CHAIR by 0.7 and lowers MHal-detect F1 by 1.1; removing LDA lowers F1 by 5.2.Source: §4.5 · p. 7

Efficiency Analysis

  • At τ=30%, PSRD achieves a 67.2% CHAIR reduction over M3ID with a 4.0× inference-time increase, demonstrating a controllable efficiency–effectiveness trade-off.Lower thresholds reduce inference time but increase hallucination severity.Source: §4.6 · pp. 7–8

Human Preference

  • PSRD is preferred over M3ID in 48.5% of cases, while M3ID is preferred in 37.5% and both are rated equally good in 14.0%.This comparison indicates competitive generation quality alongside hallucination mitigation.Source: §4.6 · p. 8

5 Related Work

LVLM hallucination mitigation has centered on training-based alignment and inference-time post-hoc correction, but existing approaches often overlook when hallucinations emerge during decoding.

Training-based Alignment

  • Training-based alignment fine-tunes LVLMs with preference datasets derived from human labeling or knowledge distillation.Methods such as HSA-DPO and HDPO construct preference pairs, while RLAIF-V and EOS use open-source model feedback.Source: §5.1 · p. 8

Post-hoc Mitigation Strategies

  • Generate-then-revise methods use multi-step pipelines to correct responses after initial generation.Volcano uses natural-language feedback, while Woodpecker adds concept extraction and visual verification.Source: §5.1 · p. 8

Motivation

  • Existing approaches often overlook the dynamic emergence of hallucinations, limiting precise intervention at critical decoding junctures.PSRD instead targets the onsets of semantic segments, where hallucination risk is most pronounced.Source: §5.1 · p. 8

Reward-guided Decoding

  • Reward-guided decoding steers generation with auxiliary reward signals, but MRGD requires extensive external annotation data for its dual reward models.This requirement incurs significant labeling costs and limits MRGD’s generalizability.Source: §5.2 · pp. 8–9

6 Discussion

The discussion frames self-rewarding as a process-oriented route to improving LVLM reliability, using specialized reward models to guide frozen generators without parameter updates.

Process-oriented Perspective

  • Self-reward modeling favors process-oriented hallucination mitigation by discriminating unreliable intermediate outputs during generation.The paper characterizes discrimination as structurally more tractable than faithful long-horizon grounded generation.Source: §6 · p. 9

Lightweight Model Self-evolution

  • Specialized reward models can provide fine-grained inference guidance through confidence calibration, process-level consistency, and grounding-oriented objectives.This approach avoids continual retraining of the base LVLM.Source: §6 · p. 9

Implications

  • Self-rewarding is presented as a lightweight paradigm for model self-evolution and a scalable path toward more reliable, controllable, and self-improving LVLMs.The paper distinguishes this direction from improving capabilities solely by scaling the generator.Source: §6 · p. 9

7 Conclusion

The paper concludes by introducing PSRD for phase-wise inference-time hallucination correction and by defining metrics for quantifying hallucination severity and propagation.

Conclusion

  • PSRD uses phase-wise self-reward signals for online hallucination correction without external supervision.A lightweight reward model distills LVLM guidance to enable targeted intervention during decoding.Source: §7 · p. 9

Hallucination Metrics

  • Hallucination severity is quantified with phase-level and word-level hallucination rates during decoding.The supplied passages define these rates as measures used to quantify hallucination severity.Source: A Details for Hallucination Rate in… · p. 11
  • The word-level hallucination rate measures the empirical probability that a word is hallucinatory at a phase position and normalized within-phase position.The normalized word position j lies in [0, 1] across m evaluated samples.Source: A Details for Hallucination Rate in… · p. 11
  • The phase-level hallucination rate measures the probability that a phase contains at least one hallucinated word at a given phase position.It is calculated over m samples using a phase hallucination indicator.Source: A Details for Hallucination Rate in… · p. 11

B Details for Uncertainty-Guided Data Generation

The reward-model data pipeline elicits hallucinations through image or instruction corruption, segments generated text into semantic phases, and uses LVLM self-evaluation for labels.

Hallucination Elicitation

  • Image corruption adds zero-mean Gaussian noise with σ∈[0.2, 0.6] while preserving overall image semantics.This procedure is used to degrade the visual signal and elicit hallucination behavior.Source: §B.1 · p. 11
  • Instruction corruption anchors responses in observable content before prompting plausible but nonexistent additions.The procedure encourages fluent responses that blend factual and fabricated details.Source: §B.1 · p. 11

Phase Segmentation

  • Punctuation, conjunctions, and spaCy syntactic parsing identify candidate boundaries and verify genuine semantic segmentation.These segments provide the phase structure used for reward-model data generation.Source: §B.2 · p. 11

Self-annotation

  • The LVLM self-annotation pipeline checks object presence, quantities, spatial relationships, and attributes to produce image–phrase–label triplets.Labels require every phrase aspect to match the image for a positive judgment.Source: §B.2 · p. 11

C.1 Details of Reward Model Training

The reward model uses weighted multi-objective training and guides PSRD through a two-stage Scout-and-Project search. The search exploits local reward trends for bounded early-phase intervention while terminating unreliable refinements.

Reward-model training

  • The reward model uses loss weights λ1 = 1.0, λ2 = 2.4, and λ3 = 0.1, with margin δ = 0.3.The Discriminative Alignment Loss is intended to dominate, while regularization maintains representation consistency.Source: §C.1 · p. 11

Scout-and-Project search

  • PSRD identifies a satisficing intervention configuration (k, α) whose reward exceeds threshold τ using few reward evaluations.The lightweight reward model R evaluates candidate configurations during decoding.Source: §C.3 · p. 12
  • Scout-and-Project first greedily evaluates top-K candidate branches at α = 0, motivated by hallucinations emerging near the initial token of each semantic phase.Candidates are ranked by their initial rewards before intervention-strength refinement.Source: §C.3 · p. 12
  • The projection stage performs bounded local refinement of α using a secant-style update rather than globally convergent optimization.The method begins with a probe step and estimates a local slope from observed reward changes.Source: §C.3 · p. 12

Local reward trends

  • PSRD bounds the search within [0, αmax] and switches branches when the local trend cannot support further projection.This combines local exploitation with safeguarded behavior across heterogeneous decoding trajectories.Source: §C.3 · p. 12, §D.1 · p. 13
  • Across 145 sentence groups, 115 showed increasing reward within the probed interval, supporting local projection for many branches.PSRD does not require the trend to hold universally.Source: §D.1 · p. 13

D.3 Hyperparameter Sensitivity

The appendix evaluates reward-model hyperparameter sensitivity and intervention timing. Default weights balance the training objectives, while early-phase intervention is more effective than delayed intervention.

Hyperparameter sensitivity

  • The reward-model hyperparameters are fixed at (λ1, λ2, λ3, δ) = (1.0, 2.4, 0.1, 0.3) across LVLMs and benchmarks.These values are used only during reward-model training, not as test-time decoding variables.Source: §D.3 · p. 13
  • The default hyperparameter setting lies in a reasonably stable region rather than a narrow isolated optimum on AMBER generative with LLaVA-1.5-7B.The analysis varies the trained reward model while keeping the PSRD framework unchanged.Source: §D.3 · p. 13, Table 8 · p. 14

Loss-weight rationale

  • The default weights balance the three reward-model objectives because the unweighted consistency loss has a substantially larger numerical scale.Balanced magnitudes yield more comparable gradient contributions during early training.Source: §D.3 · p. 13, Figure 8 · p. 14

Intervention timing

  • Delayed intervention is clearly less effective than default early-phase intervention on AMBER generative.The delayed control begins at a random position between token 1 and the phase midpoint while preserving the generated prefix.Source: §D.4 · pp. 13–14

D.5 Reliability of Self-Evaluation Signals

The self-evaluation signals provide reasonably reliable weak supervision, and several controls examine phase segmentation, reward calibration, and the representativeness of induced hallucinations.

Self-evaluation reliability

  • On 3,407 samples, pseudo-labels achieve accuracy 0.8739, precision 0.9012, recall 0.8944, and F1 score 0.8978 against AMBER references.These results support using self-evaluation signals as weak supervision.Source: §D.5 · p. 14, Figure 9 · p. 15

Phase-boundary definition

  • Punctuation-based phase segmentation performs comparably to an entropy-based alternative on AMBER generative.The comparison suggests that effective intervention depends on identifying vulnerable transition points rather than using a more elaborate boundary detector.Source: §D.6 · pp. 14–15, Table 10 · p. 15

Reward calibration

  • Raw CLIP similarity performs worse than PSRD on AMBER generative, suggesting that distilled reward calibration is better aligned with hallucination mitigation.CLIP-raw computes image–text alignment directly from pretrained CLIP representations.Source: §D.7 · p. 15, Table 11 · p. 15

Hallucination construction

  • Induced hallucinations remain broadly similar to natural hallucinations in object-grounding behavior rather than becoming purely random or noisy text.The comparison uses natural, induced-corruption, and image-Gaussian-noise settings.Source: §D.8 · p. 15

E.1 Details of the Datasets.

The evaluation covers generative, discriminative, and hallucination-classification tasks using established benchmarks and metrics. It compares PSRD with standard, fine-tuned, self-improved, and post-hoc mitigation methods under defined decoding settings.

Generative tasks

  • Generative mitigation uses AMBER, Object HalBench, and MMHalBench with CHAIR, Cover, Hal, Cog, and Overall quality metrics.Object HalBench reports CHAIR_i and CHAIR_s, while MMHalBench includes GPT-4-assessed Overall quality on a 0–6 scale.Source: §E.1 · pp. 15–16

Discriminative tasks

  • Discriminative mitigation is evaluated with Accuracy and F1-score across POPE’s Random, Popular, and Adversarial settings.POPE’s ALL metric is the arithmetic mean across those three settings.Source: §E.1 · p. 16

Classification tasks

  • Reward-model classification is evaluated on AMBER HalDet and MHal-detect, which contains 16k fine-grained VQA annotations.MHal-detect supports sentence-level binary hallucination classification for detailed image descriptions.Source: §E.1 · p. 16

Baselines

  • The mitigation comparison includes standard LVLMs, externally supervised fine-tuned models, self-improvement methods, and post-hoc approaches.The post-hoc baselines include VCD, ICD, AVISC, M3ID, OPERA, DeCo, MoD, ConVis, EAZY, ALGA, ONLY, MRGD, and Octopus.Source: §E.2 · p. 16

Evaluation settings

  • The default Scout-and-Project setting uses k = 5, δ = 0.5, αmax = 3.0, and acceptance threshold τ = 30.The method searches top-k branches and refines α up to αmax with a secant-style update.Source: §E.3 · p. 16

Cross-model evaluation

  • A reward model distilled from LLaVA-1.5-7B is applied to LLaVA-Next-7B and InstructBlip-7B for dynamic mitigation evaluation.The cross-model application is motivated in part by InstructBlip-7B’s poor intrinsic self-hallucination detection capacity.Source: §E.4 · p. 16

Related papers

Loading 2604.17982v1…