Source-linked AI summary
Online Self-Calibration Against Hallucination in Vision-Language Models
Minghui Chen, Chenxu Yang, Hengjie Zhu, Dayan Wu, Zheng Lin, Qingyi Si
TL;DR
LVLM hallucinations are aggravated when offline teacher supervision exceeds a student’s perceptual capacity, while discriminative verification offers a more reliable self-supervision signal. OSCAR uses MCTS and dual-granularity rewards to build online preference data and iteratively refine the model, achieving state-of-the-art hallucination-benchmark performance while improving general multimodal capabilities.
Problem
Offline preference supervision from stronger teachers can exceed an LVLM’s perceptual capacity, causing it to learn ungrounded details rather than reliably see them.
Method
OSCAR exploits the Generative-Discriminative Gap by combining MCTS with a Dual-Granularity Reward Mechanism to construct online preference data for iterative DPO refinement.
Results
OSCAR achieves state-of-the-art performance on hallucination benchmarks while improving general multimodal capabilities.
Takeaways & Limitations
Reliable online preference learning can respect intrinsic perceptual boundaries while supporting self-calibration and broader multimodal improvement.
Abstract
from arXiv · showhide
Large Vision-Language Models (LVLMs) often suffer from hallucinations, generating descriptions that include visual details absent from the input image. Recent preference alignment methods typically rely on supervision distilled from stronger models such as GPT. However, this offline paradigm introduces a Supervision-Perception Mismatch: the student model is forced to align with fine-grained details beyond its perceptual capacity, learning to guess rather than to see. To obtain reliable self-supervision for online learning, we identify a Generative-Discriminative Gap within LVLMs, where models exhibit higher accuracy on discriminative verification than open-ended generation. Leveraging this capability, we propose \textbf{O}nline \textbf{S}elf-\textbf{CA}lib\textbf{R}ation (OSCAR), a framework that integrates Monte Carlo Tree Search with a Dual-Granularity Reward Mechanism to construct preference data and iteratively refines the model via Direct Preference Optimization. Extensive experiments demonstrate that OSCAR achieves state-of-the-art performance on hallucination benchmarks while improving general multimodal capabilities.
1 Introduction
LVLMs hallucinate because offline teacher supervision can exceed a student’s perceptual capacity. OSCAR addresses this mismatch through self-verification, MCTS, dual-granularity rewards, and iterative preference learning.
- Motivation: LVLMs frequently generate visual content absent from or inconsistent with the input, limiting deployment in safety-critical domains.Examples include fabricated objects, incorrect spatial relationships, and misdescribed attributes.
- Motivation: Offline preference supervision from stronger teachers can force students to model details beyond their perceptual capacity, encouraging guessing rather than visual grounding.The paper reports that teacher-distilled fine-tuning can increase hallucination rates as more data is added.
- Key Observation: LVLMs show a Generative-Discriminative Gap: discriminative verification is more accurate than unconstrained generation because explicit queries reduce language-prior influence.This gap suggests models contain an underused capacity for self-verification.
- OSCAR: OSCAR combines MCTS with a Dual-Granularity Reward Mechanism to construct online preference data and suppress early tokens that risk downstream hallucinations.The framework uses search-based lookahead and extracts preference information at multiple granularities.
- Results: OSCAR achieves state-of-the-art performance on hallucination benchmarks while improving general multimodal capabilities.The reported outcome spans both hallucination mitigation and broader multimodal performance.
2 Related Work
Prior work mitigates LVLM hallucinations through data, decoding, external correction, and preference alignment. OSCAR differs by using MCTS to construct preference data while accounting for cascading hallucination risks.
- Existing Approaches: Hallucination mitigation has used dataset improvement, decoding manipulation, external post-hoc correction, and preference alignment.These approaches target different stages of generation and correction.
- Self-Improvement: Self-improvement methods reduce reliance on external annotations, but existing vision-language methods typically build preference data through simple sampling or beam search.The paper argues these strategies do not account for hallucinations cascading across generation steps.
- MCTS: MCTS guides generation by simulating future trajectories and backpropagating rewards, with prior applications in reasoning and task planning.The method’s lookahead capability is the relevant connection to hallucination mitigation.
- OSCAR: OSCAR is presented as the first approach to use MCTS for preference-data construction against hallucinations, suppressing locally plausible tokens that risk downstream errors.This directly addresses the cascading limitation identified in prior self-improvement methods.
3 Observations and Motivations
The paper identifies a mismatch between teacher-distilled supervision and student perception, then finds that discriminative self-verification can provide more reliable supervision. This motivates online OSCAR training.
- Supervision-Perception Mismatch: Offline supervision from stronger teachers may exceed a target model’s visual perception, causing it to generate ungrounded details rather than faithfully grounding them.The proposed Supervision-Perception Mismatch explains why stronger teacher data may not transfer reliably.
- Supervision-Perception Mismatch: Qwen3-VL-distilled fine-tuning reduced LLaVA-1.5-7B’s POPE F1 below its 85.87% baseline, with performance worsening as training data increased.The teacher achieved 88.91% POPE F1, but the student consistently underperformed its original baseline.
- Generative-Discriminative Gap: During open-ended generation, LVLMs follow linguistic patterns that can overshadow visual grounding, whereas discriminative verification improves accuracy against visual evidence.The paper illustrates this Generative-Discriminative Gap with image-description and verification prompts.
- Motivation: These observations motivate curating online training data with discriminative capability so description granularity better matches intrinsic perception and factual accuracy improves.The stated motivation is alignment between generated descriptions and perceptual boundaries.
4 Methodology
OSCAR uses MCTS-guided generation and dual-granularity rewards to construct online preference data, then iteratively refines the LVLM with DPO. Its rewards combine sentence-level hallucination verification with gated trajectory-level quality evaluation.
- 4 Methodology: OSCAR constructs online preference data through MCTS-guided generation and iteratively refines the model with DPO.At each iteration, the current policy generates new preference data, and the updated training distribution evolves with the model’s capabilities.
- 4.2 MCTS-Guided Generation: MCTS explores sentence-level generation through selection, expansion, evaluation, and backpropagation to assess long-term trajectory value.PUCT guides selection, candidate sentences are sampled and filtered for diversity, rewards are evaluated, and statistics propagate back to the root.
- 4.2 MCTS-Guided Generation: The Dual-Granularity Reward Mechanism combines node-level process supervision with trajectory-level outcome evaluation to identify early tokens that risk downstream hallucinations.The process reward verifies whether each sentence mentions absent objects, while the outcome reward evaluates complete responses only after a faithfulness gate.
- 4.2 MCTS-Guided Generation: The gated outcome reward checks whether extracted canonical objects are contained in the ground-truth object set before scoring logical consistency, fluency, and redundancy.The quality prompt produces a single score from 0 to 10 for trajectories that pass the faithfulness gate.
- 4.2 MCTS-Guided Generation: The final node value combines process and rollout rewards, and backpropagation elevates early tokens leading to faithful, high-quality completions.The value is defined as value(s_t, a_t) = r_proc(s_t, a_t) + r_out(y_rollout).
- 4.3 Iterative Preference Learning: Preference pairs are extracted globally from the highest- and lowest-value complete paths and locally by comparing selected nodes with poor-performing siblings.Sibling pairs are retained when their Q-value difference exceeds δQ, enabling multiple preference pairs from one search tree.
5 Experiments
OSCAR is evaluated against open-source LVLMs, preference-learning methods, and a beam-search self-rewarding baseline across generative and discriminative hallucination benchmarks. It achieves state-of-the-art hallucination performance, improves multimodal understanding, and benefits from iterative training and its full component set.
- Evaluation setup: OSCAR is compared with open-source LVLMs, hallucination-focused preference methods, and a Self-Rewarding beam-search baseline.Evaluations cover generative benchmarks Object-HalBench, AMBER, and MM-VET, plus discriminative AMBER and POPE tasks.
- Main results: OSCAR achieves state-of-the-art performance across both generative and discriminative hallucination tasks while preserving general multimodal capabilities.On MM-VET, the LLaVA-1.5-7B score increases from 32.5 to 34.6.
- Generative results: CHAIRS falls from 49.0 to 27.6 and CHAIRI from 14.3 to 8.2 on Object-HalBench for LLaVA-1.5-7B.These reductions surpass the reported POVID and SIMA results.
- Iterative improvement: Performance improves progressively through Iter3: LLaVA-1.5-7B CHAIRS decreases from 32.0 to 27.6, while AMBER Hal drops from 22.1 to 17.2.The 13B model shows the same pattern, with CHAIRS declining from 16.4 to 5.4.
- Ablation studies: Adding process reward, gated outcome reward, or MCTS reduces CHAIRS to 32.0 from 44.0, 45.6, and 46.7, respectively.The full model outperforms all partial configurations, indicating complementary contributions from the three components.
- Training-data analysis: With 10k samples, OSCAR-constructed data reduces AMBER CHAIR to 4.5, Hal to 15.4, and Cog to 1.4, unlike distilled or self-generated alternatives.The comparison uses SFT with Qwen3-VL distilled data, LLaVA-generated data, and OSCAR-selected data.
- Case study: Qualitatively, OSCAR produces fewer hallucinated objects and more fluent, less redundant, concise descriptions than LLaVA-1.5.Figure 5 marks hallucinated content in red and correct descriptions in green.
6 Conclusion
The paper identifies offline supervision as a source of hallucination and uses a generative-discriminative gap to develop OSCAR for online preference learning. OSCAR combines MCTS with dual-granularity rewards and improves hallucination benchmarks and general multimodal capabilities.
- Offline preference learning can worsen hallucinations through a Supervision-Perception Mismatch, while discriminative verification provides reliable self-supervision signals.
- OSCAR integrates Monte Carlo Tree Search with a Dual-Granularity Reward Mechanism for online preference learning.
- OSCAR achieves state-of-the-art performance on hallucination benchmarks while improving general multimodal capabilities.
- The work emphasizes respecting models’ intrinsic perceptual boundaries when building reliable vision-language systems.