Source-linked AI summary
Thinking in Uncertainty: Mitigating Hallucinations in MLRMs with Latent Entropy-Aware Decoding
Zhongxing Xu, Zhonghua Wang, Zhe Qian, Dachuan Shi, Feilong Tang, Ming Hu, Shiyan Su, Xiaocheng Zou, Wei Feng, Dwarikanath Mahapatra, Yifan Peng, Mingquan Lin, Zongyuan Ge
TL;DR
MLRMs remain prone to hallucinations, and transition words often coincide with high-entropy reasoning states associated with hallucination-prone behavior. LEAD adaptively combines discrete and latent semantic representations while injecting visual guidance, and evaluations show consistent hallucination mitigation across general-purpose and scientific benchmarks.
Problem
MLRMs remain prone to hallucinations, while transition words frequently coincide with high-entropy reasoning states associated with hallucination-prone behavior.
Method
LEAD is a lightweight plug-and-play decoder that switches between discrete and probability-weighted latent representations according to entropy and adds visual guidance during high-uncertainty phases.
Results
LEAD consistently mitigates hallucinations across different MLRMs on general-purpose and scientific benchmarks.
Takeaways & Limitations
Maintaining semantic diversity and visual grounding during high-entropy phases supports more stable multimodal reasoning.
Abstract
from arXiv · showhide
Recent advancements in multimodal large reasoning models (MLRMs) have significantly improved performance in visual question answering. However, we observe that transition words (e.g., because, however, and wait) are closely associated with hallucinations and tend to exhibit high-entropy states. We argue that adequate contextual reasoning information can be directly extracted from the token probability distribution. Inspired by superposed representation theory, we propose leveraging latent superposed reasoning to integrate multiple candidate semantics and maintain latent reasoning trajectories. The hypothesis is that reliance on discrete textual inputs may drive the model toward sequential explicit reasoning, underutilizing dense contextual cues during high-entropy reasoning stages. Therefore, we propose constructing rich semantic representations from the token probability distributions to enhance in-context reasoning. With this goal, we present Latent Entropy-Aware Decoding (LEAD), an efficient plug-and-play decoding strategy that leverages semantic context to achieve reliable reasoning. The heart of our method lies in entropy-aware reasoning mode switching. The model employs probability-weighted continuous embeddings under high-entropy states and transitions back to discrete token embeddings as entropy decreases. Moreover, we propose a prior-guided visual anchor injection strategy that encourages the model to focus on visual information. Extensive experiments show that LEAD effectively mitigates hallucinations across various MLRMs on multiple benchmarks.
1. Introduction
MLRMs remain prone to hallucinations, particularly around high-entropy transition words that mark uncertain reasoning stages. LEAD addresses this by preserving semantic diversity during uncertainty and injecting visual grounding.
- Motivation: MLRMs integrate visual understanding with explicit reasoning chains but remain highly prone to hallucinations.Existing mitigation approaches include visual reward designs, data augmentation, and training-free decoding strategies.
- Motivation: Transition words such as because, however, and wait occur frequently in reasoning chains and are associated with hallucinations.These markers structure multimodal reasoning chains and organize semantic relations through linguistic logic.
- Token-level uncertainty: Transition words consistently exhibit higher entropy, indicating high-uncertainty stages with greater competition among potential reasoning paths.The authors hypothesize that discrete textual inputs limit the use of dense contextual cues during these stages.
- Token-level uncertainty: Masking high-entropy tokens causes a significant performance drop, while masking low-entropy tokens causes only minor degradation.This identifies high-entropy tokens as critical informational nodes in the reasoning process.
- Token-level uncertainty: Earlier high-entropy tokens exert stronger directional influence on the final reasoning trajectory than later tokens.The authors therefore emphasize maintaining semantic diversity and visual grounding during high-entropy phases.
- LEAD: LEAD switches between discrete and latent reasoning by using probability-weighted representations during high entropy and discrete embeddings as entropy decreases.The method also uses visual embeddings during high-entropy phases to encourage renewed focus on visual content.
- LEAD: LEAD demonstrates hallucination-mitigating performance across different MLRMs on general and scientific multimodal reasoning benchmarks.The authors present it as a lightweight, plug-and-play decoding approach based on entropy-aware reasoning and visual injection.
2. Related Work
Related work improves multimodal reasoning through post-training while addressing hallucinations through reward optimization and other mitigation strategies.
- Multimodal large reasoning models: Supervised fine-tuning and reinforcement learning remain common approaches for improving multimodal reasoning models.Recent studies also focus on enhancing long-chain reasoning through supervised fine-tuning.
- Multimodal reasoning hallucinations: Multimodal reasoning models remain prone to hallucinations involving contradictions with visual evidence and logical inconsistencies.Existing mitigation methods include optimizing reward functions to improve perception and stabilize multimodal reasoning.
3. Methodology
LEAD adapts multimodal reasoning to token-level uncertainty by switching between discrete and latent decoding, while injecting visual guidance during uncertain phases. Its probability-weighted representations preserve semantic alternatives before returning to discrete decoding as reasoning converges.
- Method overview: LEAD combines multimodal inputs with entropy-aware decoding and visual guidance to adapt reasoning to changing uncertainty.The strategy builds on multimodal generation, dynamically switches reasoning modes, and introduces a pretrained-visual guidance vector during uncertain phases.
- MLRMs Generation: At each step, the MLRM predicts a vocabulary distribution from multimodal inputs and previously generated reasoning tokens before sampling the next token.Reasoning and answering are separated, with generation continuing until the end-of-thinking token before the answer is decoded.
- Latent reasoning decoding: Latent reasoning replaces a sampled token’s one-hot embedding with a probability-weighted embedding that propagates contextual uncertainty across reasoning steps.The weighted embedding is the expectation of token embeddings under the predicted distribution and represents a mixture of possible tokens.
- Entropy-aware mode switching: High entropy indicates competition among potential reasoning paths, whereas low entropy indicates convergence toward a more deterministic trajectory.Entropy measures uncertainty at each generation step and serves as the internal signal for adaptive reasoning behavior.
- Entropy-aware mode switching: LEAD uses probability-weighted embeddings when uncertainty rises and sampled-token embeddings when entropy falls, enabling continuous switching between latent and discrete modes.The mode-switching rule compares current entropy with a reference threshold and uses a persistence window to reduce rapid oscillation.
- Switch regulation: A global switch counter limits total mode transitions and halts further reasoning once the allowed limit is exceeded.This regulation is designed to mitigate overthinking and unnecessary transitions after reasoning has largely converged.
4. Experiments
LEAD is evaluated across multiple multimodal reasoning models, general and domain-specific benchmarks, and ablations of its entropy-aware switching and visual-anchor mechanisms. The experiments report improved hallucination mitigation, reasoning efficiency, visual attention, and sample efficiency.
- Experimental Setup: LEAD is evaluated on general reasoning, hallucination, mathematical, and scientific multimodal benchmarks using several representative MLRMs.The evaluation includes R1-Onevision-7B, Vision-R1-7B, VL-Rethinker-7B, VL-Cogito-7B, and OpenVLThinker-7B.
- Ablation Study: Dynamic entropy thresholding improves MMHalu scores by +4.7% for R1-Onevision and +4.1% for Vision-R1.Large thresholds prevent latent exploration, while small thresholds weaken discrete convergence and increase hallucination risk.
- Ablation Study: Performance improves as the persistence window grows to 128, then declines when the window becomes too large.An infinite window effectively returns the model to standard discrete chain-of-thought reasoning.
- Ablation Study: Visual-anchor performance peaks at injection strength 0.4, while stronger injection slightly reduces performance by diminishing linguistic context.The mechanism encourages visual grounding during high-entropy reasoning steps.
- Ablation Study: LEAD allocates more visual attention to query-relevant regions and produces dispersed high-entropy distributions in latent reasoning versus near-one-hot discrete reasoning.The visualization compares Base, MemVR, and LEAD and shows token probabilities and entropy across reasoning steps.
- Comparisons to State-of-the-Arts: LEAD improves average performance by +3.6% on general reasoning and understanding, +4.7% and +3.8% on MMHalu and Bingo, and shortens reasoning while retaining the highest MathVision accuracy.It also improves average accuracy by +2.0% on mathematics and +3.2% on scientific benchmarks, preserves generated-text quality, and reaches peak Pass@k accuracy with fewer samples than baselines.
5. Conclusion
LEAD adaptively alternates between discrete and latent semantic representations while adding visual guidance during high-uncertainty phases. Across general-purpose and scientific benchmarks, it strengthens reasoning reliability and reduces multimodal hallucinations.
- LEAD adaptively alternates between discrete and latent semantic representations during decoding.
- The framework incorporates visual guidance during high-uncertainty phases to enhance reasoning stability.
- LEAD strengthens reasoning reliability across general-purpose and scientific benchmarks.
- LEAD significantly reduces multimodal hallucinations across the evaluated benchmarks.