PaperlayerView exact revision on arXiv

Source-linked AI summary

DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs

Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, Jintao Chen

arXiv:2609.28570v1cs.AIcs.LG

TL;DR

Hallucination remains a challenge in MLLMs, and RL can provide weak correction signals on hard queries and confident errors. DEEPO pairs semantic-entropy-triggered expert prefixes with advantage-sign-aware R´enyi gradient preconditioning. Both branches improve over GRPO individually, with a significant interaction on VideoMMMU and additive gains elsewhere.

  • Problem

    In MLLM reinforcement learning, unanimously wrong groups on hard queries can collapse group-relative advantage, while confident errors receive vanishing corrective updates.

    Source: §1 · pp. 2–3

  • Method

    DEEPO pairs semantic-entropy-triggered expert prefixes at rollout time with advantage-sign-aware R´enyi gradient preconditioning at optimization time.

    Source: §1 · p. 3

  • Results

    Both branches improve over GRPO individually, and their interaction is significant on long-horizon VideoMMMU (+4.0, 95% CI [1.1, 6.9]) and additive elsewhere.

    Source: §1 · p. 3, §6 · p. 15

  • Takeaways & Limitations

    DEEPO reduces hallucination while preserving accuracy and training stability, with grounding content identified as the active ingredient by the reported controls.

    Source: §6 · p. 15

    The theorem is a local, sample-wise design argument, not a guarantee for global return or the deployed Adam-and-PPO-composed optimizer.

    Source: §4.2 · pp. 8, 10

Abstract

from arXiv · show

Reinforcement learning (RL) is widely used to sharpen reasoning in multimodal large language models (MLLMs), yet its effect on hallucination is uneven. We trace this to two weak points in the \emph{correction chain} from reward to parameter update. At the rollout level, hard queries---those with high semantic entropy---frequently produce unanimously wrong sample groups, collapsing the group-relative advantage to zero exactly where hallucination risk is highest. At the optimization level, confident-but-wrong tokens are gradient-invisible: a categorical policy's expected score-gradient norm vanishes as its distribution sharpens, so the predictions that most need correction receive the weakest updates. We propose Dual-Entropy Enhanced Policy Optimization (DEEPO), a dual-stage enhancement combining signal variance regularization with gradient preconditioning: semantic-entropy-triggered expert prefixes inject grounded continuations on high-uncertainty queries, providing direct supervision and restoring advantage variance, while advantage-sign-aware Renyi preconditioning counteracts logit-level saturation so correction reaches confident errors in the operational confidence regime. Both branches improve over GRPO individually; their interaction is statistically significant on VideoMMMU---the most complex long-horizon task in our evaluation suite (+4.0$, 95\% CI [1.1, 6.9])---and additive elsewhere. DEEPO reduces hallucination while preserving accuracy and training stability.

1 Introduction

Hallucination remains an obstacle in MLLMs, and RL's correction chain can fail both when hard-query groups are unanimously wrong and when confident errors receive vanishing updates. DEEPO pairs rollout-time expert hints with optimization-time gradient preconditioning; its branches interact significantly on VideoMMMU and are additive elsewhere.

1 Introduction

  • Hallucination remains a key obstacle in MLLMs, while RL's effects on it are heterogeneous rather than uniformly harmful.GRPO improves over its base model on POPE and HallusionBench but slightly degrades on VideoHallucer.Source: §1 · pp. 1–2
  • The two DEEPO branches have a positive interaction on all six benchmarks, significant only on VideoMMMU, where hints lift the score from 49.3 to 53.3; elsewhere gains are consistent with additivity.The introduction describes the VideoMMMU interaction as +4.0, 95% CI [1.1, 6.9].Source: §1 · p. 3
  • The correction chain has two weak points: unanimously wrong groups on hard queries collapse group-relative advantage, and confident errors receive vanishing corrective updates.The expected score-gradient norm of a softmax policy vanishes as its distribution sharpens.Source: §1 · pp. 2–3
  • DEEPO pairs semantic-entropy-triggered expert prefixes at rollout time with advantage-sign-aware R´enyi gradient preconditioning at optimization time.The rollout hints provide grounded continuations, while the scaling addresses token-level correction.Source: Figure 1 · p. 2, §1 · p. 3

2 Related Work

Prior hallucination work addresses calibration, decoding, grounding, or analysis, while DEEPO intervenes during on-policy RL. It combines rollout conditioning and token-level gradient scaling rather than changing rewards or relying on offline pairs or full expert traces.

2 Related Work

3 Preliminaries

GRPO uses sequence-level group-relative advantages, which can provide no signal for unanimous binary-reward groups. Semantic entropy measures uncertainty over meaning-level clusters, while collision entropy characterizes token-level sharpness.

3 Preliminaries

  • GRPO samples G responses per query and broadcasts each response's group-relative advantage to every token, so the signal cannot identify which tokens caused failure.With binary rewards, an all-wrong or all-correct group has approximately zero advantage and extracts no gradient.Source: §3 · p. 4
  • Semantic entropy measures meaning-level guessing by clustering N generations into K meaning-equivalent groups and computing entropy over their distribution.High semantic entropy indicates semantic-level guessing and elevated hallucination risk; the default estimate reuses the G current-iteration rollouts.Source: §3 · p. 4
  • At order α = 2, collision entropy H2 characterizes token-level sharpness and governs the expected score-gradient norm used for gradient scaling.The passage defines H2 as −log of the squared token-probability sum.Source: §3 · p. 5

4 Method

DEEPO addresses two breaks in the RL correction chain: collapsed advantages on uncertain queries and vanishing gradients on confident tokens. It combines entropy-triggered expert prefixes with advantage-sign-aware Rényi gradient preconditioning in a unified objective.

4.1 Expert Hint as Signal Repair

  • High semantic entropy triggers expert prefixes that provide grounded supervision and reshape group-relative advantages when unaided rollouts fail unanimously.The prefixes are truncated from model-generated solution trajectories, exclude final-answer tokens, and are used during training, not inference.Source: §4.1 · pp. 5–6
  • On triggered queries with unanimously wrong first-pass groups, hints convert 60% to mixed groups, shifting failed-rollout mean advantage from ≈0 to −0.212 and within-group variance from 0 to 0.6.All-wrong groups rise from 12% in the lowest semantic-entropy decile to 63% in the highest; confidently and consistently wrong low-entropy groups are not triggered.Source: §4.1 · p. 7

4.2 R´enyi Gradient Scaling as Delivery Repair

  • Advantage-sign-aware Rényi weighting increases corrective updates to confident tokens within the operational confidence band, while positive-advantage tokens receive weights proportional to available gradient signal.The damped weight is capped; it does not remove saturation asymptotically, and the theorem is a local, sample-wise result rather than a guarantee about the composed Adam/PPO optimizer.Source: §4.2 · pp. 8–10

4.3 The Unified Objective

  • DEEPO combines GRPO on self-generated continuations with a weighted prefix regularizer, using entropy-triggered prefixes for signal repair and Rényi preconditioning for delivery repair.The unified objective leaves the KL term unscaled; untriggered queries have no prefix loss.Source: §4 · p. 5, §4.3 · p. 10
  • The two stages address distinct bottlenecks: restored advantage variance cannot help if updates cannot reach confident errors, and preconditioning cannot create signal from a collapsed group.The complementarity hypothesis predicts that grounded hints beat shuffled hints, sign-aware weighting beats symmetric alternatives, and interaction is largest on long-horizon tasks.Source: §4.1 · p. 5, §4.2 · p. 9, §4.3 · p. 10

5 Experiments

DEEPO is evaluated on six hallucination and reasoning benchmarks, where it outperforms other evaluated Qwen-based training strategies without trading off reasoning accuracy. Ablations and controls show complementary branch gains, with a statistically significant interaction on VideoMMMU and results consistent with the proposed mechanisms.

5.1 Experiment Setup

  • DEEPO is compared on three hallucination-oriented and three general reasoning benchmarks, with Qwen2.5-VL-7B training strategies and other open- and closed-source models.Fine-tuned models are four-seed means; frozen baselines use bootstrap standard deviations, while closed-source rows are single-run.Source: §5.1 · pp. 10, 13, Table 1 · p. 11

5.2 Performance

  • DEEPO reports 85.8/57.6/69.7 on hallucination evaluations and 86.4/60.8/53.3 on general reasoning, outperforming GRPO, DAPO, and PAPO under the same backbone across six benchmarks.The margin over GRPO’s four-seed mean exceeds one seed standard deviation on five of six benchmarks and is largest on VideoMMMU (+8.6).Source: Table 1 · p. 11, §5.2 · p. 13

5.3 Ablation and Interaction

  • The interaction is positive on all six benchmarks, with only VideoMMMU’s 95% CI excluding zero ([1.1, 6.9]); adding gradient scaling to hints lifts VideoMMMU from 49.3 to 53.3.The hint branch improves five of six benchmarks, most sharply on VideoMMMU (+4.6); the scaling branch improves five and is flat on VideoMMMU alone.Source: Table 2 · p. 12, §5.3 · p. 13

5.4 Controls and Mechanism Evidence

  • Replacing the sign-aware weight with symmetric or sign-only alternatives, or replacing the entropy trigger with a rate-matched random trigger, degrades all three benchmarks.The margin over EMPG’s entropy-only factor is +0.5 on POPE and +1.8 on VideoMMMU; entropy-triggered hints improve over the base method but trail full DEEPO.Source: §5.4 · pp. 13–14
  • All-wrong groups increase from 12% to 63% across semantic-entropy deciles, and hints convert 60% of triggered all-wrong groups into mixed groups.Removing the shared baseline degrades all three tracked benchmarks; matching compute or expert traces also trails DEEPO, with the residual gap concentrated on VideoMMMU (+2.1).Source: §5.4 · pp. 14–15, Table 4 · p. 14

6 Conclusion

DEEPO pairs rollout-time expert prefixes with optimization-time gradient preconditioning to address collapsed advantages and gradient-invisible confident errors. Its branches improve over GRPO individually, with significant interaction on long-horizon VideoMMMU and additive results elsewhere; the controls identify grounding content as active, at +18% per-step cost.

6 Conclusion

  • DEEPO’s branches each improve over GRPO; their interaction is significant on long-horizon VideoMMMU and additive elsewhere, with stable training at +18% per-step cost.The guarantee concerns a local sample-wise surrogate rather than global expected return.Source: §6 · p. 15

A.1 Gradient-Mass Pilot

  • Masking low-entropy gradients preserves stable learning, whereas masking high-entropy gradients destabilizes training because high-entropy tokens carry most update signal.This pilot locates gradient mass; it does not measure a hallucination outcome.Source: Figure 3 · p. 22

A.2 Model Guess Analysis

  • DEEPO has the lowest and least dispersed semantic entropy on HallusionBench and MMStar; interpret this as reduced guessing only alongside preserved accuracy.Questions were converted to cloze-style prompts and sorted by base-model semantic entropy; lower entropy alone could also describe a confidently wrong model.Source: §A.2 · pp. 22–23, Figure 4 · p. 22

A.3 Hyperparameter Sensitivity

  • Threshold changes move results by at most 0.7 points on POPE and 0.5 on VideoMMMU; increasing N improves results up to N=8–12, while N=4 lowers VideoMMMU by 4.0.N is co-varied with GRPO group size G, and the default N=8 is cost-driven.Source: §A.3 · p. 23, Table 5 · p. 23, Table 6 · p. 23

A.4 Causal Controls and Hallucination-Targeted Baselines

  • With the same rollout pipeline and compute, separated advantage normalization drops performance relative to mixed pooling, isolating the loss of the counterfactual penalty on unhinted failures.Mixed pooling jointly normalizes nine samples; the separated variant normalizes unhinted samples internally and scores hinted continuations against a batch-level EMA baseline.Source: §A.4 · p. 24
  • Entropy-triggered hint supervision improves over its corresponding base method but remains below DEEPO, so hint supervision alone does not explain the gains.These hint rows use rollout conditioning only, unlike the main-table no-gradient-scaling ablation.Source: Table 7 · p. 24
  • Shuffled hints show no distinct advantage over GRPO on any of three benchmarks, while correct hints improve results, supporting grounding content rather than conditioning machinery as the active ingredient.Seed-to-seed variations for shuffled hints overlap substantially.Source: §A.4 · p. 24, Table 8 · p. 24
  • Deepo-self recovers most of the VideoMMMU gain over GRPO: 6.1 of 8.6, or about 71%, though it cannot hint groups without a correct rollout.The residual gap is an upper bound, not a clean isolation of collapsed-group coverage, because hint source, quality, and coverage differ.Source: §A.4 · p. 25, Table 9 · p. 25

A.5 Further Controls and Ablations

The further analyses localize DEEPO’s update allocation on failed rollouts and report a second-backbone generalization study. Error tokens are sparse, and the reported effective-step enrichment is smaller than the weighting enrichment.

Error-localization analysis

  • The localization study samples 300 failed rollouts per benchmark, labels hallucinated spans with two annotators and adjudication, and reports bootstrap confidence intervals over rollouts.Agreement was Cohen’s κ = 0.78 for HallusionBench and 0.81 for VideoMMMU; GPT-4V agreed on 92% of tokens in a 50% subsample.Source: §A.5 · pp. 25–26, Table 10 · p. 26
  • Hallucinated tokens make up 6.4% of HallusionBench and 7.1% of VideoMMMU tokens, while receiving 10.7% and 12.1% of weighted update mass, respectively.The effective-step enrichment is 1.15× and 1.25×, below the weight enrichment because error tokens have smaller raw score norms; the shift is relative, not token-orthogonal deletion.Source: §A.5 · p. 26, Table 10 · p. 26

Second backbone

  • Table 11 reports generalization results for the InternVL2.5-8B backbone as four-seed means.The supplied passage identifies the table’s scope but does not provide its outcome values.Source: §A.5 · p. 26, Table 11 · p. 26

B Proof of the Entropy–Gradient Relationship

The proof passage specifies a categorical policy’s logits and score-function gradient, then identifies the referenced equation with the categorical Fisher trace and Simpson’s index complement.

Proof setup

  • The proof defines a policy over an N-token vocabulary, with logits z and token probabilities π_j conditioned on the query and preceding tokens.The sampled-token score-function gradient is g_k = ∇_z log π_k = e_k − π.Source: B Proof of the Entropy–Gradient… · p. 26

Entropy–gradient identity

C Proof of Theorem 4.1 and Remark 4.2

The proof establishes a conservative, damped sample-wise logit step and shows why sign-independent entropy weighting cannot prioritize corrective updates. Its guarantees are local, with saturation compensation limited to the regime where curvature dominates damping.

Proof setup

  • The curvature argument fixes a generated batch and sampled token, then restricts the update to the score-gradient direction and analyzes a damped second-order proximal surrogate.The Fisher matrix and its trace depend on the current policy, while batch advantages are held fixed.Source: C Proof of Theorem 4.1 and Remark 4.2 · pp. 27–28

Damped proximal step

  • The damped proximal weight stays bounded as curvature vanishes and certifies strict sample-wise ascent, with amplification relative to the unweighted rule scaling as (τt + ε0)^−1.The damping constant ε0 caps the weight, and the guarantee degrades gracefully as τt approaches zero.Source: C Proof of Theorem 4.1 and Remark 4.2 · p. 28

Softmax-saturation regime

  • For confident tokens, curvature compensation changes the local correction rate from quadratic to first-order in tail mass when δt ≫ ε0, but it does not eliminate asymptotic saturation.When δt ≪ ε0, the damping floor dominates; the weight reaches its cap and the certified bound reverts to O(δt^2/ε0).Source: C Proof of Theorem 4.1 and Remark 4.2 · pp. 28–29

Sign-symmetry impossibility

  • Any advantage-sign-independent rule assigns equal weights to equal-entropy samples, so it cannot prioritize a negative-advantage sample over a positive-advantage one.A budget-preserving asymmetric reallocation can strictly improve the corrective objective.Source: C Proof of Theorem 4.1 and Remark 4.2 · p. 29

Scope and caveats

  • The theorem is sample-wise and local, not a guarantee of monotonic expected-return improvement, convergence, or targeted correction for the full network.It does not model shared parameters across token positions, PPO clipping, or Adam preconditioning.Source: C Proof of Theorem 4.1 and Remark 4.2 · pp. 29–30

D Relation Analysis of Different Entropies

The section distinguishes answer-level semantic entropy from token-level policy entropy and relates them through Shannon entropy. It uses established entropy bounds to place the signals on a common uncertainty footing, not to claim they are interchangeable.

D Relation Analysis of Different Entropies

  • Semantic entropy Hs measures answer-level dispersion across rollouts, while token-level Rényi-2 entropy H2 characterizes policy sharpness and expected score-gradient norm.Token-level Shannon entropy H1 links the two at their respective levels; the design does not require the signals to be interchangeable or strongly correlated.Source: D Relation Analysis of Different… · pp. 30–31

E Relation Between H1 and H2

F Protocol for the Hs–H1 Alignment Analysis

The protocol estimates answer-level semantic entropy and answer-span token entropy from stochastic samples on HallusionBench and MMStar. It reports significant positive alignment, which strengthens with the hint branch.

Quantities

  • For each question on HallusionBench and MMStar, the protocol draws M samples at temperature 1.2 and computes answer-cluster Hs and mean answer-span-token H1.“With-hint” comparisons concern models trained with or without the hint branch; hints are not used during inference.Source: F Protocol for the Hs–H1 Alignment… · p. 32

Statistics

  • The analysis reports Pearson correlation across questions, Spearman rank correlation as a robustness check, and two-tailed significance tests at α = 0.01.The appendix specifies the protocol for the empirical alignment analysis referenced in Appendix D.Source: F Protocol for the Hs–H1 Alignment… · p. 32

Statistics and result

  • All reported correlations between answer-span H1 and semantic entropy Hs are significant at p < 0.01, and alignment strengthens with the hint branch on both benchmarks.Pearson and Spearman coefficients are reported; the authors frame the result as linked predictive uncertainty, not interchangeable entropy signals.Source: F Protocol for the Hs–H1 Alignment… · p. 32, Table 12 · p. 33

G.1 Implementation Specification

The implementation combines semantic-entropy-triggered expert-prefix hints with token-level gradient scaling in a shared GRPO update. It specifies multimodal preprocessing, training settings, dataset and schedule, and reports that the hint branch accounts for most added wall-clock cost.

G.1 Implementation Specification

  • Answers are normalized and clustered by DeBERTa-MNLI entailment, and expert prefixes are truncated within the thought segment without reaching the answer segment.A semantic-leakage audit discards 3.1% of candidate prefixes on the 20k training traces.Source: §G.1 · p. 33

Hyperparameters.

  • The reported configuration sets the scaling cap to 20, ε0 to 0.05, PPO clipping to 0.2, and the KL coefficient to 0.04.The batch-mean-normalized scaling is applied after group advantage normalization and before clipping.Source: §G.1 · p. 33, Table 13 · p. 35

Data Preprocessing.

  • The unified visual-text tokenizer handles images resized to 4–16,384 tokens and videos sampled at 4–64 frames according to frame rate and duration.Preprocessing maintains aspect ratio and a consistent spatial resolution and frame factor of 2 for video.Source: §G.1 · pp. 33–34, Data Preprocessing · p. 34

Training Framework.

  • Training uses critic-free GRPO with eight responses per query, temperature 1.2, Adam learning rate 5 × 10−7, and a KL penalty of 0.04.The KL penalty constrains the updated policy within the reference-model trust region.Source: Training Framework · p. 34

Dataset and Sampling.

  • The training subset contains 20k samples from the Video-R1 multimodal dataset, with verifiable rewards combining final-answer accuracy and format compliance.Video frames are selected adaptively based on original frame rate and duration.Source: Dataset and Sampling · p. 34

Hardware and Schedule.

  • Training runs for 2,000 GRPO steps on eight NVIDIA A100 80GB GPUs, taking approximately three days of pure training time.Evaluation uses a fixed held-out prompt set every 100 steps; periodic evaluation and checkpointing add time beyond the per-step estimate.Source: Hardware and Schedule · p. 35

Per-Iteration Procedure

  • The implementation estimates semantic entropy from clustered rollout answers, triggers expert thought prefixes for high-entropy queries, and applies token-level scaling within the same GRPO update.Hinted continuations and first-pass unhinted rollouts share a group-relative baseline; the RL term masks the prefix.Source: Per-Iteration Procedure · pp. 35–36

Efficiency Decomposition

  • The hint branch accounts for most of DEEPO’s added wall-clock time, while gradient scaling adds little overhead relative to GRPO.The decomposition separates answer extraction and clustering from conditional continuation generation and the prefix forward pass.Source: Efficiency Decomposition · p. 36, Table 14 · p. 37

Related papers

Loading 2609.28570v1…