PaperlayerView exact revision on arXiv

Source-linked AI summary

Evidence-RL: Towards Evidence-intensive Visual Reasoning

Haojie Huang, Xinlei Yu, Chengming Xu, Zhangquan Chen, Cheng Yang, Qingdong He, Yu Yang, Jiangning Zhang, Xiaobin Hu

arXiv:2608.08021v1cs.CVcs.AI

TL;DR

VLM answers can appear correct while relying on language priors or irrelevant context instead of local visual evidence. CED audits counterfactual evidence dependence during GRPO post-training, and EVIDENCE-RL improves performance across four backbones and nine benchmarks without inference-time overhead.

  • Problem

    Existing methods encourage image use but do not test whether answers causally depend on the local evidence supporting them.

  • Method

    CED neutralizes an object-centric Evidence Region, compares support drops with matched non-evidence Regions, and integrates the signal with correctness in GRPO.

  • Results

    Across four backbones and nine benchmarks, EVIDENCE-RL consistently improves performance, especially where shortcut behavior is exposed.

  • Takeaways & Limitations

    The results support rewarding correct VLM answers for depending on relevant visual evidence rather than matching answer distributions alone.

  • Takeaways & Limitations

    On blank-image items lacking visual evidence, CED suppresses prior-following answers and can therefore score lower despite the base model’s correct prior-based response.

Abstract

from arXiv · show

Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context. Existing perception-aware post-training methods encourage image use through global perturbations or attention proxies, but they do not test whether a sampled answer causally depends on the local evidence that supports it. We propose Counterfactual Evidence Disentanglement (CED), a training-time evidence audit for VLM grounding. For each response, CED neutralizes an object-centric Evidence Region and compares the resulting support drop against matched non-evidence Regions. We combine this signal with answer correctness inside GRPO, rewarding correct answers that rely on the evidence path rather than shortcut or nuisance paths. CED uses weak object-level proposals, requires no question-specific evidence annotations, and adds no inference-time overhead. Across nine public benchmarks and four backbones, CED outperforms prior RL-based post-training methods, with targeted analyses verifying its object-centric signal.

1 Introduction

The introduction identifies a gap between coarse visual dependence and causal dependence on the specific evidence supporting an answer. It proposes CED and the EVIDENCE-RL pipeline to audit and reward evidence-specific grounding during post-training.

  • Motivation: Correct-looking VLM answers may rely on language priors, dataset regularities, or scene-level common sense instead of relevant visual evidence.This failure appears in hallucination, counting, spatial, and shortcut-reasoning evaluations.
  • Related work: PAPO and VPPO reintroduce image information during post-training but target coarse visual dependence rather than causal dependence on answer-supporting evidence.PAPO compares original and corrupted images, whereas VPPO uses attention-derived visual dependency.
  • Causal framing: The causal formulation separates language and world priors, irrelevant visual context, and target evidence as distinct paths that can produce the same observed answer.This explains why a correct answer alone does not establish evidence grounding.
  • Method: CED tests whether neutralizing an object-centric Evidence Region reduces response support more than neutralizing matched non-evidence Regions.The method converts the causal graph into a training-time counterfactual evidence audit without requiring question-specific evidence annotations.
  • Results: CED outperforms prior RL-based post-training methods across public benchmarks covering VQA, hallucination detection, counting, spatial reasoning, and perception-heavy visual reasoning.Targeted analyses support evidence-specific dependence, cleaner mean-replacement interventions, and alignment with plausible target evidence rather than arbitrary regions.

2 Related Work

Related work spans the evolution of VLMs into general-purpose multimodal reasoners and the use of reinforcement learning for multimodal post-training. Existing grounded-reward methods compare global image perturbations or audit textual reasoning, motivating localized counterfactual evidence audits using object proposals.

  • VLM reasoning: VLMs have evolved from task-specific perception systems into general-purpose multimodal reasoners supporting visual QA, counting, spatial, diagrammatic, and scientific problem solving.
  • Post-training VLMs: Reinforcement learning has become a central post-training mechanism, extending from PPO-style optimization to verifiable-reward and self-evolution recipes for multimodal reasoning.
  • Grounded Rewards: Image-conditional rewards compare original and perturbed images, but global perturbations do not test whether the decisive region supports the answer.
  • Grounded Rewards: Process-level and self-evolving rewards audit reasoning traces, but this audit remains text-closed.
  • Grounded Rewards: Counterfactual intervention can enter reinforcement learning as a reward signal using off-the-shelf object proposals instead of question-specific region annotations.

3 Method

EVIDENCE-RL trains VLMs with Counterfactual Evidence Disentanglement (CED), which compares answer-conditioned likelihood drops after local feature-space interventions on proposed Evidence Regions and matched non-evidence Regions. The resulting evidence margin complements correctness in GRPO, favoring correct answers that depend on localized visual evidence while keeping correctness dominant.

  • CED pipeline: CED samples a candidate answer, contrasts proposed Evidence Regions with matched non-evidence Regions, intervenes in feature space, computes an evidence margin, and uses it in GRPO.The pipeline is training-time and adds no inference-time procedure in the supplied context.
  • Counterfactual probe: The counterfactual sensitivity s(Ω) is the candidate answer’s log-likelihood drop after replacing visual tokens in region Ω with the mean of neighboring tokens.Mean replacement removes region-specific information while preserving the local representation manifold better than zeroing or noise injection.
  • Contrastive evidence score: A positive evidence margin means the candidate answer depends more on the proposed Evidence Region than on area-matched, spatially separated non-evidence Regions.Non-evidence Regions form a sample-local null for generic masking artifacts or irrelevant visual context.
  • Grounding design: CED scores the sampled response itself and intervenes locally after spatial merging, isolating the target-evidence path while preserving features outside the intervened region.The Evidence Region is a weak spatial proposal resolved from dataset metadata, using object-level COCO annotations in the experiments.
  • GRPO reward: When answers are correct, the reward favors stronger Evidence Region support; when answers are wrong, the bounded evidence term is only a tiebreaker, so correctness remains dominant.The supplied reward parameters are τg=0.20 and εtie=0.10, with |εtiem| ≤ εtie.

4 Experiments

Experiments show that CED provides an evidence-dependence signal beyond correctness matching, improves performance across nine benchmarks and four backbones, and remains sensitive to evidence relevance under intervention and proposal perturbations. Controlled analyses favor Answer-CED and attribute gains to structured local counterfactuals rather than masking magnitude alone.

  • Signal validation: 99.5% of counting-task groups have non-constant rewards, while 90.0% contain same-answer trajectories with different rewards.These results show within-group discrimination beyond answer-string matching; presence tasks have higher zero variance because they are binary yes/no tasks.
  • Benchmark evaluation: Ours achieves the highest average score and largest mean improvement in matched-backbone evaluation, with non-negative gains on all nine benchmarks.The largest gains occur on perception-heavy benchmarks, while MathVista, MMBench, and ScienceQA also remain positive.
  • Controlled comparison: Under matched Qwen3.5-9B conditions, correctness-only and VPPO yield net-negative average deltas through negative transfer, whereas CED enables transferable improvements.The comparison uses identical backbone, 15,314-sample training data, router, 2,000-step schedule, and LoRA configuration.
  • Cross-backbone validation: CED produces positive mean improvement on every tested backbone and non-negative gains across all 36 benchmark–backbone cells.The validation covers four backbones from two model families.
  • Variant analysis: Answer-CED obtains a slightly higher mean gain and stronger general transfer than CoT-CED, despite CoT-CED producing larger training-side rewards.Answer-CED directly applies the counterfactual margin to the final answer, so it remains the default variant.
  • Robustness analysis: The evidence margin falls from m̄=0.268 with COCO proposals to m̄=0.015 with random boxes, while mean replacement outperforms zero and Gaussian-noise replacement.Across keying modes, the intervention AUCs are 0.669 > 0.641 > 0.629; global random masking performs below chance at an above-baseline rate of 0.490.

5 Discussion

CED redirects answers from language-prior shortcuts toward local visual evidence, including in prior-conflict cases, while preserving text-only reasoning performance. Its blank-image behavior also shows that evidence dependence can reject unsupported answers, though this may reduce accuracy when visual evidence is absent.

  • Visual evidence over language priors on FREAK: CED-trained models recover image-grounded answers in prior-conflict FREAK cases where PAPO and LLaVA-v1.6-7B follow plausible language or object priors.Figure 6 reports five such cases, with non-CED models following object priors and CED following local visual evidence.
  • Behavior in the absence of visual evidence: On blank-image ScienceQA items, CED answers “Cannot determine from image” when the frozen base selects a prior-consistent answer and is scored correct.The example asks for a cement truck’s mass despite a blank 448×448 placeholder, illustrating the cost of rejecting unsupported visual answers.
  • No text-only degradation: −0.27 pp mean accuracy change across eight text-only benchmarks indicates no text-only degradation from the Qwen3.5-9B Answer-CED checkpoint.No individual benchmark drops more than 1.5 pp across commonsense, science, and factual reasoning tasks.
  • Evidence-closed text-only self-evolution: An image-conditional signal is structurally necessary to distinguish grounded from shortcut trajectories in VLM reinforcement learning.Text-only self-evolving audits can collapse to language because cheap verifiable targets such as code or arithmetic lack a per-sample-cheap visual analogue.

6 Conclusion

EVIDENCE-RL integrates CED into GRPO to reward VLM responses that causally depend on relevant visual evidence, not answer correctness alone. The framework produces grounded reasoning behavior without additional inference cost and frames faithful evidence dependence as essential to visual reasoning.

  • Framework: EVIDENCE-RL integrates CED’s causal evidence diagnostic into GRPO as an evidence-aware reward for VLM post-training.CED evaluates whether a sampled response depends on the Evidence Region through spatial-region interventions.
  • Framework: The framework produces grounded reasoning behavior with no additional inference cost.Its evidence-aware reward is applied during training through spatial-region interventions.
  • Implication: Post-training should reward correct answers for depending on question-relevant visual evidence rather than matching the answer distribution alone.The conclusion identifies faithful evidence dependence as a capability absent from current VLMs and argues that accuracy without relevant evidence is not visual reasoning.

A Supplementary Experiments and Implementation Details · A.1 Reproducibility Details

The supplementary section specifies the filtered COCO-derived data splits, task-family composition, and GRPO training configuration used for CED. It also situates CED’s causal audit among perception-aware methods and notes potential grounding benefits alongside counterfactual-probing risks.

  • A Supplementary Experiments and Implementation Details: 15,314 samples are loaded for online training after filtering 21,758 COCO-derived samples into train, validation, and probe subsets.The valid splits contain 17,502 train, 2,352 validation, and 1,904 probe samples.
  • A Supplementary Experiments and Implementation Details: The loaded training set comprises 6,562 attribute, 2,188 counting, and 6,564 spatial samples.These are the three reported task families.
  • A Supplementary Experiments and Implementation Details: Training uses balanced_no_replacement sampling, GRPO groups of 32, 2,000 optimization steps, and updates restricted to the model’s last four layers.Full training and reward hyperparameters are provided in Table A.1.
  • A.1 Reproducibility Details: CED is a training-time diagnostic and reward component intended to improve visual grounding in VLMs without directly interacting with end users.The stated application is improving grounding rather than generating user-facing content.
  • A.1 Reproducibility Details: Improved grounding may reduce shortcut-driven errors in safety-relevant visual reasoning tasks.
  • A.1 Reproducibility Details: CED’s causal audit comparison asks whether perception-aware VLM reinforcement-learning methods test causal evidence dependence for candidate answers.The comparison’s causal-audit column records this methodological property.
  • A.1 Reproducibility Details: Perception-R1 reaches region-level granularity through proxy localization and requires additional supervision.
  • A.1 Reproducibility Details: Counterfactual probing can reveal model vulnerabilities, but CED exposes only a scalar evidence margin rather than a detailed attack surface.

A.2 Intervention Implementation Details

CED interventions operate on last-layer vision-encoder outputs after spatial merging, replacing evidence tokens with mean, zero, or Gaussian-noise values. Evidence regions come from prioritized object-level proposals, while matched non-evidence regions use low-IoU sampling and token-subset fallbacks.

  • Replacement strategies: Three replacement strategies—mean, zero, and Gaussian noise—are applied to Evidence Region visual tokens after spatial merging.Mean replacement is the default; it uses neighboring tokens outside T (Ω), while zero replacement sets selected tokens to 0 and Gaussian replacement uses zero-mean Gaussian samples.
  • Spatial-merge-aware token mapping: With 2×2 spatial merging, original coordinates map to merged-token indices using (gy // sh) × wmerged + (gx // sw).The mapping uses merge dimensions (sw, sh) = (2, 2).
  • Non-evidence Region construction: Non-evidence Regions support Chebyshev-distance ring expansion on the merged feature map.
  • Proposal construction pipeline: Positive proposals are object-level COCO bounding boxes, and Evidence Region selection prioritizes target_bbox over argument_bbox, metadata.bbox, and center-box fallback.The full priority order is proposal_bbox > target_bbox > argument_bbox > metadata.bbox > center-box fallback; proposals are not question-specific region annotations.
  • Proposal construction pipeline: Non-evidence Regions use area-matched random boxes with IoU ≤0.05, falling back to matching-size random token subsets when box sampling fails.Quality diagnostics filter proposals with extreme area fractions or insufficient token coverage before training.

A.3 Comparison with Perception-Aware RL Methods

Table A.2 compares CED with representative perception-aware VLM reinforcement-learning methods, emphasizing whether training tests candidate-answer dependence on local visual evidence rather than coarse image dependence or text-only consistency.

  • Comparison with Perception-Aware RL Methods: CED is distinguished by testing candidate-answer dependence on local visual evidence, whereas representative methods use coarse image dependence or text-only consistency.The comparison focuses on the nature of the training signal rather than merely whether images are involved.

A.4 Additional Evidence-Signal Diagnostics

Evidence-region interventions show stronger sensitivity than matched non-evidence regions, especially for counting tasks, while presence-task separation is weaker. Blanking the image reduces the signal below chance, indicating dependence on real image content.

  • A.4 Additional Evidence-Signal Diagnostics: 63.4% of counting samples show higher Evidence Region sensitivity, with mean s = 2.566 versus 0.638 for non-evidence Regions.For presence tasks, the corresponding rate is 55.0%, with mean s = 1.082 versus 0.302.
  • A.4 Additional Evidence-Signal Diagnostics: Counting tasks retain higher relevant-vs-random rates than presence tasks across InternVL3.5-8B and LLaVA-v1.6-Mistral-7B under pixel-level intervention.The validation uses N=50 per task per model.
  • A.4 Additional Evidence-Signal Diagnostics: 0.404 and 0.448 relevant-vs-random rates after image blanking fall below the 0.50 chance level for counting and presence, respectively.Mean delta collapses from +1.928/ + 0.780 to −0.114/ + 0.070.

A.5 Intervention-Type Ablations … A.12 Formalizing Evidence-Closed Self-Evolution

The appendix validates CED’s intervention and reward design, tests robustness across proposals and candidate comparisons, clarifies its evidence interface, and formalizes why evidence auditing adds grounding information beyond closed-loop rewards. It also shows that Answer-CED avoids the reasoning collapse observed with CoT-CED.

  • A.5 Intervention-Type Ablations: Mean replacement achieves the strongest local intervention discrimination, with AUC 0.669 versus 0.641 for zero replacement and 0.629 for Gaussian-noise replacement.Mean replacement preserves the context-consistent baseline more closely, while zero replacement removes the baseline and Gaussian replacement adds sample-specific noise.
  • A.5 Intervention-Type Ablations: Global masking falls below chance at above-baseline rate 0.490, whereas local counterfactual intervention retains discriminative utility with AUC 0.620.
  • A.6 Reward Hyperparameter Sensitivity: Across λ ∈[0.05, 0.20], the non-constant reward rate remains 96–100%, while τg around 0.15–0.30 yields balanced selection and τg = 0.20 is used by default.
  • A.7 Proposal Robustness to Spatial Perturbations: CED evidence margins and evidence-non-evidence gaps decrease as proposal IoU falls, approaching the random floor at the lowest IoU, while tested scale and shift perturbations remain above that floor.The results indicate that moderate detector imprecision is a second-order effect for this diagnostic within the tested ranges.
  • A.8 Shared-Candidate Baselines and Reward Diagnostics: Correctness-only has zero-variance rate 0.949 and non-constant rate 0.051, while Additive and ours reduce zero variance to 0.035 and raise non-constant rewards to 0.965.Ours also improves same-answer-different-reward discrimination to 0.555 versus 0.535 for Additive and reduces the shortcut bias gap to 0.015.
  • A.9 Reward Discrimination and Failure Modes: Evidence-intensive tasks increase non-constant reward rate from approximately 0.05 under Correctness-only to approximately 0.97 with Additive and Routed rewards, while presence tasks remain variance-limited by their binary structure.Same-answer rollouts receive different CED rewards when their visual evidence quality differs.
  • A.10 Scope and Extensions of the Evidence Interface: CED targets greater answer dependence on the evidence path than on shortcut or nuisance paths, instantiated here through object-centric regions without question-specific evidence annotations.The behavioral target is defined on the causal graph and is not restricted to bounding-box interventions.
  • A.11 Answer-CED vs. CoT-CED Details: CoT-CED attains higher training reward but transfers worse, collapsing its mean counting chain to 3.6 tokens versus 49.3 for Answer-CED, which preserves natural multi-sentence reasoning.CoT-CED’s dense per-token reward can incentivize truncation, whereas Answer-CED applies the counterfactual only to the final answer.
Loading 2608.08021v1…