Source-linked AI summary
Evidence-Order Calibration for Selective Visual Reasoning under Progressive Loss of Question-Critical Evidence
Muhamathu Ameer Ali Aacaas Muhamath
TL;DR
The paper asks whether VLM answer reliability preserves the order induced by progressively removing question-critical evidence. Using controlled evidence trajectories and a lightweight post-hoc head, it finds improved evidence-order consistency under masking and unseen blur, without generic confidence superiority.
Problem
The paper examines whether answer reliability respects the evidence-loss order for question-conditioned visual evidence, beyond aggregate confidence changes.
Method
The study uses frozen VLMs, progressive scene-graph-localized critical-region masking, matched non-critical controls, and a post-hoc head with correctness and evidence-order supervision.
Results
Evidence-order supervision reduces EMVR under masking and unseen local blur, while AUROC, Brier, and AURC differences remain inconclusive and native confidence retains stronger selective-risk ranking.
Takeaways & Limitations
Evidence-order consistency and conventional confidence quality are related but distinct reliability objectives.
Takeaways & Limitations
The study uses 176 trajectories from one GQA-derived subset and one primary VLM family, with limited degradation and evidence settings tested.
Abstract
from arXiv · showhide
Vision-language model (VLM) confidence may change in aggregate when visual evidence is degraded while remaining structurally inconsistent within individual examples. We study answer-level reliability along five-step, question-conditioned evidence-loss trajectories. Using a frozen Qwen2.5-VL-3B-Instruct model, we construct 176 accepted GQA-derived trajectories (880 masking conditions) by progressively masking scene-graph-localized question-critical regions. Native sequence confidence has an evidence monotonicity violation rate (EMVR) of 0.436, and 92.0% of trajectories contain at least one adjacent violation. A matched non-critical-region control shows that full critical masking reduces accuracy by 28.2 percentage points, compared with 0.6 points for equally sized non-critical masks; the paired difference is 27.6 points (95% CI [20.0, 34.7]). We train a lightweight post-hoc reliability head on frozen hidden states, sequence confidence, and entropy. Adding evidence-order supervision to binary cross-entropy (BCE) reduces masking EMVR from 0.330 to 0.303 (paired difference -0.027, 95% CI [-0.044, -0.010]). The same mask-trained objective reduces EMVR from 0.449 to 0.402 on held-out question IDs under unseen local Gaussian blur (difference -0.0468, 95% CI [-0.0739, -0.0199]). AUROC, Brier, and AURC differences between the two learned heads are statistically inconclusive, and native confidence remains stronger for selective-risk ranking. The results separate evidence-order consistency from conventional correctness discrimination rather than establishing generic confidence superiority.
1 Introduction
The paper asks whether answer reliability follows the order induced by progressively removing question-critical visual evidence. It evaluates a frozen VLM and trains a post-hoc reliability head with evidence-order supervision.
- Motivation: The study tests whether reliability respects progressively increasing loss of question-critical visual evidence.Five ordered local evidence states are constructed for each image–question pair.
- Approach: The frozen Qwen2.5-VL-3B-Instruct supplies answers and features, while a lightweight post-hoc head learns correctness and adjacent-state evidence ordering.Inputs include sequence log-confidence, predictive entropy, and final-layer hidden representations.
- Findings: Evidence-order supervision lowers EMVR under masking and transfers to unseen local Gaussian blur, while conventional confidence-quality gains remain limited.The comparison targets structural consistency rather than generic confidence superiority.
- Contributions: The protocol combines progressive question-critical trajectories, matched non-critical controls, and a task-specific order-violation diagnostic.These components distinguish evidence-specific effects from ordinary correctness discrimination.
2 Related Work
Prior work studies hallucination, visual dependence, uncertainty, intervention, and ordinal calibration through complementary methods. This paper combines these ingredients to test question-conditioned evidence ordering with a frozen answer model.
- Existing reliability approaches: Existing methods address hallucination, visual influence, uncertainty, abstention, answerability, and region intervention, but target different reliability questions.The paper distinguishes its diagnostic from generic answerability and answer-model decoding.
- Ordinal calibration: Ordinal calibration and frozen-representation confidence probes establish relevant precedents, so ordinal VLM calibration is not presented as novel.The paper positions its contribution in the specific intervention and evaluation protocol.
- Positioning: The study investigates whether progressive question-critical local evidence loss provides useful ordinal supervision for a frozen-model reliability head.It also tests whether the learned ordering survives a change in degradation mechanism.
3 Method
The method creates five-level question-conditioned evidence trajectories, extracts frozen VLM features, and trains a post-hoc reliability head with correctness and ordinal supervision. EMVR measures whether reliability rises as critical evidence is removed.
- Question-conditioned trajectories: Each trajectory applies severity-controlled local corruption to a question-critical region across five evidence states, from unchanged to fully covered.Mask area scales linearly with severity, and the VLM produces answer and feature tuples at each state.
- Post-hoc head: The reliability head combines final-layer representations, mean semantic-token log-confidence, and mean predictive entropy into a linear score.Special and control tokens are excluded from token-level aggregates.
- Training objective: The order-aware model adds an adjacent-severity ordinal hinge loss to the BCE objective while leaving the answer-generating VLM unchanged.The loss uses fixed margin and weighting settings.
- Evaluation: EMVR counts adjacent cases where reliability increases after more evidence is removed; lower values indicate greater evidence-order consistency.The metric is a task-specific diagnostic, alongside AUROC, AUPRC, Brier, and AURC.
- Deployment: Trajectories are required for training and evaluation supervision, but deployment uses only the current image–question–answer state.The deployed head does not repeatedly mask images or generate a complete trajectory.
4 Experimental Setup
Experiments use a frozen Qwen2.5-VL-3B-Instruct model on an accepted GQA-derived subset with question-critical masking and matched controls. Mask-trained heads are also evaluated on held-out question IDs under unseen local Gaussian blur.
- Data and grounding: The dataset contains 176 accepted trajectories and 880 masking conditions from GQA-derived questions focused on color, identity, material, and shape.The accepted set came from 250 candidates after visual quality control.
- Model and generation: The primary Qwen2.5-VL-3B-Instruct model is frozen, deterministic, and paired with lightweight linear heads trained on extracted features.The setup uses a non-quantized checkpoint and short-answer prompting.
- Control design: Matched non-critical controls use equally sized, spatially separated boxes, with six samples excluded when no defensible placement exists.The mean critical-control IoU is 0.00506, but dense annotations prevent treating all controls as pure background.
- Transfer evaluation: Mask-trained models are evaluated on local Gaussian-blur trajectories for held-out question IDs, with no blur examples or blur-specific tuning during training.This tests transfer across degradation mechanisms using grouped cross-validation.
- Uncertainty: Final differences use out-of-fold predictions and paired bootstrap intervals at the whole-question-trajectory level.Severity rows are not resampled independently.
5 Results
Progressive removal of question-critical evidence reveals substantial trajectory-level inconsistency in native confidence, while evidence-order supervision reduces violations and transfers to unseen blur without improving conventional selective-risk ranking.
- 5.1 Aggregate behavior under progressive masking: Aggregate masking behavior was directionally sensible: accuracy and native sequence log-confidence decreased while entropy increased as critical evidence was removed.Accuracy fell from 0.642 on clean images to 0.358 under full critical masking, while log-confidence changed from −0.318 to −0.570 and entropy from 0.826 to 1.382.
- 5.2 Native confidence is trajectory-wise non-monotonic: Native confidence violated the desired direction in 307 of 704 adjacent pairs, producing EMVR 0.436, and 92.0% of trajectories contained at least one violation.Among clean-correct trajectories, EMVR remained 0.383 and 88.5% still contained a violation.
- 5.3 The degradation is evidence-specific: Critical masking reduced accuracy by 0.282 versus 0.006 for matched non-critical masking, yielding a paired accuracy-drop difference of 0.276 with 95% CI [0.200, 0.347].The control supports associating the main degradation with removing question-relevant evidence rather than covering an equal number of pixels.
- 5.4 Evidence-order supervision reduces masking violations: The order-supervised head reduced masking EMVR from 0.330 to 0.303, with a paired difference of −0.026989 and 95% CI [−0.044034, −0.009943].The trajectory violation rate also fell from 0.813 to 0.767.
- 5.4 Evidence-order supervision reduces masking violations: Conventional metric differences between learned heads were statistically inconclusive, while native confidence remained the strongest masking selective-risk score with AURC 0.265 versus 0.315 for BCE and 0.316 for BCE plus order.The proposed objective therefore improved evidence-order consistency without improving generic selective prediction.
- 5.5 Ordering transfers from masking to unseen blur: Under held-out local Gaussian blur, order training reduced EMVR from 0.449 to 0.402, a paired difference of −0.046778 with 95% CI [−0.073864, −0.019886].This statistically supported transfer followed training on masking and testing on an unseen degradation.
6 Analysis and Discussion
The analysis separates evidence-order consistency from correctness discrimination: native confidence ranks correctness best while violating within-trajectory evidence order, and order supervision transfers structurally to blur.
- Divergent reliability objectives: Native confidence achieves the best masking risk ranking despite EMVR of 0.436 and directional violations in 92.0% of trajectories.Its masking AUROC is 0.769, while Figure 6 reports lower AURC than either learned head.
- Interpreting EMVR: A flat non-critical control can still have high EMVR because tiny fluctuations determine adjacent-difference signs without changing aggregate accuracy or confidence.EMVR therefore needs effect-size curves and paired changes in accuracy, confidence, and entropy for interpretation.
- Cross-degradation transfer: Mask-trained order supervision reduces EMVR under held-out local Gaussian blur, whereas conventional uncertainty gains remain statistically unresolved.The transfer experiment changes the intervention while retaining local question-critical geometry; AUROC, Brier, and AURC differences are not statistically resolved.
- Interpretive boundary: The evidence-order prior is not a logical law for every sample because masking can remove distractors, alter segmentation, expose priors, or leave correlated context sufficient.EMVR is therefore most informative in aggregate, together with task degradation and matched controls.
- Qualitative failure modes: Qualitative trajectories include confidence rebounds, correct answers surviving full masking, stable confident errors, and non-monotonic switches under masking or blur.These patterns motivate reporting behavioral trajectories alongside aggregate discrimination metrics.
7 Limitations
The study’s scope is constrained by a small, single-family GQA-derived benchmark, approximate evidence annotations, limited auditing, and unresolved generalization beyond local blur.
- Data and model scope: The benchmark contains 176 accepted trajectories from one GQA-derived subset and one primary VLM family, concentrated in four question categories.This limits how broadly the findings can be generalized across models, datasets, and question types.
- Evidence annotation and audit: Evidence regions use scene-graph object boxes rather than exhaustive human rationales, and visual quality received only limited human spot-checking.The main audit relied on an automated visual-quality procedure.
- Evaluation scope: Strict normalized exact matching treats semantically close answers such as “tan” and “khaki” as different without a semantic-equivalence analysis.This affects interpretation of correctness labels.
- Evaluation scope: Only local Gaussian blur is strongly tested as an unseen degradation, while learned-head AUROC and Brier differences remain statistically unresolved.The method also does not improve AURC over native confidence.
- Generalization boundary: The study calls for testing multiple VLM families, datasets, evidence annotations, and intervention mechanisms before generalizing beyond the controlled setting.Matched non-critical boxes can overlap other annotated objects, and order constraints may fail when correlated context remains.
8 Conclusion
The study finds that evidence-order consistency and conventional confidence quality are distinct reliability objectives: order supervision improves structural consistency under masking and unseen blur, but native confidence remains stronger for selective-risk ranking.
- Conclusion: Order-supervised post-hoc reliability reduces structural inconsistency under masking and transfers statistically to unseen local Gaussian blur.The transferred improvement concerns evidence ordering rather than generic uncertainty quality.
- Conclusion: Native confidence remains stronger for selective-risk ranking, while AUROC and Brier differences between learned objectives are inconclusive.The paper therefore does not claim generic confidence superiority.
A.1 Benchmark audit and taxonomy
The benchmark audit accepted 176 of 250 reviewed candidates, with corrected category counts reported across 880 condition rows.
- Benchmark audit: 176 of 250 candidates were accepted, 71 rejected, and 3 marked unsure by the automated visual-quality procedure.A deterministic ten-example human spot-check was retained but is not full manual auditing.
- Taxonomy: Across 880 condition rows, corrected category counts are 315 color, 400 identity, 60 shape, and 105 material.The taxonomy cleanup changed 205 labels without changing correctness labels.
A.2 Matched non-critical box search
Matched non-critical masks are selected by deterministic placement search under size, image-bound, and overlap constraints, with an objective based on overlap with non-critical scene-graph boxes. Reported intervals preserve trajectory-level pairing through paired bootstrap resampling.
- A.2 Matched non-critical box search: A 41×41 deterministic placement grid searches equal-sized, in-bounds boxes with IoU ≤0.05 relative to each critical region.Ties favor greater center distance and then coordinate order; seed 42 is metadata only because selection is deterministic.
- A.2 Matched non-critical box search: The placement objective sums intersection-area/control-area ratios across non-critical scene-graph boxes.
- A.2 Matched non-critical box search: Confidence intervals use 1,000 trajectory-level paired bootstrap resamples, preserving within-trajectory dependence without assuming condition-level independence.Each resampled unit includes every severity state under both compared methods or interventions, and intervals are percentile intervals.