Source-linked AI summary
The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Models
Jisoo Yang, Jaeho Han, Trung X. Pham, Junyeong Kim
TL;DR
Outcome-level calibration metrics do not reveal whether a VLM’s confidence reflects the specific trajectory behind an answer. The paper tests that relationship through trajectory interventions and proposes trajectory-grounding measures and a benchmark. Confidence is largely trajectory-independent in the evaluated models and calibration methods, and calibration can worsen the disconnect.
Problem
Method
The paper evaluates confidence under content variation, token masking, and hesitation, and proposes TGS-self, TGS-pair, and TGS-Bench for trajectory-level evaluation.
Source: §1 · p. 2, §4.1 · p. 5, §4.2 · p. 6Results
Takeaways & Limitations
Outcome-level calibration and trajectory-level grounding measure different properties, so strong conventional calibration results do not establish trajectory-grounded confidence.
Source: §1 · p. 2, §5.2 · p. 8Evaluation centers on Qwen3-VL and mathematical reasoning with verifiable answers; unrestricted open-ended generation remains an open question, and TGS-Bench relies on external trajectory-generation models.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
A calibrated Vision-Language Model (VLM) can repeatedly self-correct, say "Wait, I should recheck," arrive at the wrong answer, and still report high confidence. We find that this occurs because verbalized confidence is largely trajectory-independent in the VLMs and calibration methods we evaluate. We examine this through three complementary lenses: content variation, token masking, and the model's own hesitation markers. We show that confidence is insufficiently sensitive to what the reasoning trajectory actually contains, and that calibration training can paradoxically worsen this disconnect. Since existing metrics like ECE and AUROC cannot detect this problem, we propose the Trajectory-Grounding Score (TGS) in two complementary forms: TGS-self, which compares confidence with and without access to the model's own trajectory, and TGS-pair, which tests whether the model assigns higher confidence to correct trajectories than to flawed ones along the vision, reasoning, and answer axes. We propose TGS-Bench, a model-agnostic suite spanning 10 benchmarks with controlled good/bad trajectory pairs, and show that conventional calibration rankings diverge from trajectory-grounding rankings, exposing a blind spot in current evaluation practice.
1 Introduction
Outcome-level calibration metrics do not establish whether confidence reflects the specific visual, reasoning, or answer trajectory. Across content variation, token masking, and self-doubt, confidence often responds weakly or inconsistently, and calibration can worsen this disconnect.
1 Introduction
- Verbalized confidence calibration has reduced ECE and improved AUROC, but these aggregate metrics leave open whether confidence reflects the quality of a particular reasoning process.The motivation is that low ECE could result from learning correctness base rates by difficulty tier rather than inspecting instance-specific perception or reasoning.Source: §1 · p. 1
- Conventional metrics can miss trajectory-blind confidence: a model may learn correctness base rates by difficulty tier without checking instance-specific perception or reasoning.The paper describes a calibrated model that hedges and revisits its reasoning yet remains highly confident in a wrong answer.Source: §1 · pp. 1–2, Figure 1 · p. 1
- Across content replacement, trajectory-token masking, and hesitation markers, confidence responds weakly or non-monotonically, with calibration sometimes increasing trajectory-independence.Confidence often changes little after the model expresses self-doubt, with no systematic downward shift.Source: §1 · p. 2
- The paper introduces TGS-self and TGS-pair, and TGS-Bench, a model-agnostic suite spanning 10 benchmarks with controlled good/bad trajectory pairs.TGS-self compares confidence with and without the model’s trajectory; TGS-pair tests confidence across correct and flawed trajectories on vision, reasoning, and answer axes.Source: §1 · p. 2
2 Related Work
Prior work has improved verbalized confidence calibration and questioned whether model self-assessments reflect their processes. This paper extends trajectory-independence analysis to VLMs and reports that it persists across models, with vision showing the strongest independence.
2 Related Work
- Text-only studies report persistent overconfidence and answer-insensitive confidence, while VLM studies address visual degradation and calibration through visual and reasoning scores.The cited approaches include Brierscore alignment, RL with self-reflective rationales, and visual-input interventions.Source: §2 · p. 2
- Prior aggregate calibration metrics assess population-level confidence–accuracy correlation, not whether individual scores reflect a specific reasoning process.The paper frames this missing dimension as trajectory-grounding.Source: §2 · p. 2
- Prior work found verbalized confidence in text-only LLMs nearly answer-independent; this work extends the analysis across vision, reasoning, and answer trajectories in VLMs.It also examines whether models track their own epistemic signals and reports that calibration training can worsen trajectory-independence.Source: §2 · p. 2
- For Qwen3-VL-4B, vision P(JSD ≤0.1) is ≈40% at base and 82–90% with VL-Calibration, so calibration does not resolve trajectory-independence.Figure 2 compares JSD distributions when trajectory components are varied.Source: Figure 2 · p. 3
3 Trajectory-Independence of Verbalized Confidence in VLMs
Across content substitution, token masking, and hesitation tests, verbalized confidence responds weakly or inconsistently to trajectory content and availability. Calibration training can further increase trajectory-independence.
3.1 Preliminary
- The study compares Qwen3-VL-4B-Instruct, a base VLM with holistic confidence, against VL-Calibration-4B, RL-finetuned from it to produce decoupled confidence with intrinsic visual certainty supervision.Its trajectory comprises visual-perception rationale, reasoning chain, and final answer, followed by verbalized confidence on a discrete 0–10 scale.Source: §3.1 · p. 3
3.2 Confidence Distribution Analysis
- P(JSD ≤0.1) is 82–90% for vision, 34–48% for reasoning, and 78–89% for answer in VL-Calibration, with higher rates than the base model.The base model’s corresponding rates are approximately 40%, 18–23%, and 58–72%.Source: §3.2 · p. 4
- Across substitution, masking, and hesitation tests, confidence remained weakly or inconsistently trajectory-sensitive, with calibration increasing near-zero-JSD rates.Complete masking can raise confidence, and neither model systematically lowers confidence after hesitation.Source: §3.3 · p. 4, §3.2 · p. 4, §3.4 · p. 5
3.3 Token Masking Analysis
- At 100% token masking, confidence rises after modest declines through 75% for both models; VL-Calibration’s curve is flatter.At complete masking, the model has no access to trajectory content.Source: §3.3 · p. 4, Figure 3 · p. 4
3.4 Intra-Trajectory Confidence Analysis
- P(|cpre − cpost| ≤.02) is 78.5% for Qwen3-VL and 49.0% for VL-Calibration; neither systematically decreases confidence after hesitation.A representative Qwen3-VL trajectory includes six successive hesitation markers yet ends with a wrong answer and written confidence of 9/10.Source: §3.4 · p. 5, Figure 4 · p. 5
4 Trajectory-Grounding Score
Outcome-level metrics assess calibration without testing whether confidence reflects trajectory content. TGS pairs self-trajectory access tests with comparisons of externally constructed good and bad trajectories.
4.1 TGS-self: Grounding in Own Trajectory
- TGS-self compares confidence with access to the model’s self-generated trajectory against confidence when that trajectory is fully attention-masked.Positive scores indicate reliance on the trajectory; near-zero scores indicate blindness, and negative scores indicate confidence rises under masking.Source: §4.1 · p. 5
4.2 TGS-pair: Discriminating External Trajectories
- TGS-pair tests whether confidence is higher for externally constructed good trajectories than for bad alternatives.TGS-Bench supplies model-agnostic pairs across 10 benchmarks, differing along exactly one axis while holding the others fixed.Source: §4.2 · p. 6
- TGS-pair’s signed scores distinguish correct grounding from trajectory-blindness and anti-grounding.Positive scores mean higher confidence for better trajectories; near-zero scores indicate blindness, while negative scores indicate higher confidence for worse trajectories.Source: §4.2 · p. 6
- TGS-self and TGS-pair address distinct failures: reading one’s own trajectory and discriminating trajectory quality along each axis.Near-zero scores on both provide the strongest evidence of trajectory-independence.Source: §4.2 · p. 6
- VL-Calibration substantially improves AUROC and ECE over the base model, yet trajectory-level scores remain near zero or negative.Table 2 averages TGS-pair across vision, reasoning, and answer axes.Source: Table 2 · p. 7
- The pair construction isolates vision, reasoning, and answer quality by injecting one error along the tested axis.Bad answers use rule-based perturbations; bad vision and reasoning trajectories introduce a false observation or a single logical or arithmetic error.Source: §4.2 · p. 7
5 Experiments
The experiments compare calibration methods on two model scales and 10 benchmarks using outcome-level and trajectory-level metrics. VL-Calibration improves ECE but does not improve trajectory-grounding, which is weakest on the vision axis.
5.1 Experimental Setup
- The evaluation spans six calibration methods, two Qwen3-VL scales, and 10 benchmarks, reporting ECE and AUROC alongside TGS-self and TGS-pair.Training-stage comparisons include Base and VL-Calibration; TGS-pair averages the vision, reasoning, and answer axes.Source: §5.1 · p. 7
5.2 Results
- ECE fell from .432 to .116 at 4B and .402 to .081 at 8B, while TGS-self stayed near zero or turned negative and TGS-pair fell (.059 →.053; .084 →.065).Figure 6 shows lower ECE without higher TGS-self or TGS-pair.Source: Figure 6 · p. 7, §5.2 · p. 8
- Vision is the weakest TGS-pair axis across displayed methods and scales; at 8B, vision and reasoning scores are substantially below answer-axis scores.VL-Calibration has low but positive vision-axis TGS-pair at both scales.Source: Figure 7 · p. 8, §5.2 · p. 8
- VL-Calibration has negative TGS-self on 8 of 10 benchmarks at 4B and 9 of 10 at 8B, while TGS-pair remains positive on every displayed benchmark.Negative TGS-self means masking the model’s own trajectory raises confidence on most benchmarks; ECE does not reveal this pattern.Source: §5.2 · p. 8
- Both Qwen3-VL-8B and InternVL3.5-8B show weak TGS-self and TGS-pair, but InternVL3.5-8B shares a Qwen3-8B language-model backbone.The comparison spans different vision encoders and multimodal architectures, not independent language-model families.Source: §5.2 · p. 8
6 Analysis
The generator sensitivity analysis finds that weak TGS-pair is not driven by the size of the good-trajectory generator.
6.1 Sensitivity to TGS-Bench Generator
- TGS-pair stays below .1 across generator-scale settings, and VL-Calibration beats Base in only one of six generator–scale comparisons.Regenerating the 5.8K-item pool retains 97.6% of items at 32B and 93.4% at 7B; smaller generators can bias the suite toward easier items.Source: §6.1 · p. 8
7 Conclusion
The paper finds verbalized confidence largely independent of trajectory content, availability, and the model’s own self-doubt. It introduces TGS and TGS-Bench to evaluate trajectory-grounding, finding that conventional calibration and trajectory-grounding capture different properties.
7 Conclusion
- Verbalized confidence changes little when trajectory components are replaced, removed, or contradicted by the model’s self-doubt, and calibration training can worsen the disconnect.The finding applies to the VLMs and calibration methods evaluated.Source: §7 · p. 9
- TGS-self compares confidence with and without the model’s own trajectory, while TGS-pair tests confidence discrimination between correct and flawed trajectories.TGS-Bench supplies controlled good/bad trajectory pairs across 10 benchmarks.Source: §7 · p. 9
- Methods that excel on conventional metrics can score near zero or negative on TGS, and vision grounding is consistently the weakest axis.The paper concludes that outcome-level calibration and trajectory-level grounding measure fundamentally different properties.Source: §7 · p. 9
Limitations
The evaluation’s scope is limited by its model coverage, benchmark composition, and reliance on external trajectory generators; the work diagnoses the problem but does not propose a solution.
Limitations
- Evaluation covers Qwen3-VL at 4B and 8B, with a cross-architecture check on InternVL3.5-8B; broader architectures and larger scales remain untested.The authors say extending to additional architectures and larger scales would strengthen generalizability.Source: Limitations · p. 9
- The 10-benchmark suite is predominantly mathematical reasoning with verifiable answers, leaving generalization to unrestricted open-ended generation unresolved.MM-Vet and GQA diagnostic subsets broaden task coverage.Source: Limitations · p. 9
- TGS-Bench relies on external models to construct trajectories, despite reported robustness across generator scales.The work is diagnostic and does not propose a training method to improve trajectory-grounding.Source: Limitations · p. 9
Ethics Statement
The section cautions that standard calibration metrics alone are insufficient for certifying confidence reliability in high-stakes settings, and documents the prompts, trajectory construction, alternatives, and divergence computation used in the analyses.
Ethics Statement
- The authors caution that calibration metrics alone are insufficient for certifying confidence reliability in high-stakes deployment scenarios such as medical imaging or autonomous systems.They state that the benchmarks are public and that TGS-Bench and evaluation code will be released upon acceptance.Source: Ethics Statement · p. 9
A.1.1 Confidence elicitation templates
- Qwen3-VL prompts elicit free-form reasoning, an answer, and a 0–10 confidence score, while VL-Calibration prompts require separate vision and reasoning blocks and confidence scores.The Qwen3-VL template requests visual observations and derivation inside a think block; the VL-Calibration template separates visual analysis, logical deduction, and confidence outputs.Source: §A.1.1 · p. 11
A.1.2 Trajectory construction
- Qwen3-VL trajectories are generated freely and then segmented into vision and reasoning components; manual checks found over 95% agreement on 100 samples.VL-Calibration natively produces tagged vision and reasoning blocks, which are extracted directly without further segmentation.Source: §A.1.2 · p. 12, Trajectory construction for… · p. 12
A.1.3 Alternative trajectory generation
- For each sample, five alternative trajectories per axis are generated with diverse sampling; vision and reasoning alternatives preserve the original answer, while answer alternatives replace it.The vision and reasoning alternatives use different descriptions or reasoning approaches, respectively.Source: §A.1.3 · p. 12
A.1.4 JSD computation
- For each alternative-trajectory pair, the analysis elicits confidence distributions from digit-token logits and computes their Jensen–Shannon divergence.For VL-Calibration, vision- and reasoning-confidence divergences are averaged for the per-sample divergence.Source: §A.1.4 · pp. 12–13
A.1.5 Additional results across models and scales.
Across models and scales, confidence remains trajectory-independent, especially for vision, while masking and validation analyses probe how confidence responds to removed or edited trajectory content.
A.1.5 Additional results across models and scales.
- Trajectory-independence holds across evaluated models and scales: vision shows the strongest independence, while reasoning shows the weakest, though it remains substantial.The same qualitative pattern extends to Verbalize, P(True), SteerConf, ConfTuner, and RLCR at 4B and 8B.Source: §A.1.5 · p. 13
Additional results across models and scales
- At 100% attention-level masking, the paradoxical confidence uptick persists across evaluated models, indicating no consistent reliance on trajectory content when generating confidence.The analysis masks trajectory tokens from downstream confidence-generation attention while retaining them and their positional encodings in the sequence; ratios range from 0% to 100%.Source: Additional results across models and… · p. 13, §A.2 · p. 13
A.3 Intra-Trajectory Confidence Analysis
- Hesitation analysis compares confidence before and after self-doubt or self-correction segments while holding the final answer fixed.Pre- and post-hesitation trajectories are truncated at defined boundaries, and confidence is elicited using an 11-way digit readout.Source: §A.3 · pp. 13–14
B.1 Construction and Independent-Judge Validation
- TGS-Bench contains 18,295 records per axis, with each comparison changing only its target axis and copying the other trajectory components unchanged.Vision and reasoning pairs have .958 character similarity and preserve length; answer perturbations include letter shifts, numeric or structured-numeric changes, and Boolean flips.Source: §B.1 · p. 14
- An image-aware judge identified the intended error side in 75.5%, 72.0%, and 72.8% of vision, reasoning, and answer pairs, respectively.Edited vision and reasoning trajectories were judged natural in 93.0% and 91.8% of cases; confidence-change correlations with edited-passage fluency were small for vision but larger for reasoning.Source: §B.1 · p. 14
B.2 Statistical Robustness of the Main Results
The robustness analysis averages across benchmarks and uses two-stage bootstrap intervals and signed-rank tests; the section also specifies how TGS-self and TGS-pair measure trajectory grounding.
B.2 Statistical Robustness of the Main Results
- Point estimates average across 10 benchmarks, with 95% percentile confidence intervals from 5,000 two-stage bootstrap draws and exact two-sided Wilcoxon tests across benchmark means.The bootstrap resamples benchmarks and then samples within each selected benchmark; reported standard deviations are across benchmark means.Source: §B.2 · p. 14
C.1 Evaluation Metrics
- TGS-self is the signed change in holistic confidence when the model’s designated rationale is visible rather than fully attention-masked, with positive values indicating grounding.The answer remains visible in both conditions.Source: §C.1 · p. 15
- TGS-pair is the signed confidence difference between a good trajectory and its target-axis-perturbed counterpart, computed for vision, reasoning, and answer axes and averaged.The evaluation uses the corrected 11-way confidence readout and averages within benchmarks, then across 10 benchmarks.Source: §C.1 · p. 15
- Outcome-level evaluation uses holistic confidence for Base and the harmonic mean of normalized visual and reasoning confidences for VL-Calibration; ConfTuner ECE uses its native 0–9 distribution.ConfTuner coverage is 73.9% at 4B and 76.1% at 8B.Source: §C.1 · p. 15
C.2 Evaluation Datasets
The evaluation spans mathematical, geometric, logical, and multidisciplinary visual reasoning benchmarks, with additional diagnostic subsets and figures extending the trajectory analyses across methods and scales.
C.2 Evaluation Datasets
- The benchmark set covers mathematical and geometric reasoning, logical reasoning, and multidisciplinary reasoning; MM-Vet and GQA diagnostic subsets are reported separately from the 10-benchmark averages.The additional subsets assess transfer to broader visual reasoning tasks.Source: §C.2 · p. 15
- The mathematical and geometric benchmarks include DynaMath, Geo3K, MathVerse, MATH-Vision, MathVista, and WeMath, covering varied visual-mathematics settings.The supplied descriptions include generated problem variants, geometry problems, diagram-focused questions, competition problems, and hierarchical subproblem decomposition.Source: §C.2 · p. 15
- LogicVista evaluates inductive, deductive, numerical, spatial, and mechanical reasoning, while A-OKVQA, MMK-12, and MMMU-Pro represent multidisciplinary reasoning tasks.MMMU-Pro reduces text-only shortcuts, expands answer choices, and includes vision-only questions.Source: §C.2 · p. 15
Additional trajectory results
- Across models and methods, confidence is most trajectory-independent on the vision axis and somewhat less independent on reasoning, while answer substitutions also leave it substantially insensitive.The 4B and 8B figure captions report that calibration training reduces but does not eliminate trajectory-independence.Source: Figure 9 · p. 16, Figure 10 · p. 17
Masking figures
- Figures 11 and 12 plot expected holistic confidence and its change under increasing designated-rationale masking, averaged over 10 benchmarks at 4B and 8B.Their within-scale common-valid populations are n = 729 and n = 904, respectively.Source: Figure 11 · p. 18, Figure 12 · p. 18
Hesitation figures
- Figures 13 and 14 compare confidence before and after eligible hesitation events, with points below the diagonal indicating higher pre-hesitation confidence.Each point represents one eligible event, and panels report mean confidence change and scored-event counts.Source: Figure 13 · p. 18, Figure 14 · p. 19
Calibration and grounding figures
- Figures 15 and 16 compare locally recomputed ECE with TGS-self and TGS-pair across eight checkpoint–scale pairs, preferring lower ECE and higher grounding scores.The TGS-pair figure reports the raw signed confidence difference; the two figures use panel-wide common-valid populations for their respective measures.Source: Figure 15 · p. 19, Figure 16 · p. 20