Visual Dependence Does Not Imply Answerability: A Systematic Analysis of Answerability Signals in Frozen InstructBLIP

• Confidence measures how strongly the model favors its prediction, whereas answerability asks whether the image–question pair contains sufficient evidence for a valid answer.A model can be confident even when required evidence is incomplete, ambiguous, or unavailable.Source: §1 · p. 1
• The authors hypothesize that removing question-relevant visual evidence should change behavior on answerable inputs, while cautioning that visual dependence may coexist with insufficient evidence.Whether visual dependence is a reliable answerability proxy is treated as an empirical question.Source: §1 · p. 1
• The study audits frozen-VLM proxies across controlled perturbations, resampling, and heterogeneous failure modes rather than proposing another answerability model design.It evaluates uncertainty, alignment, grounding, visual dependence, representations, and learned probes on UNK-VQA.Source: §1 · p. 2
• Test AUROC remains approximately 0.50–0.54 across signals and probes; targeted masking changes confidence more than random masking (0.1426 vs. 0.0258), but intervention AUROC is 0.5004.The intervention produces a larger confidence change without reliably distinguishing answerable from unanswerable inputs.Source: §1 · p. 2
• Across generalization and failure-mode analyses, no evaluated signal or probe provides consistently robust discrimination across heterogeneous forms of unanswerability.The audit includes paired bootstrap analysis, multiple seeds, post-freeze cross-validation, and perturbation- and reason-wise evaluation.Source: §1 · p. 2

2.1. Reliable VQA and Selective Prediction

• VQA and selective-prediction work addresses plausible but unsupported answers and prediction reliability, while answerability asks whether available evidence supports a valid answer [8] [9] [22] [33] [36].Strong VQA performance has been associated with pretrained visual and language representations [21].Source: §2.1 · p. 2

2.2. Unanswerable VQA and UNK-VQA

• UNK-VQA provides controlled answerable and unanswerable pairs annotated by causes; this paper tests intervention-based visual dependence as a frozen-model answerability signal [12] [25].The cited concurrent work uses the distinction to construct a benchmark and evaluate prompted abstention, whereas this study tests the signal directly.Source: §2.2 · p. 2

2.3. Answerability Signals and Evidence Sufficiency

• Prior work motivates uncertainty, alignment, grounding, intervention, and representation probes, but their reliability as evidence-sufficiency proxies remains unclear [14, 16, 19, 23, 26, 28].These signals characterize prediction uncertainty, visual-text relations, behavior under evidence changes, or feature-space separability rather than directly establishing evidence sufficiency.Source: §2.3 · p. 3

3.1. UNK-VQA Benchmark

• The stratified UNK-VQA subset contains 10,000 examples across train, validation, and test splits balanced by answerability and perturbation type.The full release contains 99,996 examples; source questions are disjoint across splits, while unanswerability reasons retain their natural imbalance.Source: §3.1 · p. 3, Table 1 · p. 4

3.2. Frozen VLM and Answerability Signals

• Frozen InstructBLIP signals include generation uncertainty, CLIP image–question similarity, grounding scores and features, and Q-Former and question representations.Generation confidence is the geometric mean of greedily selected answer-token probabilities, excluding special tokens.Source: §3.2 · pp. 3–4
• For visual-dependence analysis, question-based target strategies are fixed before detection, and Grounding DINO/SAM masks create evidence-removed images for confidence comparisons.When no valid target mask is obtained, a zero mask is used; a same-size random mask controls for generic sensitivity to image removal.Source: §3.2 · p. 4

3.3. Answerability Probes

• Probes include regularized logistic regression, a seven-feature combination, representation and feature-fusion variants, a nonlinear MLP, and reason-aware multitask learning.The multi-signal probe combines confidence, intervention and answer-consistency features, grounding score, entity coverage, and mask-area ratio.Source: §3.3 · pp. 4–5

3.4. Evaluation Protocol

• Evaluation reports test AUROC and unanswerable recall, bootstrap intervals and paired differences against Confidence, post-freeze five-fold robustness, failure-mode slices, and coverage under selective risk ≤0.3.A global validation threshold is applied unchanged to test and perturbation subsets; selective risk is the unanswerable fraction among retained examples.Source: §3.4 · p. 5

4.1. Overall Answerability Signal Benchmark

• AUROC spans 0.5004–0.5430 across proxies; CLIP Alignment leads at 0.5430, narrowly above Confidence (0.5388), with little evidence of robust standalone discrimination.Grounding max scores 0.5061 and Intervention 0.5004.Source: Table 3 · p. 5, §4.1 · pp. 5–6

4.2. Visual Dependence Does Not Imply Answerability

• Targeted masking drops fixed-answer confidence 0.1426 versus 0.0258 for random masking, yet intervention AUROC is only 0.5004; both answerability classes can be sensitive.Only 54.7% of test examples received a valid target mask; invalid localizations remained in the end-to-end evaluation.Source: §4.2 · p. 6

4.3. Representation and Learned-Probe Analysis

• PCA and fusion add little; Binary MLP overfits (0.7718 train, 0.5147 test AUROC), while reason-aware MTL offers no clear advantage and clusters reasons as R2.Overall, dimensionality reduction, feature fusion, nonlinear capacity, and reason-aware supervision do not resolve weak generalization.Source: §4.3 · p. 6

4.4. Statistical Robustness

• CLIP Alignment’s +0.0042 AUROC over Confidence is unsupported (95% paired-bootstrap CI [−0.0298, 0.0377]); seeds and post-freeze CV remain similarly weak.Across seeds 42, 43, and 44, Binary MLP and reason-aware MTL both obtain test AUROC 0.5153 on average.Source: §4.4 · pp. 6–7

4.5. Selective Answering under Risk–Coverage

• At risk ≤0.3, Confidence retains 2.85% and Entropy 0.20% of examples, while CLIP Alignment has no feasible operating point.The evaluated simple signals cannot retain a substantial subset under this specified low-risk criterion.Source: §4.5 · p. 7

5.1. Perturbation-Wise Performance

• No method dominates across perturbations, and all are near chance on I-2; Intervention’s high fixed-threshold U-Recall coexists with near-chance AUROC.Rejecting many unanswerable cases at one operating point does not imply reliable ranking.Source: §5.1 · p. 7

5.2. Reason-Wise Performance

• Reason-wise AUROC remains weak, with no method consistently reliable across R1–R4; Intervention stays near chance for every reason.Source: §5.2 · p. 7

6. Discussion & Limitations

• Targeted visual removal changes confidence more than same-size random masking, but intervention sensitivity remains near chance for answerability discrimination.The discussion distinguishes sensitivity to available visual evidence from whether that evidence is sufficient to justify an answer.Source: §6 · pp. 7–8
• Across frozen signals and probes, answerability discrimination remains weak, without robust gains from fusion or reason-aware supervision.Nonlinear probing improves training separation without corresponding test gains; the weak pattern persists across bootstrap, seeds, and post-freeze cross-validation.Source: §6 · p. 8
• Relative signal performance changes across perturbations and reasons, indicating that unanswerability is not a uniform state.Table 4 reports perturbation-wise AUROC and U-Recall; Table 5 reports AUROC by unanswerability reason.Source: §6 · p. 8, Table 4 · p. 8, Table 5 · p. 8
• Weak discrimination yields little feasible low-risk coverage in selective answering.The paper suggests assessing whether required evidence is available and sufficient, and abstaining or requesting clarification when it is absent.Source: §6 · p. 8
• The study uses one frozen backbone and a balanced UNK-VQA subset, limiting transfer to architectures with different visual pathways and natural deployment prevalence.The test subset was inspected during development; cross-validation supports the trend, but a fully unseen benchmark would provide stronger evidence.Source: §6 · p. 8

7. Conclusion

• Across uncertainty, alignment, grounding, visual dependence, representations, and learned probes, answerability discrimination remains consistently weak.Increasing probe capacity, combining signals, and adding reason-aware supervision do not provide robust improvements; the pattern persists across robustness and failure-mode analyses.Source: §7 · p. 8
• Removing question-relevant visual evidence changes confidence more than random removal, yet intervention sensitivity remains near chance for answerability discrimination.The result separates model reliance on relevant visual content from evidence sufficiency for a valid answer.Source: §7 · p. 8
• The findings motivate methods that directly assess whether the evidence required by a question is available and sufficient.A model may rely on relevant visual content without having enough evidence to support a valid answer.Source: §7 · p. 8