PaperlayerView exact revision on arXiv

Source-linked AI summary

The Visual Insensitivity Gap: Diagnosing When Vision-Language Models Fail to Use Visual Evidence

Genpei Zhang

arXiv:2609.00868v1cs.CVcs.CLcs.LG

TL;DR

VLM evaluation often relies on aggregate accuracy, implicitly assuming that models use visual input, but the supplied evidence shows substantial output insensitivity to relevant visual perturbations. The paper formalizes and measures this gap with VSI, traces it to an encoder–LLM disconnect, and finds that its diagnostic value varies by task. VSI is strongest for certain multi-choice reasoning settings and should be used conditionally rather than as a universal abstention signal.

  • Problem

    Aggregate multimodal benchmark accuracy does not distinguish answers that engage with visual evidence from confident answers that ignore it.

  • Method

    The paper measures VSI by blurring the question-relevant image region and quantifying the resulting change in the full next-token distribution, then tests its cross-model structure, mechanism, and diagnostic utility.

  • Results

    The gap appears on 40%–97% of samples, with ρ=+0.40 cross-model VSI-rank correlation and an encoder–LLM gap of 0.66–0.71; AUROCVSI reaches 0.85–0.87 on MMStar reasoning.

  • Takeaways & Limitations

    Visual insensitivity is a sample-intrinsic indicator of vision-ignoring failure and is best used as a conditional component of a signal ensemble.

  • Takeaways & Limitations

    VSI is not universally the best confidence signal: softmax confidence leads on several well-calibrated factuality cells, and its relationship with error can invert on MMVP.

Abstract

from arXiv · show

Vision-language models are evaluated by aggregate accuracy on multimodal benchmarks, a practice that implicitly assumes the model uses its visual input. We show this assumption fails on 40%--97% of samples across six VLMs and three perceptual benchmarks: blurring the question-relevant visual region leaves the next-token distribution nearly unchanged. We name this phenomenon the Visual Insensitivity Gap and quantify it with a per-sample Visual Sensitivity Index (VSI). The gap is a property of samples, not of models: VSI ranks correlate across models (grand-mean Spearman rho=+0.40, permutation p<10^-3), so the same samples are flagged insensitive by VLMs sharing no architectural detail beyond a contrastively pretrained vision tower. The mechanism is concrete: on the insensitive samples, a linear probe on each model's own vision tower distinguishes perturbed from clean images at 0.72--0.79 accuracy, yet the model's argmax token changes on only 2%--11% of the same samples, an encoder--LLM gap above 0.65 on every model. Mapping VSI's diagnostic utility cell by cell surfaces a strong regime (multi-choice reasoning on capable VLMs: AUROC=0.85--0.87) and a weak regime (well-calibrated factuality, where softmax confidence already leads). VSI is not a universal best abstention signal; it is a sample-intrinsic indicator of vision-ignoring failure, best used as a conditional ensemble component.

1 Introduction

The paper identifies a Visual Insensitivity Gap in multimodal evaluation: many VLM predictions remain unchanged when question-relevant visual evidence is removed, despite the vision encoder detecting the perturbation. It formalizes this sample-level phenomenon, explains its encoder–LLM mechanism, and evaluates when VSI is diagnostically useful.

  • Motivation: 40%–97% of perceptual-benchmark samples retain nearly unchanged outputs after question-relevant visual regions are perturbed.This occurs across six recent VLMs spanning three architecture families, even when their vision encoders detect the perturbation.
  • Formalization: The Visual Insensitivity Gap measures the discrepancy between visual representations and the language head’s output, using per-sample VSI to quantify distributional movement after blurring relevant regions.VSI distributions exhibit a heavy left tail separating insensitive samples from sensitive ones.
  • Cross-model structure: ρ=+0.40 grand-mean Spearman correlation shows VSI rankings are shared across models, with permutation p<10^-3 on every tested pair.The result supports visual insensitivity as a sample-intrinsic property rather than a model-specific one.
  • Mechanism: 0.72–0.79 own-tower probe accuracy contrasts with only 2%–11% argmax-token changes, yielding an encoder–LLM gap of 0.66–0.71 on all six models.The vision tower detects the perturbation, but the language head often does not propagate that signal to the output.
  • Diagnostic value: AUROCVSI=0.85 and 0.87 on MMStar math and science show strong diagnostic value, while softmax confidence leads on several POPE-style factuality cells.The paper therefore treats VSI as conditionally useful rather than universally optimal.
  • Study design: The paper tests the phenomenon through cross-model structure, encoder–LLM mechanism analysis, and cell-by-cell comparisons with softmax and verbalized-confidence signals.Its thesis is that visual insensitivity is robust and sample-intrinsic in contemporary VLMs.

2 The Visual Insensitivity Gap

The Visual Insensitivity Gap measures when a VLM’s language output fails to reflect changes in question-relevant visual content, despite a heavy tail of such samples across models and benchmarks. VSI reveals that this pattern is sample-intrinsic and localized: vision towers encode perturbations, but language outputs often do not propagate them.

  • 2 The Visual Insensitivity Gap: VSI measures the KL divergence between next-token distributions before and after blurring the question-relevant image region.The default blur strength is σ=20; localized perturbation tests use of the specific content queried, rather than image reliance in general.
  • 2.2 A heavy left tail of insensitive samples: 40% to 97% of samples have VSI<0.05 across model–benchmark cells, forming a heavy left tail rather than statistically established bimodality.The tail appears on every tested model and benchmark, and its prevalence is not monotone in model scale.
  • 2.2 A heavy left tail of insensitive samples: ≈6% is the insensitive fraction under whole-image perturbation, showing that the localized heavy tail is not a generic any-perturbation artifact.Region-level VSI therefore targets which visual content the model attends to, not merely whether it attends to images.
  • 2.3 Visual insensitivity is sample-intrinsic: ρ=+0.40 is the grand-mean Spearman correlation of region-blur VSI across 15 model pairs, supporting a sample-intrinsic pattern across models.Cross-family agreement remains informative, although it is benchmark-dependent: mean ρ is 0.55 on POPE, 0.34 on MMVP, and 0.32 on HallusionBench.
  • 2.4 Encoder–LLM disconnect: the mechanism: 0.72–0.79 own-tower probe accuracy contrasts with only 2%–11% argmax-token changes on low-VSI samples, yielding an encoder–LLM gap of 0.66–0.71.High-VSI controls show probe accuracy of 0.86–0.91 and increased argmax changes, localizing the disconnect to low-VSI samples rather than the architecture as a whole.
  • 2.4 Encoder–LLM disconnect: the mechanism: The evidence rules out weak perturbations, greedy-decoding concealment, and a uniform fixed visual prior as sufficient explanations for the disconnect.The authors characterize the mechanism as an input–output usage gap consistent with routing failure: representations carry the information, but affected samples do not propagate it to output.

3 Diagnostic Value

VSI has conditional diagnostic value: it identifies confidently wrong, vision-ignoring samples and is strongest for multi-choice reasoning, but its relationship with errors varies by benchmark and model.

  • Failure modes within the low-VSI quintile: Low-VSI samples partition into four correctness–confidence quadrants, including a substantial confidently wrong while ignoring vision group on HallusionBench.This quadrant is especially relevant for abstention because high softmax confidence can conceal vision-ignoring errors.
  • Failure modes within the low-VSI quintile: 10.5% versus 0.0% is the bottom- versus top-quintile error rate on Qwen3-VL POPE, while MMVP reverses the relationship at 15.0% versus 25.0%.Pooled across both benchmarks, the error-rate ratio is 2.33, but the six-model averages are around 1.7 on POPE and 1.0 on MMVP.
  • Failure modes within the low-VSI quintile: 42% of samples on the six-model POPE grid fall below VSI=0.05, where the confidently wrong quadrant is 2.1× denser than in a matched top-VSI population.Tightening to VSI<0.01 adds a 1.4× enrichment on Qwen-family models but reverses on LLaVA-NeXT; VSI<0.05 captures most of the signal at manageable false-positive cost.
  • VSI is strongest on multi-choice reasoning: Typical aggregate AUROCVSI values are 0.45–0.60, but MMStar’s math and science categories provide the strongest signals on capable models.MMStar uses whole-image VSI because diagram- and chart-style layouts make localized region detection unreliable.
  • VSI is strongest on multi-choice reasoning: VSI sharpens on multi-choice reasoning because a small answer set makes distributional movement more likely to flip the argmax, while image-tied questions make ignored evidence answer-determining.These are proposed mechanisms for why sensitivity and answer changes align more tightly in math and science.

4 Conditional Signal Selection

VSI is most useful conditionally: it complements confidence signals where calibration is poor or hallucination drives errors, but no signal dominates across cells. Its rankings are empirically distinct from verbalised confidence and robust to perturbation and threshold choices.

  • VSI-containing ensembles win where calibration is poor or hallucination, while softmax max-probability leads primarily on well-calibrated cells.For example, max-probability reaches AUROC 0.85 versus region-VSI 0.54 for LLaVA-NeXT on POPE.
  • Negative-VSI and negative-verbalised-confidence correlations range from −0.08 to +0.05 across all six models on POPE.This supports treating visual-use sensitivity and elicited confidence as largely independent signals.
  • No signal dominates across cells: max-probability wins on 10/18, VSI-containing hybrids win on 7/18, and VSI alone wins on 1/18.The best signal depends on model–benchmark calibration and failure mode.
  • AUROC reaches 0.636 versus 0.544 on Qwen3-VL POPE and 0.676 versus 0.496 on Qwen2.5-VL-7B MMVP for the region-plus-whole-image VSI hybrid versus max-probability.These gains are +0.09 and +0.18, respectively; the first is significant by paired bootstrap with p<0.01.
  • VSI rankings remain robust across perturbation strengths: AUROCVSI varies by at most 0.05, with cross-strength rank correlation ρ=0.76–0.97.Across insensitive-fraction thresholds, the direction of error differences remains invariant and the audited error-rate ratio changes by at most 1.3×.

5 Related Work

Prior work evaluates multimodal hallucination, reasoning, calibration, and internal representations through complementary benchmarks and diagnostics. This paper connects these strands by studying input-level visual sensitivity alongside internal probing and prediction-time confidence.

  • POPE, MMVP, HallusionBench, and MMStar target object existence, visual primitives, visual and language hallucination, and visually grounded reasoning, respectively.
  • Broader suites such as MMBench and MMMU expand multidisciplinary coverage but report aggregate accuracy rather than per-sample reliability.
  • Selective-generation research commonly uses temperature-scaled softmax max-probability or verbalised confidence as abstention signals.These signals were developed in unimodal settings.
  • Internal probing, compositional probes, attention analysis, and reasoning-time grounding decay examine related encoder–decoder evidence-routing questions from different internal or decoding perspectives.
  • Related VLM studies identify bag-of-words behavior, CLIP blind spots, and vision-transformer token repurposing as diagnostic perspectives on multimodal failures.

6 Discussion

The paper scopes VSI as a sample-intrinsic diagnostic rather than a universal abstention method. Its strongest claims concern replicated cross-model behavior and conditional usefulness, while several benchmark, calibration, and intervention limits remain.

  • VSI is not a universal selective-generation signal: softmax max-probability beats it on 10 of 18 well-calibrated cells.
  • Cross-model agreement is benchmark-conditional, with mean Spearman ρ=0.55 on POPE, 0.34 on MMVP, and 0.32 on HallusionBench.
  • The VSI–error relationship can invert for Qwen3-VL on MMVP perception subtypes, so deployed thresholds require per-cell calibration.The reported p-value is 0.043 and uncorrected.
  • Because the intervention is applied at the input, the measured disconnect is causal at the input–output level but does not localize a structural failure within the language head.
  • The paper reports a sample-intrinsic property replicated across six VLMs and robust to perturbation strength and threshold choice, with AUROCVSI up to 0.87 on MMStar reasoning.

A Implementation Details

The implementation constructs question-relevant perturbations, evaluates probes on controlled image subsets, and handles parsing and confidence-extraction edge cases explicitly. These choices define the measured VSI, probe gap, and selective-generation populations.

  • The six VLMs span three architecture families and four vision towers, making cross-family agreement independent of shared encoder weights.Exact checkpoint identifiers and revision hashes are pinned in released code.
  • Region-blur VSI parses the question’s principal noun phrase, obtains a Grounding-DINO box, refines it with SAM, and dilates the mask by 4 pixels.Whole-image perturbation is used when no box exceeds confidence 0.3; fallback is below 3% on POPE and MMVP and approximately 19% on HallusionBench.
  • The answer-parseable subset excludes unparseable generated answers from accuracy-dependent metrics, while the raw sample pool remains identical across models.Degenerate greedy decoding is the dominant exclusion cause, including 333/500 POPE samples for Qwen2.5-VL-7B.
  • Probes use L2-regularised logistic regression on normalised final-layer image features with grouped 5-fold cross-validation that keeps both image views in one fold.
  • On the shared low-VSI subset, own-tower probe accuracy ranges from 0.722 to 0.789, while matched high-VSI controls reach 0.86–0.91.The same 83 images support both probe accuracies and argmax-change rates, making the encoder–LLM gap a same-subset comparison.
  • A frozen external CLIP probe provides an image-only reference witness, independent of the six VLM vision towers.
  • Verbalised confidence is elicited in a second forward pass and parsed from the first integer, with samples lacking an integer dropped only from that analysis.The per-cell drop rate is 4%–11%.
  • The abstention target is the bottom-VSI, wrong-and-high-confidence quadrant, where high confidence is defined relative to each cell’s median.

B.2 Cross-model VSI agreement detail

VSI rankings show substantial cross-model agreement, including across models from different families, supporting a sample-intrinsic interpretation. Complementary analyses relate this agreement to joint errors, calibration, and capability-conditional diagnostic performance.

  • Joint error structure: Low-low joint VSI quintiles concentrate the highest joint error rates across representative model pairs, with elevated errors also along the diagonal.The pairs span same-family and cross-family comparisons, including disjoint vision towers.
  • Cross-model agreement: ρ=+0.40 is the grand-mean region-blur VSI correlation across 15 model pairs and three perceptual benchmarks.Same-family pairs average ρ=0.51 versus 0.37 for cross-family pairs.
  • Cross-model agreement: The weakest tested pair remains separated from its permutation null by more than three orders of magnitude in p.The tested pairs use a 105-shuffle two-sided permutation null.
  • Signal complementarity: None of the six models exceeds |r|=0.10 between negative VSI and negative verbalised confidence on POPE.Off-diagonal cells still contain additional error mass, motivating hybrid use of the signals.
  • Calibration: Max-probability is best calibrated in most cells, while normalised-VSI and verbalised confidence are noisier, especially at low confidence.Calibration is plotted using empirical accuracy versus predicted confidence in 10 equal-width bins; ECE and Brier scores are reported per cell.
  • Capability-conditional utility: MMStar math and science columns peak on the strongest model, while LLaVA-1.5-7B math reverses direction with AUROCVSI=0.29.The table marks AUROCVSI values of at least 0.65 in bold and values below 0.5 as inverse direction.

C.2 σ-robustness numerical values

VSI rankings are robust to the blur-strength parameter and retain meaningful correspondence across localized and whole-image perturbations. The agreement varies by benchmark, reflecting differences in what the questions test.

  • σ robustness: AUROCVSI varies by at most 0.048 across σ∈{10,20,40} on any audited cell.The widest swing is 0.543 at σ=10 to 0.591 at σ=20 on Qwen3-VL-8B POPE.
  • σ robustness: Per-sample VSI rank correlation across σ pairs remains as high as 0.76 on the most sensitive audited comparison.This variation is smaller than the more-than-0.17 AUROC range across the four audited cells at fixed σ.
  • Perturbation correspondence: ρ=+0.24 is the grand-mean rank correlation between region-blur and whole-image VSI across 18 model–benchmark cells.Per-cell values range from −0.00 to +0.65; MMVP averages +0.36, POPE +0.17, and HallusionBench +0.20.
  • Perturbation correspondence: MMVP shows stronger correspondence because discrete visual primitives are captured by both localized and global blurring.POPE’s lower correlation reflects greater sensitivity to local queried-object content than to global image statistics.

D.1 Reproducibility, hardware, and computational budget

The experiments use fixed hardware, decoding, software, statistical, and data-release procedures to support reproducibility. The full study spans 42 model–benchmark–perturbation cells with fixed random seeds.

  • Hardware and budget: 32 A100-hours is the approximate total compute for all results, using BF16 NVIDIA A100-80GB GPUs.Qwen2.5-VL-32B uses two GPUs; other jobs use one GPU.
  • Hardware and budget: 0.7–1.1 s is the approximate per-sample wall-clock time for 7B-class models at σ=20 region blur.Qwen2.5-VL-32B takes approximately 2.3 s per sample under the same setting.
  • Decoding: Greedy decoding is used throughout, with max new tokens of 8 for POPE, MMVP, and MMStar and 64 for HallusionBench.VSI uses the top-50 next-token distribution after answer-prefix tokens.
  • Software: Pinned scikit-learn 1.8.0 supports bitwise reproduction because adjacent versions shift the reference-probe mean by up to 0.01.The reported fold standard deviation is ±0.04.
  • Statistics: Bootstrap analyses use B=1000, cross-model permutation tests use N=105 shuffles with seed 42, and Table 3 comparisons are exploratory without multiple-testing correction.Per-pair null distributions are released with the code.
  • Data release: 42 cells cover six models and four benchmarks, with perceptual benchmarks evaluated under region-blur and whole-image perturbations and MMStar under whole-image perturbation only.Full per-cell aggregates and per-sample CSVs are released; seeds are fixed at 42.

D.2 Metric definitions

The paper defines AUROCVSI and PRR@80 so visual-sensitivity rankings and rejection performance are directly comparable with confidence-based signals.

  • AUROCVSI: AUROCVSI uses per-sample error as the positive class and −VSI as the ranking score.Values above 0.5 mean lower visual sensitivity predicts higher error risk; max-probability and verbalised confidence use the same convention.
  • PRR@80: PRR@80 is the achieved risk reduction divided by the baseline risk when the worst-ranked 20% of predictions are rejected.The metric evaluates prediction–rejection performance at 80% coverage.

D.3 Limitations beyond the main paper

The paper’s diagnostic signals have important scope and calibration boundaries: VSI can complement confidence measures, but it is not uniformly superior, and several analyses remain input-output characterisations.

  • Low-VSI, high-confidence predictions concentrate confidently wrong, vision-ignoring errors across the evaluated examples.The highlighted cases include object-denial, visual-primitive, and colour-comparison errors.
  • Negative-VSI and negative-verbalised confidence have Pearson correlation |r| < 0.10 on every model, indicating largely independent signals.
  • VSI and verbalised confidence are noisier than max-probability reliability curves because their dynamic range is limited.
  • VSI sensitivity is stable across perturbation strengths: AUROCVSI varies within ±0.05, while per-sample rank correlations reach 0.76–0.97.
Loading 2609.00868v1…