PaperlayerView exact revision on arXiv

Source-linked AI summary

How Murky Before a VLM Knows It Cannot See? Accuracy, Confidence and Abstention on a Physical Visibility Axis

Mochamad Rizal Fauzan, Irgi Surya

arXiv:2610.22981v1cs.CVcs.LG

TL;DR

VLM robustness studies often use ordinal corruption levels, leaving confidence and abstention less examined against physical visibility. This paper measures those signals as visibility declines, finding that confidence can remain high after accuracy collapses, while abstention tracks accuracy when its wording names the reason.

  • Problem

    Robustness evaluations commonly use ordinal corruption levels, and prior work has not graded VLM confidence and explicit abstention along a physical visibility scale.

    Source: §I · p. 1

  • Method

    The study applies physical fog and underwater degradations across eight visibility levels to 1000 COCO scenes and measures accuracy, two confidence signals, and abstention in five small open VLMs.

    Source: §I · pp. 1–2, D. Models and read-outs · p. 3

  • Results

    In fog, accuracy reaches its halfway point at 12.1–15.9 m while confidence stays high into the failure band, with a blind-spot width of at least a factor of 6.2 in visibility.

    Source: §VII · p. 15

  • Takeaways & Limitations

    ImageNet-C fog severities 1–5 correspond to 55–110 m and leave accuracy within five percentage points of clean, whereas the physical axis locates failure below about 12.1–15.9 m under the assumed depth scale.

    Source: §V · p. 12

    Absolute visibility values are not comparable across the fog and underwater arms because they are defined by different channels.

    Source: §VI · p. 13

Abstract

from arXiv · show

Robustness studies of vision-language models (VLMs) grade degradation on nominal severity scales ("level 1-5") that carry no physical units, and they rarely ask whether a model's confidence follows the loss of visibility. We measure accuracy, internal (token-probability) confidence, verbalized confidence and an explicit abstain option on a visibility axis in metres: fog follows the Koschmieder model, an underwater arm scales the absorption spectrum of pure water, both are applied with a monocular metric depth map, and the visibility range V is swept over eight log-spaced levels from 250 m to 1.95 m. On 1000 COCO scenes and five small open VLMs (2-5 B parameters), accuracy on a boxed-object identification task falls from near its clean value to chance within a narrow band (half-way visibility V_acc of 12.1-15.9 m in fog and 20.5-38.6 m under water) while, for the four models that are confident on clean images, the confidence stays high: at V=1.95 m in fog their accuracy is at chance but their mean internal confidence is 0.68-0.84 and their verbalized confidence 0.61-1.00, a blind-spot width W=log10(V_acc/V_aware) of at least 0.79 decades, i.e. a factor of 6.2 in visibility. The remaining model is under-confident throughout and its internal confidence follows accuracy. An explicit abstain option follows accuracy when it names the reason (W from -0.21 to -0.06 in fog) but, for one model, lags with a generic "I don't know" (W up to 0.18). ImageNet-C fog severities 1-5 correspond to only 55-110 m of physical visibility, outside the failure band, and leave the accuracy of the models within 5 percentage points of its clean value. Blank-image and shuffled-image controls, a prior-matched answer set and a 200-image pilot that preceded the study are reported. Findings are descriptive: one draw of 1000 images, depth of assumed scale, and models of 2-5 B parameters.

I. Introduction

The paper asks when VLM accuracy and awareness fail as visibility decreases, addressing a gap in evaluations that use ordinal corruption levels rather than physical units. It places accuracy, confidence, and abstention on a visibility axis and relates this measurement to prior robustness, confidence, and abstention work.

I. Introduction

  • Existing robustness and confidence evaluations generally use ordinal corruption severities, not physical visibility units, and the cited confidence studies do not pair that axis with explicit abstention.The authors note that ordinal levels are not comparable across libraries or directly interpretable for a sensor or site.Source: §I · p. 1
  • The study asks at what visibility range VLMs start answering incorrectly and at what range they start signaling that they cannot see.It calls the difference between these ranges the blind spot.Source: §I · p. 1
  • The protocol measures accuracy, internal and verbalized confidence, and explicit abstention on a metre-based axis using fog and underwater degradation.The physical arms use the Koschmieder fog model and a pure-water absorption spectrum, with monocular metric depth and eight visibility levels from 250 m to 1.95 m.Source: §I · p. 1
  • Across 1000 COCO scenes and five small open VLMs, the reported evidence includes confidence lagging accuracy, wording-sensitive abstention, and ImageNet-C fog severities outside the failure band.The study also reports blank and shuffled image controls, a prior-matched answer set, and diagnostics from a preceding 200-image pilot.Source: §I · p. 2

II. Related Work

  • Prior work covers corrupted-input robustness, confidence calibration, abstention and physical degradation models, while the authors distinguish their study by combining physical visibility with confidence and abstention read-outs.Related work includes ordinal-corruption VLM benchmarks and abstention studies that do not grade visual evidence on a physical scale.Source: §II · pp. 2–3

III. Method

The method degrades depth-informed scenes with physical image-formation models and sweeps visibility in metres. Fog uses a single visibility parameter; the underwater arm applies wavelength-dependent pure-water absorption and has a distinct visibility definition.

III. Method

  • Each scene and estimated metric depth are degraded at swept visibility levels before a frozen VLM answers a boxed-object question and yields four read-outs.The read-outs are answer, internal confidence, verbalized confidence, and abstention.Source: §III · p. 3

A. Image formation and the visibility axis

  • The image-formation model attenuates scene radiance by transmission and adds veiling light, with transmission decreasing exponentially with depth.The operations are applied in linear RGB, then re-encoded to 8-bit sRGB.Source: A. Image formation and the visibility… · p. 3
  • For fog, equal channel attenuation and veiling light A = 0.8 make meteorological visibility V = 3.912/β the single physical strength parameter.V is defined by the distance at which a black target falls to a 2% contrast threshold.Source: A. Image formation and the visibility… · p. 3
  • The underwater arm scales pure-water wavelength-dependent absorption and defines visibility from its least-attenuated, blue-dominated channel.It models absorption without particle scattering and uses monochromatic channels; because its visibility and fog V use different channels, their absolute metres are not comparable.Source: A. Image formation and the visibility… · p. 3
  • Visibility is swept over eight levels from 250 m to 1.95 m, in successive factor-of-two steps, with a clean image as an additional condition.Figure 2 shows one scene at six sweep levels.Source: A. Image formation and the visibility… · p. 3

B. Depth and its assumed scale

The study uses estimated monocular depth to apply the degradation model, so reported visibility in metres depends on the assumed depth scale. The blind-spot width is invariant to a global depth-scale error.

B. Depth and its assumed scale

  • Depth is estimated on each clean image with Depth Anything V2, clipped to 0.5–80 m; the median of image-level median depths is 13.8 m.The metric outdoor model is used because monocular depth estimators are sensitive to weather and lighting corruptions.Source: B. Depth and its assumed scale · p. 3
  • The absolute scale of monocular depth is not verified for COCO, so visibility values in metres are conditional on that estimated scale.Only the product βz = 3.912 z/V enters the image-formation equation.Source: B. Depth and its assumed scale · p. 3
  • A global depth-scale error by factor κ rescales every reported crossing visibility by κ but leaves blind-spot width W unchanged.The invariance follows because the degradation depends on the product of attenuation and depth.Source: B. Depth and its assumed scale · p. 3

C. Data and tasks

The evaluation uses boxed-object questions on selected COCO scenes, frozen small VLMs, and multiple read-outs. It defines visibility-crossing metrics and includes controls for image content and nominal corruption severity.

C. Data and tasks

  • The dataset comprises 1000 COCO validation scenes with one frequency-balanced, boxable vehicle or outdoor-object target per image, marked by a red box after degradation.Targets meet stated area and resolution criteria, and the selected scenes make fog a plausible condition.Source: C. Data and tasks · p. 3
  • Identification asks which of four options names the boxed object, while verification asks whether the box contains a stated true or distractor category.Identification has four-choice chance of 0.25; verification uses true and distractor questions with chance of 0.5.Source: C. Data and tasks · p. 3
  • The evaluation includes blank and shuffled-image controls and ImageNet-C fog severities 1–5 as a nominal-corruption control.The blank image retains the box; the shuffled control pairs a question with another image’s clean pixels.Source: C. Data and tasks · p. 3

D. Models and read-outs

  • Five frozen open VLMs are evaluated with first-token answers, token-probability internal confidence, verbalized confidence, and an explicit abstain choice.A second abstention prompt substitutes “I don’t know”; the prompt templates were fixed before the main run.Source: D. Models and read-outs · p. 3

E. Metrics

  • Accuracy is the fraction correct, while confidence calibration uses ten-bin ECE and confidence-based error detection uses AUROC; results are image-level means with bootstrap intervals.Intervals use percentile bootstrapping over images.Source: E. Metrics · pp. 4–5
  • Crossing visibility is where a signal reaches the midpoint between its clean value and reference floor; W compares accuracy and awareness crossings on a log visibility ratio.A crossing not reached by 1.95 m is censored there and yields a lower-bound W; W is invariant to global depth scale.Source: E. Metrics · p. 5

F. Experimental configuration

The main run used fixed experimental settings.

F. Experimental configuration

G. Pre-registration and relation to the pilot

The study preregistered design changes after a 200-image pilot revealed task-design confounds. In the main visibility sweep, accuracy fell from near-clean levels to the blank-image floor within a narrow visibility band.

G. Pre-registration and relation to the pilot

  • The pilot’s occlusion-ill-posed existence task was replaced with boxed-object identification, and distractors were prior-matched after blank-image accuracy fell below chance.The box was thickened from 3 px to 8 px, and the sweep extended to 1.95 m so confidence signals could cross their mid-way level; a second abstain wording was also collected.Source: G. Pre-registration and relation to… · pp. 5–6
  • Table III lists the fixed settings of the main run.The supplied passage does not specify the table’s individual settings.Source: Table III · p. 6

IV. Results

  • The main run reports t2 unless stated otherwise; t2’s blank-image accuracy is within 0.05 of chance for every model.Intervals are 95% bootstrap intervals over images; t3 and complete sweep tables are reported separately.Source: §IV · p. 6

A. Accuracy collapses within a narrow visibility band (RQ1)

  • Fog Vacc is 12.1–15.9 m and underwater Vbest is 20.5–38.6 m across the five models.In fog, every model’s accuracy stayed near its clean value through 62.5 m, then fell to its blank-image floor between 15.6 and 3.9 m.Source: A. Accuracy collapses within a narrow… · p. 6

B. Confidence lags accuracy (RQ2)

For most tested models, confidence remains high after accuracy reaches its floor, while abstention depends on its wording. Changing the verbalized-confidence prompt changes reported levels but does not make them track accuracy.

B. Confidence lags accuracy (RQ2)

  • At 1.95 m in fog, accuracy is at floor while internal confidence remains 0.68–0.84 for four models; internal-confidence W is at least 0.80 decades for three confident models.Smol-2.2B is the exception: its internal confidence falls with accuracy (W = 0.07 [0.03, 0.10] in fog), but is low even at 62.5 m, where accuracy is 0.86.Source: B. Confidence lags accuracy (RQ2) · p. 6

D. Wording sensitivity of the verbalized confidence

  • At 1.95 m in fog, verbalized confidence stays high across three prompts despite t2 accuracy of 0.22 for Qwen3-VL-2B and 0.26 for InternVL3.5-2B.Prompt wording shifts confidence levels but does not make confidence follow accuracy; SmolVLM2 produced no parseable verbalized confidence.Source: D. Wording sensitivity of the… · pp. 6, 9

E. Nominal severities do not reach the failure band (RQ4)

ImageNet-C fog severities correspond to physical visibility ranges outside the study’s failure band, leaving model accuracy near its clean level.

E. Nominal severities do not reach the failure band (RQ4)

  • ImageNet-C fog severities 1–5 correspond to median gain-matched Veq values of 110, 82, 67, 65, and 55 m.Only 52% of images have a gain-matched Veq that decreases monotonically across severities 1–5.Source: E. Nominal severities do not reach the… · p. 9
  • Across severities 1–5, t2 accuracy spans 0.97–0.98 for Qwen-2B, 0.96–0.97 for Intern-2B, 0.94–0.96 for Gemma-E2B, 0.97–0.99 for Qwen-4B, and 0.85–0.89 for Smol-2.2B.Pooled across models and tasks, nominal-series accuracy has a 2.2 percentage-point dynamic range, versus 55.4 for the physical fog sweep.Source: E. Nominal severities do not reach the… · p. 9

F. Verification task and controls

On verification task t3, accuracy reaches chance because models default to “No,” while internal confidence remains high; the task therefore measures visibility judgment rather than answer calibration.

F. Verification task and controls

  • Blank and shuffled t2 controls are near chance, supporting the conclusion that prior matching prevents solving identification from the box or category prior alone.Blank-image accuracy is 0.24–0.30 and shuffled-image accuracy is 0.25–0.29, against chance 0.25.Source: F. Verification task and controls · p. 9
  • Figure 7 places five ImageNet-C fog severities at median equivalent visibility on the physical fog curve, with interquartile visibility ranges and 95% accuracy intervals.Source: Figure 7 · p. 10
  • At 1.95 m in fog, t3 accuracy is 0.00–0.01 on true-category questions and 0.99–1.00 on false-category questions, while internal confidence remains 0.92–1.00.The models answer “No” on almost every blank-image question, so chance-level accuracy reflects answer bias.Source: Figure 8 · p. 10, F. Verification task and controls · p. 10

G. Complete sweep

  • Tables IX and X report accuracy, internal and verbalized confidence, and abstain rates at every sweep level for each model and degradation arm.These tables provide the data behind Figs. 3 and 8.Source: G. Complete sweep · p. 10

H. Qualitative examples

  • Figure 9 presents qualitative t2 examples for Qwen3-VL-2B, showing answers, internal and verbalized confidence, and abstain decisions across degradation levels.Source: H. Qualitative examples · p. 10

V. Discussion

The discussion finds that confidence can remain poorly aligned with accuracy as visibility falls, while a reason-specific abstain cue tracks accuracy more closely. Physical visibility makes the failure band clearer than nominal fog severity, though deployment conclusions remain bounded by the tested conditions.

Discussion evidence

  • Supporting displays include Qwen3-VL-2B qualitative examples, confidence comparisons across three request wordings, physical visibility equivalents, and blank-image controls.The wording table uses the first 300 images; the controls table defines blank and shuffled images.Source: Figure 9 · p. 11, Table VI · p. 11, Table VII · p. 12

Knowing versus saying

  • Internal confidence retains information at mild fog but barely changes as accuracy collapses; verbalized confidence is nearly flat for three models, with Qwen3-VL-4B falling to 0.61.At the murkiest levels, internal confidence discrimination is gone; the authors caution against reading the Qwen3-VL-4B difference as a scale effect.Source: §V · pp. 10–11

Discussion evidence

  • The reported image summaries include retained contrast gain, medians and interquartile ranges over 1,000 images, with P90/P10 indicating spread across images.Source: §V · p. 12

An abstain option is a cue, not a capability

  • The abstain rate follows accuracy with the main wording (W −0.21 to −0.06 in fog) but lags with “I don’t know” (W up to 0.18).The measured abstention awareness depends on the option naming the right reason.Source: §V · p. 12

What a physical axis adds

  • ImageNet-C fog severities 1–5 correspond to 55–110 m visibility and leave accuracy within 5 percentage points of clean, whereas failure occurs around 12.1–15.9 m.Equivalent visibility varies by image, with a factor of 6.2–8.4 spread.Source: §V · p. 12, Table VII · p. 12

Fog versus underwater

  • Under water, accuracy reaches chance at a larger best-channel visibility, Vacc 20.5–38.6 m; internal-confidence ECE is 0.56 and 0.44, matching fog values.The authors caution that lower-bound blind-spot widths should not be compared across arms because the sweep ends at 1.95 m.Source: §V · p. 12

Practical reading

  • For small VLMs in fog or turbid water, the authors advise against relying on verbalized confidence, uncalibrated internal confidence, or a generic abstain option.A separate image-quality or visibility estimate that triggers abstention is proposed, but testing it is outside this study.Source: §V · p. 12

VI. Limitations •

The study’s conclusions are bounded by its image and model sample, estimated depth scale, and synthetic degradation assumptions. Task design, wording, and the lack of a human baseline further limit interpretation.

Scope

  • One draw of 1000 images, five 2–5 B-parameter models, and one prompt family bound the findings; two models use only image subsets.Other model families, larger models, repeated image samples, and prompts may behave differently.Source: §VI · p. 12

Depth

  • Reported visibility in metres depends on an unverified monocular depth scale, although a global depth-scale error leaves W unchanged.Non-global depth errors, including those for sky or reflective surfaces, are not modelled.Source: §VI · p. 12

Synthetic physics

  • Synthetic fog uses one homogeneous medium, while the underwater arm models clear-ocean pure-water absorption without particles or scattering.It omits forward scattering and blur and does not represent coastal or turbid water.Source: §VI · pp. 12–13

Cross-arm comparison

  • Absolute visibility metres and lower bounds on W are not comparable across fog and underwater arms because their defining channels differ.Source: §VI · p. 13

Task design

  • The red box may be hard for low-resolution models to find, distractor matching is approximate, and both tasks are limited to COCO object categories.A frequency guesser scores above chance, and the balanced verification task retains model-specific answer bias.Source: §VI · pp. 13–14

Wording

  • Abstention depends on wording, but only two wordings were collected at stage 1 and verbalized-confidence wording was varied on a 300-image subset.Source: §VI · p. 14

Nominal equivalence

  • The nominal-equivalence proxy uses retained contrast, and its preregistered statistic failed.Source: §VI · p. 14

Evaluation scope

  • There is no human baseline, so the visibility at which people fail on these tasks is unknown; the manuscript is also a preprint not peer reviewed.Source: §VI · p. 15

VII. Conclusion

On a physical visibility axis, small open VLMs show a narrow accuracy failure band and confidence that can remain high well into it. The conclusion also identifies limits, follow-up directions, and data and code availability.

VII. Conclusion

  • 12.1–15.9 m is the half-way accuracy visibility in fog on identification, while confidence stays high into the failure band with a blind-spot width of at least 6.2×.An explicit abstain option follows accuracy only when it names the reason; ImageNet-C fog severities 1–5 lie on the plateau, equivalent to 55–110 m.Source: §VII · p. 15
  • Repeated image draws, more and larger models, measured or sensor-based depth, scattering and turbid-water spectra, a human baseline, and a visibility-triggered abstention estimator are identified as next steps.Source: §VII · p. 15
  • The frozen configuration, per-question results, analysis scripts, and figure and table generators are available from the authors on request; a public repository is planned for a later version.The datasets and models used are publicly available.Source: Data and Code Availability · p. 15

Related papers

Loading 2610.22981v1…