I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models

1 INTRODUCTION

• The protocol compares stated modality dependence with realized behavior under matched interventions, asking whether single-modality claims hold and whether restoring missing evidence changes predictions.Missing inputs may arise from sensor failure, occlusion, privacy constraints, transmission errors, or acquisition costs.Source: §1 · pp. 1–2
• The evaluation covers retrospective attribution and prospective counterfactual self-knowledge across four tasks with complementary, isomorphic, and real-world multi-view evidence.The tasks are mm-IMDb, IsoBench function parity and graph max-flow, and Golden View driving QA.Source: §1 · p. 2
• The study evaluates ten VLMs spanning open-weight and proprietary models.The introduction presents the protocol and its evaluation against interventional ground truth as a contribution.Source: §1 · p. 2
• Restoring a missing modality changes answers on a median of up to 70.1% of inputs per task, yet models rarely anticipate these changes.The introduction also reports that insufficiency claims are often precise but severely under-produced.Source: §1 · pp. 2–3
• Retrospective attributions over-credit sufficiency: models claim one input would suffice more often than it does on complementary data, and overstate interchangeability on isomorphic data.These are the two modality-relation regimes highlighted in the introduction.Source: §1 · p. 3

Learning under missing modalities

• Missing-modality learning primarily evaluates reconstruction quality or downstream performance under incomplete inputs, rather than whether models can report changed evidence and its implications for their predictions.The paper takes up this complementary question.Source: §2 · p. 3

Modality reliance and its observability

• Modality-reliance studies find that fluent explanations or reasoning traces do not by themselves establish actual modality dependence.Prior findings include misleading textual influence, variable visibility in reasoning traces, and descriptions of images never supplied.Source: §2 · p. 3

Faithfulness of self-explanations

• Prior self-explanation work reports mixed faithfulness evidence, while this paper tests prospective and retrospective modality claims by executing interventions.The cited approaches infer reliance from feature attribution or reasoning traces, or test adherence to elicited decision rules.Source: §2 · p. 3

3 MEASURING MODALITY SELF-EXPLANATION FAITHFULNESS

• Behavioral faithfulness is agreement between a model’s stated account of modality dependence and its behavior under a matched intervention.The protocol grades a claimed answer against the answer the same model produces under the corresponding input configuration.Source: §3 · p. 3
• The criterion is behavioral and makes no claim about internal computation; output-level agreement may establish self-consistency rather than faithfulness proper.The stated rationale is that both the claim and its behavioral grading are observable.Source: §3 · p. 3

3.1 SETUP AND NOTATION

• The setup compares answers in a shared answer space, using separate plain and assessment prompts and one greedy generation per arm.The assessment prompt elicits the answer and its self-report together; deterministic decoding makes differences across arms attributable to instances rather than the decoder.Source: §3.1 · p. 4

3.2 RETROSPECTIVE ATTRIBUTION

• Retrospective attribution compares which single-input answers reproduce the joint answer, applying the same rule to model claims and behavior.The reported attribution distributions are marginals, so gaps indicate over-produced or withheld attributions but do not establish per-item agreement.Source: §3.2 · p. 4
• For evidence-grounded attribution, GoldenView compares stated and behavioral attribution to the annotated input containing the answer-supporting evidence.The paired accuracy comparison tests whether that annotated input has behavioral value, while the attribution comparison assesses how each account tracks it.Source: §3.2 · p. 4

3.3 PROSPECTIVE COUNTERFACTUAL SELF-KNOWLEDGE

• Prospective tests compare a claimed answer after restoring a missing input with the executed joint answer, and compare stated sufficiency with executed change.The section defines predicted and executed change rates and treats a sufficiency claim followed by a changed answer as a silent sufficiency failure.Source: §3.3 · p. 5

3.4 ELICITING THE SELF-REPORT

• The instrument elicits a named counterfactual answer and a forced choice between substantive sufficiency claims, rather than relying on yes/no requests for more input.Alternative wording tests include removing the cue to repeat the current answer; on mm-IMDb, median predicted change rises from 10.3% to 32.0%.Source: §3.4 · p. 5

4 EXPERIMENTAL SETUP

• The tasks operationalize mixed evidence with mm-IMDb, redundant representations with IsoBench, and unique evidence with GoldenView.The study subsamples 300 mm-IMDb examples and uses GoldenView’s 55 questions as a targeted probe rather than a headline number.Source: §4 · p. 5
• The study evaluates ten instruction-tuned VLMs: two proprietary models and eight open-weight models spanning Qwen3.5 and Gemma 4, with sizes from 2B to 35B parameters.The open-weight models include small self-served models and larger hosted-API models, with dense and mixture-of-experts variants in each family.Source: §4 · p. 5
• All models run in non-thinking mode because thinking traces can enter repetition loops and exhaust generation budgets; paired reruns on two models reproduce both axes.The paired rerun is limited to the two models whose thinking arms retain their rows.Source: §4 · p. 6
• The design yields 80 model-task-condition cells, scoring only groups with all required arms present and relevant fields parsed.Attrition is uneven across the grid, and task performance is reported for reference rather than used in the answer-comparison indicators.Source: §4 · p. 6

• On mm-IMDb, executed BOTH attributions were 38% versus 14.3% stated, while claimed text-only sufficiency was 48.8% versus 25.5% executed.On IsoBench, EITHER was claimed on 92.6% of parity and 88.3% of max-flow items, but executed at 80.9% and 54.6%, respectively; GoldenView EITHER was claimed on 41.3% versus 59.3% executed.Source: §5 · p. 6, Table 2 · p. 7
• 78 of 80 model-condition cells had more executed than predicted answer changes after modality restoration; task-level median executed change reached 70.1%, while predicted rates never exceeded 9.6%.The gap persisted under matched-template checks and alternative genre-set scoring; neither model scale, family, sparsity, nor prompt rewording closed it.Source: §5 · p. 6, Figure 2 · p. 7, Table 3 · p. 7
• Insufficiency claims were rarely made despite changed answers: median recall was 13.9% on mm-IMDb, 58.6% on GoldenView, and 0% on both IsoBench tasks.Only 11.6% of mm-IMDb and 16.5% of GoldenView inputs were claimed insufficient; the remaining 86.1% and 41.4%, respectively, were labeled sufficient despite silent failures.Source: §5 · pp. 7–8, Table 4 · p. 8, Figure 3 · p. 8
• On GoldenView, the annotated evidence set yielded a median 27.9 pp accuracy advantage, and stated attribution aligned with it more than answer reproduction for seven of ten models.Among cases where one set was identified, stated attribution selected the annotated set in 97.3% of cases versus 86.9% for answer reproduction; this does not establish per-item dependence.Source: §5 · p. 8, Figure 4 · p. 9
• The self-report prompt changes the measured model: across open-weight models it costs a median 3.6 pp of task performance, so claims concern prompted rather than plain models.Whole-modality withholding is tested, but partial corruption remains untested; six open-weight and both proprietary models were also untested with thinking enabled.Source: §5 · pp. 8–9

6 CONCLUSION

• Across eight open-weight and two proprietary VLMs and four tasks, both retrospective and prospective self-explanations overstate sufficiency; predicted change is lower than executed change in 78 of 80 cells.Most inputs whose answers change after restoration are not flagged as insufficient.Source: §6 · p. 9
• The protocol grades stated modality dependence against executed behavior on the same instance and in the same answer space.Source: §6 · p. 9
• The authors report that generative AI assisted with implementation, scoring, result interpretation, and readability edits, with authors reviewing and checking the assisted work.Source: AI USE STATEMENT · p. 9
• The study uses published benchmarks under their released terms, including mm-IMDb, IsoBench validation, and GoldenView questions over nuScenes camera images.Source: ETHICS STATEMENT · pp. 9–10
• On a driving task, partial camera views can be claimed sufficient even when withheld views change the answer; self-reported sufficiency should not replace behavioral checks in safety-relevant settings.The authors make no claim that any evaluated model is fit for such use.Source: ETHICS STATEMENT · p. 10

A.1 TASK PERFORMANCE BY MODEL AND ARM

• Task-performance entries score all attempted rows, so truncation or parse failures can lower a cell; the assessment arm of Gemma 4 E2B on parity text + image retains 311 of 384 rows.Excluding lost rows changes its 97.9 to 83.7 drop to 93.6; all runs use thinking-disabled mode.Source: §A.1 · p. 13
• At film grain on mm-IMDb, Qwen3.5 35B-A3B recovers 93% of its joint-arm score from the title alone, while Gemma 4 E2B recovers 60%.The passage attributes the high title-only floor to genre prediction rewarding parametric knowledge.Source: §A.1 · p. 13
• On IsoBench, text-only exceeds image-only in 19 of 20 plain cells, by median margins of 13.7 points on parity and 31.6 on max-flow.The exception is Qwen3.5 9B on max-flow: 26.6 for text versus 55.5 for image.Source: §A.1 · p. 13, Table 6 · p. 14

A.2 PER-MODEL PREDICTED AND EXECUTED CHANGE RATES

• No cell in Figure 5 predicts more change than it executes across ten models and four tasks spanning three modality regimes.The per-cell pattern shows the reported effect is not an artifact of task medians.Source: Figure 5 · p. 15, §A.2 · p. 15

A.3 EFFECT OF THE ASSESSMENT PROMPT

• The assessment prompt changes answers even when accuracy is stable: on mm-IMDb it moves parsed answers on a median 59.8% of inputs despite costing at most 8.9 pp macro-F1 in any cell.On the GoldenView joint arm, accuracy changes by +1.8 pp while 11.8% of answers move.Source: §A.3 · p. 15
• Under the plain-matched wiring, task medians remain 51.8, 6.7, 26.0 and 25.3 pp above predicted rates; executed change exceeds predicted change in 63 of 64 cells.Matched-template estimates correlate at 0.96 and 0.97 with the reported estimate.Source: §A.3 · pp. 15–16, Table 9 · p. 16
• Matched-template checks remove the template as an explanation for the gap, but cannot separate change carried by restored evidence from sensitivity to a different input.Source: §A.3 · p. 16
• Matching the template widens retrospective over-crediting: executed BOTH rises from 41.9% to 54.1% on mm-IMDb against a stated 14.8%, and from 8.8% to 20.2% on max-flow against a stated 2.6%.Executed EITHER falls on all three tasks; the reported plain-arm comparison is described as conservative.Source: §A.3 · p. 16, Table 10 · p. 17
• Across 96 grid cells, asking for the self-report has a median task-performance cost of 3.6 pp; on the max-flow joint arm, the median cost reaches 18.4 pp.No cell shows a gain distinguishable from zero.Source: §A.3 · p. 16, Table 11 · p. 17
• The assessment template loses more rows to truncation and parse failure, but each rate comparison uses the intersected item set within its cell.The comparison can still mix template and modality effects when its two sides use different templates.Source: §A.3 · pp. 15, 17

A.4 PROMPT ABLATIONS: WORDING OF THE SELF-REPORT PROMPT

• Across alternative wordings, executed change exceeds predicted change in 52 of 64 cells, equals it in 8, and falls below it in 4.Two open-weight models were tested across four tasks and both single-input conditions; all wordings retained the same three output labels.Source: §A.4 · p. 17, Table 13 · p. 18, Figure 6 · p. 18

A.5 THINKING MODE

• Only the Gemma 4 E2B and E4B pair is used for thinking-mode comparisons because the Qwen3.5 checkpoints lose up to 99% of thinking rows to truncation and repetition loops.Four open-weight hosted models were run with thinking disabled only; the remaining six open-weight models and both proprietary models are untested with thinking enabled.Source: §A.5 · p. 19
• Executed change exceeds predicted change in all 16 thinking-mode cells, and the median prospective gap shifts by only +0.2 pp across tasks.Per-task shifts stay within ±5.0 pp; the largest move is −3.2 pp on max-flow.Source: §A.5 · p. 19
• Thinking raises pooled insufficiency claims from 3.7% to 8.4% and recall from 5.7% to 9.4%, while precision is 73.6% against 75.8%.On mm-IMDb, 24.1% of inputs are called insufficient while the restored modality changes 78.4% of answers.Source: §A.5 · p. 19, Table 15 · p. 20
• On poster-only mm-IMDb, E2B’s insufficiency rate rises from 5.9% to 56.7% and E4B’s from 15.2% to 39.3%, but executed change remains 92.5% and 83.6% against predicted 45.5% and 11.9%.Source: §A.5 · p. 19
• False sufficiency remains the larger retrospective error in seven of eight thinking cells, at 24.9% pooled against 6.3% false insufficiency.Without thinking, the respective pooled rates are 25.2% and 7.8%; max-flow false sufficiency falls from 42.6% to 38.6%.Source: §A.5 · p. 20, Table 16 · p. 20

A.6 SENSITIVITY TO THE ANSWER-MATCHING RULE

• The original rule requires exact genre-set equality; graded rescoring thresholds Jaccard overlap, with ρ = 1 reproducing equality and leaving prompts and comparisons unchanged.The rule was chosen because it matches the prompt, applies to stated and executed answers alike, and requires no threshold.Source: §A.6 · p. 20
• At ρ = 0.75, median executed and predicted change rates are 62.9% and 6.5%, leaving a +56.5 pp median per-cell gap; at ρ = 0.5, no cell reverses.At ρ = 0.75, all 16 model-condition cells retain the gap’s direction.Source: Table 17 · p. 21, §A.6 · p. 21
• At ρ = 0.5, 77.0% of stated attributions collapse into EITHER, so the authors treat that setting as a bound rather than an alternative grading rule.This threshold accepts some substantially different genre sets as equivalent.Source: §A.6 · p. 21
• At ρ = 0.75, sufficiency-flag precision is 76.0% against a 62.9% random baseline, with a +9.6 pp median per-cell lift and 12.8% recall.The reported lift is the median of per-cell differences, not a difference of medians.Source: §A.6 · p. 21, Table 17 · p. 21
• At ρ = 0.75, models understate BOTH attribution by 21.5 pp and over-credit the plot by 17.8 pp; both differences retain their signs under set equality.Under set equality, the corresponding differences are 27.0 pp and 21.2 pp.Source: §A.6 · p. 21, Table 18 · p. 22

A.7 GOLDENVIEW ATTRIBUTION DIAGNOSTICS

• Alignment counts only the attribution naming the single annotated evidence set; EITHER or BOTH cannot identify which set contains the relevant evidence when both single-set runs reproduce the answer.The diagnostic reports singleton-attribution rates and, among those cases, the share identifying annotated evidence.Source: §A.7 · p. 22
• Across 504 cases, answer reproduction returns EITHER in 60.5% and BOTH in 4.8%, versus 41.1% and 1.2% for stated attribution.The greater ambiguity contributes to answer reproduction’s lower unconditional alignment.Source: §A.7 · p. 22, Table 19 · p. 23
• Among single-set cases, stated attribution identifies annotated evidence in 97.3% (283/291), versus 86.9% (152/175) for answer reproduction; these are different subsets.All eight open-weight models have higher stated percentages; the authors treat this comparison as diagnostic, not a separate statistical finding.Source: §A.7 · p. 22, Table 19 · p. 23

B.1 MM-IMDB

• The mm-IMDb subset takes 300 deterministic test films and yields 625 plot-summary rows; comparisons including title-only inputs use film grain, with sibling rows sharing posters and labels.The title-only arm measures label recovery from a name and bounds how the other arms can be interpreted.Source: §B.1 · pp. 23–24

B DATASETS

• The paper’s four arms span mixed mm-IMDb evidence, redundant IsoBench representations, and annotated-unique GoldenView evidence; chance is answer-identity agreement without self-knowledge.The appendix introduces the task-arm summary and directs readers to dataset construction details.Source: Table 20 · p. 23, B DATASETS · p. 23

B.2 ISOBENCH

• IsoBench uses all public validation items—384 parity and 128 maxflow; parity’s approximate 1/3 chance agreement is estimated from observed labels, while maxflow serves as a control.The two tasks help separate genuine agreement from inflation due to parity’s label space.Source: §B.2 · p. 24

B.3 GOLDENVIEW

• GoldenView uses 55 questions across 53 scenes and annotation-blind 3/3 camera splits; 48 of 52 annotated decisive views are forward-facing, limiting rear-view claims.Three questions annotated NONE OF THE ABOVE remain in behavioral metrics but are excluded from annotation-grounded metrics.Source: §B.3 · pp. 24–25

B.4 CONTAMINATION

• Contamination may affect absolute accuracy, but the same-model, same-condition comparison of stated and measured counterfactuals follows the shift caused by increased input redundancy.The authors mark reported absolute accuracy as potentially affected; they note that GoldenView postdates the two model families and its annotation is not recoverable from the underlying nuScenes release.Source: §B.4 · p. 25

C.1 STRUCTURE

• Withheld inputs are removed from both the user message and task description, but GoldenView names missing cameras even in answer-only conditions, unlike mm-IMDb and IsoBench.Thus GoldenView models are told that views are missing, while models in the corresponding answer-only conditions on the other datasets are not.Source: §C.1 · p. 25

C.2 DESIGN

• Prompts elicit the answer and self-report in one generation, keeping each claim attached to the answer being evaluated.A second turn could condition the report on the model’s prior answer and a new instruction, without holding it to the answer produced in the first turn.Source: §C.2 · p. 26
• Self-reports distinguish mentioning an input from using it, permit inputs to be judged irrelevant, and ask models to credit an input only if removing it would change the answer.The prompt avoids a generic instruction to be honest or accurate and instead asks for the task-specific judgment directly.Source: §C.2 · pp. 26–27
• The self-report instrument requests a named counterfactual and a forced-choice sufficiency statement, providing two elicitations of the same construct to compare.A named answer is gradeable in the same answer space as behavior, while the sufficiency choice avoids a bare yes/no item.Source: §C.2 · p. 26

C.3 MM-IMDB GENRE PREDICTION

• mm-IMDb title-only runs have no self-report prompt because restoring text versus image yields different counterfactual questions; title-only remains a reference condition.The task is multi-label prediction over 23 standard genres, with the full label set supplied in prompts.Source: §C.3 · p. 27

C.4 ISOBENCH

• IsoBench pairs informationally equivalent visual and textual representations, using LaTeX rather than code for math parity so representation comparisons do not conflate text with code.For graph maxflow, conditional prompt wording preserves equivalence by specifying source and sink nodes via matrix conventions or drawing colors, as appropriate.Source: §C.4 · p. 29
• IsoBench prompts cover parity and maximum-flow tasks, eliciting either sufficiency and counterfactual answers or separate answers from each input alone.The tasks use text, image, and paired-input conditions; self-reports ask models to distinguish describing an input from relying on it.Source: §C.4 · p. 31

C.5 GOLDENVIEW

• GoldenView divides each question’s six camera views into two three-view evidence sets by hashing question and camera identifiers, independently of decisive-view annotations.For single-set conditions, prompts identify withheld cameras; answer-only conditions share a system message because it does not vary by condition.Source: §C.5 · pp. 32–33

C.6 INSTANTIATED USER MESSAGES

• The instantiated-message examples show one real evaluation message per dataset and per IsoBench subtask, preserving input order and the decoded images carried in the message.The IsoBench subtasks differ in their text input: a LaTeX function definition or an adjacency matrix.Source: §C.6 · p. 34
• The IsoBench example presents a four-node graph maxflow instance with reference answer 24.It pairs with Prompt C.19.Source: §C.6 · p. 35
• The GoldenView example uses Set A and has reference answer A.It pairs with Prompt C.22; the decisive view, CAM_FRONT, is in Set A.Source: §C.6 · pp. 35–36
• The GoldenView prompt supplies CAM_FRONT, CAM_BACK, and CAM_FRONT_LEFT, withholds three right- and rear-facing cameras, and asks why moving right is more feasible.Options include a truck-and-worker lane constraint, alternative explanations involving a red light or oncoming vehicle, and insufficient evidence.Source: §C.6 · pp. 35–36

D GENERATION AND PARSING

• Rows are classified in precedence order as repetition, truncation, provider error, refusal, unparsed, or usable; only usable rows enter the metrics.Answer-only and self-report prompts run separately, and headline metrics use self-report rows; parse failure is also reported among completed rows to separate formatting from generation loss. For mm-IMDb image-only, each generation is copied to an average of 2.08 summary rows in row-grain metrics.Source: D GENERATION AND PARSING · pp. 36–37
• 3.0% of 45,514 attempted self-report rows lose at least one field; answer and self-report parsing failures are 1.6% and 2.6%, respectively.Of the 41,396 finished rows, the two failure sets overlap on 472 rows.Source: D GENERATION AND PARSING · p. 36, Table 23 · p. 39, Table 24 · p. 40
• 0.6% of self-report rows survive in the worst cell, mm-IMDb text-only for Qwen3.5 4B; Qwen3.5 9B retains 2.2%.Across mm-IMDb, 9B truncates 48.8% of self-report rows and loses another 28.9% to repetition loops, leaving 22.3% usable; 4B leaves 45.8%. Thinking-disabled runs show no such problem.Source: D GENERATION AND PARSING · p. 37, Table 22 · p. 38, Table 24 · p. 40
• Qwen3.5 35B-A3B returns a parsable answer on all but 0.3% of graph maxflow self-report rows, but its text-only self-report field fails to parse on 71.3%.Thinking-mode Gemma 4 E2B loses at least one field on 33.4% of completed mm-IMDb text + image rows; affected outputs omit the required literal label.Source: D GENERATION AND PARSING · p. 37, Table 23 · p. 39, Table 24 · p. 40
• GPT-5.6 loses 2 rows to refusal and 3 fields to parse failure, while Claude Sonnet 5 loses 32 rows to refusal and 8 fields.All refusals occur in mm-IMDb conditions containing a film poster; Claude Sonnet 5's largest parse failure is 6 of 128 graph maxflow image-only self-report rows (4.7%). Repetition and refusal signals are unavailable for some model groups, and GoldenView has 55 rows per condition (1.8% per row).Source: D GENERATION AND PARSING · p. 37, Table 22 · p. 38

E.1 MODEL CONFIGURATIONS

• Generations use greedy decoding with a 8,192-token budget; locally served models run in both thinking modes, while OpenRouter models run with thinking disabled.Outputs are unconstrained by grammar or logit masking, and remotely served models are pinned to a provider and precision.Source: §E.1 · p. 40

E.2 REPRODUCIBILITY

• Greedy decoding is not bitwise reproducible: temperature 0 fixes sampling, while vLLM cache sizing lets batch composition and floating-point reduction order drift.Source: §E.2 · p. 40
• The ten-model roster includes two open-weight families, each with locally served and larger hosted models, plus two frontier proprietary models.Each family’s hosted models include one dense and one mixture-of-experts model.Source: Table 25 · p. 41
• 0.3584, 0.3553, and 0.3573 sample-averaged macro F1 across three fixed-condition reruns; the authors treat 0.005 F1 as the noise floor.One or two rows flipped in each rerun.Source: §E.2 · p. 41
• 10,752-token local max model length exceeded the longest observed prompt of 2,597 tokens, so no prompt was truncated.Source: §E.2 · p. 41
• Run records capture provider, generation identifier, reasoning-token count, and cost; prompts were checked byte for byte against live template code.Source: §E.2 · p. 41