EDCT-Bench: Uncovering Faithfulness Gaps in VLMs via Explanation-Driven Counterfactual Testing

1. Introduction

• EDCT targets a gap in plausibility-based evaluation: explanations can cite ignored or hallucinated features, while sounding reasonable does not guarantee consistency with visual evidence.Such post-hoc rationalizations may mask biases or faulty logic [2].Source: §1 · p. 1
• EDCT alters visual concepts cited in a model’s explanation and checks whether its answer and explanation remain consistent with the edited image.An unchanged answer can be valid when remaining evidence supports it, but the explanation must no longer rely on altered evidence.Source: §1 · p. 1
• All evaluated VLMs, including strong proprietary models, exhibit substantial gaps when tested against counterfactual changes to evidence cited in their explanations.EDCT’s model-specific interventions also make it harder to optimize against a fixed set of externally specified edits.Source: §1 · p. 1
• The authors operationalize EDCT as a pipeline of baseline response acquisition, explanation-grounded concept extraction, counterfactual editing, localized verification, and consistency scoring.The benchmark comprises 300 verified tests from OK-VQA [25], DriveLM [33], and 3DSRBench.Source: §1 · pp. 1–2
• In the racket example, EDCT replaces the cited stringed surface with a solid paddle-like surface; continuing to mention strings conflicts with the verified edit.This inconsistency may indicate image misunderstanding or reliance on priors.Source: Figure 1 · p. 2

2. Related Work

• EDCT extends intervention-based tests of explanation faithfulness to visual evidence by editing cited objects, attributes, or spatial relations and checking answer and explanation consistency.Unlike text-only interventions, its visual counterfactuals require edit locality and semantic validity checks.Source: §2 · p. 2
• Because EDCT derives each intervention target from the model’s own explanation, different explanations for one image-question pair can yield different tests rather than a fixed counterfactual set.This adaptivity reduces direct overfitting to fixed interventions.Source: §2 · p. 2
• EDCT tests responses to edits of cited image content, complementing work on textual reasoning traces and prompt-conditioned visual editing.It uses targeted edits to alter cited entities or attributes while preserving the surrounding scene.Source: §2 · pp. 2–3

3. Approach

• EDCT takes an image, question, answer, and explanation through four stages: baseline response and cited-concept extraction, counterfactual generation, localized edit verification, and consistency testing.A passing test is not proof of internal causal reasoning; a failure is behavioral evidence of a faithfulness gap, while a pass means only that no inconsistency was detected under tested interventions.Source: §3 · p. 3

3.1. Baseline Response and Concept Extraction

• The target VLM first provides a baseline answer and explanation, from which an LLM extracts the objects or attributes the explanation presents as supporting the answer.Extracted concepts become edit instructions for the next stage.Source: §3.1 · p. 3

3.2. Counterfactual Generation

• Text-guided image editors generate counterfactuals that minimally alter each extracted concept while aiming to preserve the rest of the scene.Edit instructions specify attributes to add, modify, or remove.Source: §3.2 · p. 3

3.3. Localized Edit Verification

• Localized verification uses a detector and segmentation model to mask the target region, then requires sufficient change inside the mask and minimal change outside it.Pixel and structural changes are measured after aligning the edited image to account for minor geometric distortions.Source: §3.3 · pp. 3–4

3.4. Consistency Testing

• Answer consistency allows an unchanged answer when unedited evidence still supports it, but explanation faithfulness fails if the new explanation continues to rely on removed or altered evidence.A concept is necessary when changing it leaves the original answer unsupported; other remaining cues may be sufficient to support that answer.Source: §3.4 · p. 4

3.5. Multiple Concepts

• Binary CCS multiplies answer-consistency and explanation-faithfulness scores, while rubric-graded CCS assigns a normalized score from 1 to 5 based on counterfactual faithfulness.For multiple concepts, EDCT generates a compatible chronological edit chain, re-queries after each edit, and averages CCS across the chain.Source: §3.4 · p. 4, §3.5 · p. 4

4.1. Setup

• The evaluation covers Pixtral-12B, Gemma3-27B, two Qwen3-VL models, GLM-4.6V-106B, and Gemini 2.5 Flash, with k = 2 visual concepts prioritized per explanation.The pipeline also uses specified models for concept extraction, editing, and localized edit verification.Source: §4.1 · p. 5

4.2. Datasets

• EDCT-Bench contains 300 manually curated image-question pairs: 160 OK-VQA, 80 DriveLM, and 60 3DSRBench samples spanning knowledge, driving, and 3D spatial reasoning.DriveLM uses front-facing-camera questions; the 3DSRBench examples target orientation and related spatial relationships.Source: §4.2 · pp. 5–6

4.3. EDCT Results

• Gemini 2.5 Flash reaches only 0.600 CCSBinary and 0.603 CCSGraded, while other evaluated models score 0.435–0.566 CCSBinary, revealing substantial counterfactual consistency failures.These scores measure consistency after cited visual concepts are minimally altered.Source: §4.3 · p. 6
• CCS does not uniformly track capability or CoT-Bias diagnostics; Qwen3-VL-8B’s higher aggregate scores than the 30B-A3B model are descriptive, with overlapping confidence intervals.Answer-side consistency is generally higher than explanation-side consistency, and CCSBinary is stricter than CCSGraded, which may give partial credit for implicit updates.Source: §4.3 · p. 6, §4.7 · p. 8
• 3DSRBench is the hardest domain for every model, while model ordering varies across datasets, making EDCT-Bench a domain-sensitive stress test rather than a universal leaderboard.Qwen3-VL-8B exceeds Qwen3-VL-30B-A3B on DriveLM and 3DSRBench, but the 30B-A3B model leads on OK-VQA under CCSBinary.Source: §4.3 · p. 6

4.4. Human Validation of CCS and Edit Validity

• On 100 interventions scoreable by both annotators and EDCT, annotators agreed in 88 cases and all three labels matched in 83; automatic edit-validity agreement was 89.8% and 95.3% with the two annotators.The 89.8% and 95.3% values are automatic-gate agreement with Human 1 and Human 2, respectively; the annotators agreed on edit validity in 86.7% of cases.Source: §4.4 · pp. 6–7

4.5. Statistical Stability Across Runs

• Across five benchmark runs, aggregate CCS measurements do not change substantially despite variation in individual samples and intermediate trajectories.The authors report this stability as support for the metrics.Source: §4.5 · p. 7, Table 5 · p. 7

4.6. Counterfactual Artifacts as High-Impact Training Signals

• Qwen3-VL-8B scores 69.0% on original samples but 34.5% on counterfactuals, with initial training loss 4× higher on counterfactuals.The 48-sample preliminary study finds qualitatively more localized attention around edited axles, but does not establish improved held-out faithfulness; larger controlled evaluation is needed.Source: §4.6 · pp. 7–8, Figure 4 · p. 8

5. Conclusion

• EDCT edits visual evidence cited in VLM explanations and evaluates whether answers and explanations remain consistent with the edited image.Human agreement and five-run stability analyses further support the reliability of CCS and the overall pipeline.Source: §5 · p. 8
• All evaluated models exhibit counterfactual-consistency gaps on 300 samples from OK-VQA, DriveLM, and 3DSRBench.The benchmark spans these three datasets.Source: §5 · p. 8
• Intervention difficulty may vary across models because the interventions derive from model-specific explanations.The authors call for evaluation on more samples and counterfactual variants, and for human-verified scene descriptions to condition editing and judging without exposing them to the target VLM.Source: §5 · p. 8
• In a preliminary 48-sample study, EDCT counterfactuals produce 4× higher initial loss and more localized attention, but held-out faithfulness improvement requires confirmation using post-finetuning CCS.Source: §5 · p. 8

A. More Examples

• Figure 6 presents additional EDCT examples on OK-VQA.Source: Figure 6 · p. 12
• Figure 7 presents another chain of counterfactual edits based on extracted visual concepts.Source: Figure 7 · p. 13
• Figure 8 compares an original image with a dog-breed edit and shows pixel/structural differences plus masks verifying edits stay within the target object while preserving the background.Source: Figure 8 · p. 13

B. Prompts

• The extraction prompt selects specific, distinct visual concepts in the explanation that relate directly to the question and answer for counterfactual testing.It discourages redundant or semantically similar concepts and requests short concept phrases without extra explanation.Source: B. Prompts · p. 14
• Editing prompts target cited features while keeping the rest of the scene unchanged, producing plausible, realistic edits that leave the original question relevant.Instructions specify what to keep, change, how to render the edit, and constraints; they favor changing the attribute over replacing the whole object unless necessary.Source: B. Prompts · p. 14
• For multiple concepts, each edit is applied after earlier edits, and evaluators judge the answer and explanation against the final cumulative image state.The binary scheme scores prediction change and explanation concept consistency separately, then defines CCS as PCS multiplied by NCC; unchanged answers can be valid when unedited evidence remains sufficient.Source: B. Prompts · pp. 14–15
• A separate 1–5 rubric rates faithfulness by whether both the edited answer and explanation fit the counterfactual image, distinguishing implicit or partial grounding from unfaithful inertia.If an answer stays the same because other evidence remains valid, its explanation should shift to that evidence and stop citing the removed feature.Source: B. Prompts · p. 16

• The prompt requests a 300–400-word scene analysis and a separate 300–400-word analysis of the right score.Source: B. Prompts · pp. 16–17
• The final response must contain only the score, formatted as “CCS: [1 to 5]” within the specified tags.Source: B. Prompts · p. 17

C. Additional Attention Visualizations

• Figure 9 shows attention maps for held-out axle examples across fine-tuning epochs.Source: Figure 9 · p. 18
• In the shown examples, attention becomes more localized around the target features over fine-tuning epochs.Source: Figure 9 · p. 18
• Peaks at the top of the image are identified as VLM attention sinks.Source: Figure 9 · p. 18