PaperlayerView exact revision on arXiv

Source-linked AI summary

CCRV-Bench: Constraint-Based Evaluation of Causal Reasoning in Vision-Language Models

Linyuan Gao, Yuan Wu, Yi Chang

arXiv:2609.30979v1cs.CV

TL;DR

VLM evaluations do not reliably distinguish visual causal reasoning from reliance on statistical shortcuts. CCRV-Bench tests four causal task dimensions under controlled constraints in single-image physical scenarios. Its results show that unconstrained performance does not determine constrained robustness, while the benchmark is limited to static single-frame scenarios.

  • Problem

    Existing evaluations struggle to distinguish causal reasoning based on visual evidence from reliance on statistical correlations, potentially overestimating VLMs’ actual capabilities.

    Source: §1 · pp. 1–2

  • Method

    CCRV-Bench evaluates four causal reasoning dimensions in single-image scenarios using an orthogonal framework of task dimensions and constraints.

    Source: §1 · pp. 1–2

  • Results

    Unconstrained performance does not determine constrained robustness, and effective degradation varies across causal tasks and constraint types.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CCRV-Bench provides a standardized framework for diagnosing image-grounded causal reasoning under controlled constraints.

    Source: §1 · p. 2

    The benchmark covers single-frame static scenarios, while real-world embodied agents also operate in dynamic temporal environments.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Vision-language models (VLMs) have demonstrated excellent performance in visual tasks, but their visual causal reasoning capabilities still lack reliable evaluation. Existing evaluations struggle to distinguish whether a model is performing causal reasoning based on visual evidence or relying on statistical correlations for shortcut learning, thereby potentially overestimating their actual capabilities. This paper proposes CCRV-Bench, a constraint-driven visual causal reasoning benchmark for single-image physical scenarios. We construct an orthogonal framework that evaluates four causal task dimensions: causal relation discovery, state prediction, causal diagnosis, and intervention. We further introduce entity symbolization, spatial grounding, the factual adversarial constraint, and minimalist output constraints to reduce shortcut cues while preserving the physical commonsense required by the task. Experiments across 15 multimodal models show that constraint sensitivity is task- and model-dependent: intervention has the largest average effective degradation among the four causal tasks, spatial grounding is the most damaging constraint on average, and the factual adversarial constraint improves DCR for all evaluated models. These results show that unconstrained performance does not determine constrained robustness and that a single aggregate score can obscure distinct failures in causal identification, spatial grounding, and constraint-compliant expression. CCRV-Bench provides a standardized framework for diagnosing image-grounded causal reasoning under controlled constraints. The code is available at https://github.com/0815linyuan/CCRV-Bench-Constraint-Based-Evaluation-of-Causal-Reasoning-in-Vision-Language-Models

1 Introduction

Single-image visual causal reasoning matters for judging VLM reliability, but existing evaluations do not systematically combine causal tasks with controls against shortcut learning. CCRV-Bench evaluates four causal dimensions under multiple constraints and finds that unconstrained performance does not determine constrained robustness.

1 Introduction

  • Existing benchmarks often isolate causal reasoning from evaluation constraints and lack controls for visual grounding, symbolic binding, false-premise verification, and compressed causal expression.The authors warn that relying only on those evaluations may overestimate models’ actual causal understanding.Source: §1 · p. 1
  • CCRV-Bench evaluates causal relation discovery, state prediction, causal diagnosis, and intervention as complementary dimensions of visual causal reasoning.The motivation is to assess whether models reason from image evidence rather than language priors or statistical correlations.Source: §1 · pp. 1–2
  • The benchmark combines entity symbolization, spatial grounding, factual adversarial prompts, and minimalist outputs in a hierarchical, orthogonal evaluation framework.Its stated contributions include quantitative evaluation across causal dimensions and constraint mechanisms.Source: §1 · p. 2
  • Unconstrained performance does not determine constrained robustness: degradation varies by causal task and constraint, exposing failures that aggregate scores can conceal.The paper frames this as evidence that distinct failures in causal identification, spatial grounding, and constraint-compliant expression need separate diagnosis.Source: §1 · p. 2

2 Related Work

Prior work evaluates multimodal causal reasoning, structured constraints, and visual grounding through different benchmarks and methods. The paper argues that conventional unconstrained prompts can leave shortcut learning difficult to distinguish from image-grounded causal reasoning.

2 Related Work

  • Prior VLM benchmarks cover intervention, counterfactual reasoning, causal hierarchies, video events, 3D interactions, and attention to causally relevant image regions.The cited examples include CELLO, CausalVLBench, InterveneBench, CausalVQA, CAUSAL3D, and structured relevance graphs.Source: §2 · p. 2
  • The authors argue that conventional unconstrained prompts can encourage shortcut learning, making genuine visual causal reasoning difficult to distinguish from learned statistical regularities.They describe reliance on language priors, memorized patterns, correlations, or positional cues as possible shortcuts.Source: §2 · p. 2
  • Constraint-based evaluation has used structured generation and predefined conditions, while preference-based selection has been explored to mitigate multimodal hallucination.The paper notes that outputs may remain visually ungrounded even in structured settings.Source: §2 · pp. 2–3

3 CCRV-Bench

CCRV-Bench organizes single-image causal evaluation around local visual causal models, four task dimensions, and four constraints. Its construction and scoring procedures make the task–constraint design explicit while separately assessing causal quality and constraint compliance.

3 CCRV-Bench

  • The benchmark pipeline comprises data extraction, orthogonal dataset construction, and model evaluation.Figure 1 presents these as the three core modules.Source: §3 · p. 3

3.1 Single-Image Causal Modeling

  • Each image is modeled as a local graph of visible entities and evidence-supported causal edges, supporting relation discovery, state prediction, retrospective diagnosis, and intervention by disabling an interaction.The graph’s nodes are visual entities with spatial attributes, and its directed edges encode causal interactions.Source: §3.1 · pp. 3–4

3.2 Constraint Mechanism Design

  • Four constraints test symbol-to-region binding, target-region grounding, rejection of false textual premises, and preservation of causal conclusions under restricted output length.Spatial grounding requires a coordinate in the target region plus a substantive explanation; the factual adversarial constraint requires correcting the misleading premise using image evidence.Source: §3.2 · p. 4

3.3 Data Construction Pipeline

  • The dataset starts from 100 selected Visual Genome images, filters and human-verifies causal relations, and generates 800 baseline pairs plus 3,200 constrained variants.An independent stratified audit assessed 160 baseline pairs, or 20% of the baseline set.Source: §3.3 · pp. 4–5

3.4 Evaluation Metrics

  • DCR averages cascaded pass rates for entity grounding, causal mechanism, and task-specific conclusion, while CSR measures compliance with the assigned constraint.The cascade withholds higher-level credit when a prerequisite fails, and DCR permits partial credit.Source: §3.4 · p. 5
  • CDI compares baseline and constrained DCR means, while eCDI additionally incorporates constraint violations; positive differences indicate degradation and negative ones indicate gains.Standard summaries use available-record means, which need not have identical matched-record denominators.Source: §3.4 · pp. 5–6

4 Experiments

The experiments evaluate 15 VLMs on CCRV-Bench and find that high baseline scores do not ensure robust constrained causal reasoning. Constraint-related losses vary across models, and complete-chain correctness falls under constraints.

4.1 Experimental Setup

  • The evaluation includes 15 representative VLMs: six closed-source models and nine open-weight models.The paper identifies the model set as representative and lists the models in each group.Source: §4.1 · p. 6

4.2 Results

  • Gemini-3.5-Flash scores 89.9% versus GPT-5.5’s 89.3% at baseline, but loses 29.8 DCR points under constraints versus GPT-5.5’s 10.3-point loss.Across all models, mean L3 falls from 67.2% to 49.8%, leaving fewer than half of constrained responses correct across the full chain.Source: §4.2 · p. 6
  • For 13 of 15 models, constrained DCR declines more than scores lost through constraint violations, while MiniMax-M3 and GLM-5V-Turbo show that causal correctness and constraint following can diverge.MiniMax-M3 has 74.7% constrained DCR but 38.5% CSR and 34.5% eDCR; GLM-5V-Turbo has 62.5% DCR, 93.3% CSR, and 60.7% eDCR.Source: §4.2 · p. 6

5 Additional Analysis

The additional analyses show that degradation depends on both the causal task and the imposed constraint. Intervention has the largest mean DCR decline, spatial grounding is especially damaging, and factual adversarial prompting raises DCR for every evaluated model.

5 Additional Analysis

  • Mean DCR declines by 11.8 points for discovery, 18.3 for prediction, 14.3 for diagnosis, and 22.1 for intervention, the largest drop among the four tasks.Table 3 reports DCR percentages and eCDI percentage-point drops across X1–X4.Source: §5 · pp. 6–7, Table 3 · p. 7
  • Small task-specific declines do not necessarily indicate strong constrained capability: GPT-5.5’s discovery score stays at 82.3%, while Pixtral-12B’s discovery score drops from 70.8% to 48.6%.DeepSeek-V3.2 changes little on discovery, from 47.8% to 46.2%, despite starting from a low baseline.Source: §5 · p. 7

Y Dimension: Constraint-Specific Effects. Ta-

  • Spatial grounding lowers mean constrained DCR to 41.1% and gives the lowest constrained DCR for 10 of 15 models; under factual adversarial constraint, all 15 exceed baseline DCR.The factual adversarial setting yields mean DCR of 90.1% and mean eCDI of −13.1 points; entity symbolization and minimalist output cause more moderate but substantial degradation.Source: Y Dimension: Constraint-Specific… · p. 8, Table 4 · p. 8

5.1 Independent Human Validation

  • A blinded audit of 300 responses across 15 models, five settings, and four tasks reports Fleiss’ κ from 0.719 to 0.849, supporting automated evaluation alongside human auditing.Human-majority agreement with GPT-5.1 ranges from 81.67% to 87.67% across L1–L3 and is 78.33% for full-chain pass.Source: §5.1 · p. 8

6 Conclusion

The conclusion presents CCRV-Bench as a constrained framework for evaluating VLM visual causal reasoning across four task dimensions. Its results indicate that unconstrained evaluation does not capture the performance degradation observed when shortcut cues are reduced.

6 Conclusion

  • CCRV-Bench evaluates visual causal reasoning under constraints across causal structure discovery, prediction, attribution, and intervention.The framework is designed to diagnose VLM capabilities under constrained conditions.Source: §6 · p. 8
  • The authors report that traditional unconstrained evaluations overestimate causal capabilities, with higher-order generative reasoning degrading severely once shortcut cues are reduced.The reported results characterize structural limitations in current multimodal models’ causal reasoning.Source: §6 · p. 9
  • The paper positions CCRV-Bench as an evaluation foundation for assessing and developing multimodal and embodied intelligence systems with reliable causal reasoning.Source: §6 · p. 9

Limitations

The benchmark’s findings are bounded by its single-frame physical scenarios, dataset coverage, constraint-generation process, and focus on semantic, spatial, and logical shortcuts.

Limitations

  • The benchmark covers single-frame static scenarios, so it does not evaluate causal reasoning in continuous temporal environments, videos, or 3D interactions.The authors suggest paired-image or video extensions while retaining the X1–X4 tasks and L1/L2/L3 structure over time.Source: Limitations · p. 9
  • Physical scenes are primarily sourced from Visual Genome and focus on everyday commonsense interactions, limiting coverage of specialized or complex physical domains.Source: Limitations · p. 9
  • The LLM-in-the-loop constraint-mutation pipeline may introduce subtle linguistic biases or prompt structures that favor models with similar alignment paradigms.The authors note this possibility despite rigorous manual verification.Source: Limitations · p. 9
  • The constraints target semantic, spatial, and logical shortcuts; the authors identify temporal modalities and broader physical domains as extensions.Source: Limitations · p. 9

A Data Construction Details

CCRV-Bench filters and verifies directional physical relations before generating baseline and constrained QA pairs, then audits a stratified sample of baseline questions. Its additional results trace constrained failures across visual grounding, mechanism understanding, and causal outcomes.

A.1 Causal-Edge Filtering, QA Generation, and Human Audit

  • Candidate relations were filtered for physical interactions, then retained only when image-grounded entities supported a directional mechanism expected to affect the target’s state, motion, or stability.Reviewers excluded unsupported causal directions, ambiguous mechanisms, and relations requiring assumptions about hidden intentions or prior events.Source: §A.1 · p. 11
  • The pipeline generated 800 baseline QA pairs from verified relations and expanded each into four constraint-specific variants, yielding 3,200 variants.Each image contributed two baseline pairs for each of the four causal dimensions; there were 200 variants per dimension–constraint combination.Source: §A.1 · p. 11
  • 95.625% of 160 audited items matched across all three categorical labels, with seven disagreements adjudicated and all seven QA pairs retained unchanged.Agreement was 100.0% for Discovery, 90.0% for Prediction, 95.0% for Diagnosis, and 97.5% for Intervention.Source: §A.1 · pp. 11–12
  • The audit covered 160 baseline pairs, while each of the 200-item dimension–constraint cells has an approximate worst-case standard error of 3.5 percentage points.Questions are paired on the same images and causal relations; image-level pretraining exposure cannot be fully ruled out.Source: §A.1 · p. 12

B Additional Results

  • Under constraints, several models retain partial entity grounding but lose mechanism-level or outcome-level credit, so constrained drops are not all attributable to causal inference alone.The reported breakdown follows the transition from visual perception (L1) through physical mechanism (L2) to causal outcome (L3).Source: B Additional Results · p. 12

B.1 Analysis of the Factual Adversarial Constraint Gain

The factual adversarial constraint raises task-conditioned DCR across all 15 models, while generic reasoning prompts do not reproduce that consistent pattern. Supplementary controls show that visual evidence strongly affects cascaded performance and that supplying causal content does not remove all constraint-specific difficulty.

B.1 Analysis of the Factual Adversarial Constraint Gain

  • All 15 models exceed their baseline DCR under Y3, with mean DCR of 90.1% and mean eCDI of −13.1 points.The gain is a task-conditioned score under explicit verification, not an unconditional increase in latent causal ability.Source: §B.1 · p. 12
  • DCR values share a cascaded scale, but the constraints are not matched-difficulty manipulations, and Y3 DCR and CSR are overlapping but non-equivalent signals.DCR gives graded cascaded causal credit, whereas CSR is an all-or-nothing constraint-compliance decision; the Y3 eDCR gate can reinforce behavior already reflected in DCR.Source: §B.1 · pp. 12–13
  • Y3 task-conditioned eDCR exceeds DCR under all three comparison prompts for every evaluated model, with a mean of 93.1%.CoT-A lowers mean DCR from 82.2% to 79.7%; CoT-B leaves mean DCR nearly unchanged at 82.1% versus 82.2%.Source: Table 5 · p. 13, §B.1 · p. 13

B.2 Supplementary Diagnostic Controls

  • With images, mean DCR is 81.58%, versus 22.67% without images; raw L3 is 84.12% versus 56.10%.The five-model diagnostic controls vary access to visual evidence or causal relations without changing model parameters or adding training data.Source: §B.2 · p. 13, Table 6 · p. 14
  • In the no-image Y1 control, mean DCR rises from 54.82% with text-only input to 83.81% with an oracle-informed reference, near the 81.58% image-based baseline.Because the oracle answer encodes causal content, this control does not isolate visual causal-edge discovery.Source: §B.2 · p. 14
  • Image plus oracle-informed text yields mean gains of 18.74 points for Y1, 11.03 for Y2, and 20.02 for Y4, yet Y2 remains below the image-based baseline.The supplied semantic content can reveal the final causal answer but does not provide the Y2 coordinate or Y4 compressed output form.Source: §B.2 · p. 14

C Full Experimental Results

This section reports the full-results tables and illustrates how baseline causal questions are adapted to the benchmark’s four constraints. The examples show the distinct response requirements imposed by symbolization, spatial grounding, false-premise rejection, and minimalist output.

C Full Experimental Results

  • The full-results tables report baseline DCR by causal task and constrained-set DCR by constraint, with eDCR and eCDI aggregating across constraints.For harmonized Y2, DCR and eDCR use a fixed 800-item denominator, while paired eCDI uses matched baseline–constrained pairs and unrounded scores.Source: C Full Experimental Results · p. 14, Table 9 · p. 16, Table 10 · p. 17

D Examples of Constraint Construction

  • The construction example transforms one image interaction into four constrained evaluations targeting symbolization, spatial grounding, false-premise rejection, and minimalist causal expression.The illustrated interaction is a man catching a frisbee.Source: D Examples of Constraint Construction · p. 14, Figure 3 · p. 15

D.1 Example 1: Causal Discovery (X1)

  • For causal discovery, entity symbolization replaces real entity names with node placeholders, while spatial grounding asks for an image coordinate marking the action.The example asks which action Node_D performs on Node_C and requests a coordinate plus explanation for the catching action.Source: §D.1 · p. 15

D Examples of Constraint Construction

  • The displayed adapted answers are illustrative only; inference omits them, the hidden node-to-entity mapping, real entity names, object IDs, and causal annotations.The manuscript also distinguishes canonical model names from internal aliases and excludes Qwen3.6-35B as a separate baseline-only record.Source: §D.1 · p. 15, §D.2 · p. 17

D.1 Example 1: Causal Discovery (X1)

  • The factual adversarial example asks the model to check a potentially false claim against the image before identifying the actual action affecting the frisbee.The adapted answer rejects the claim that the man ignores the frisbee and describes him catching and holding it.Source: §D.1 · p. 16, §D.2 · p. 16

D.2 Example 2: Retrospective Diagnosis (X3)

  • In the diagnosis example, symbolization asks which node causes the frisbee’s contact state, while spatial grounding requests a coordinate marking the causal origin.The displayed coordinate explanation identifies the man’s hand as the physical cause.Source: §D.2 · p. 16
  • The retrospective-diagnosis example asks why the frisbee remains against the man, with a misleading wind explanation to reject and a five-word response limit.The example’s adapted answer attributes the state to the man’s hand intercepting and holding the frisbee.Source: §D.2 · pp. 16–17

E Independent Human Agreement Analysis

The authors audit model responses with blinded human annotations and compare human-majority judgments with GPT-5.1. They also describe the generation and evaluation prompts and the shared inference setup used across evaluated models.

E Independent Human Agreement Analysis

  • Three blinded annotators independently label perception, mechanism, and outcome for 300 responses sampled across all models, settings, and causal tasks.The sample covers every model–setting–task combination, with 20 responses per model.Source: E Independent Human Agreement Analysis · p. 17
  • Human-majority agreement with GPT-5.1 reaches 87.67% for L3, and full-chain agreement is 78.33%, supporting automated judging alongside human auditing.Across L1–L3, Fleiss’ κ ranges from 0.719 to 0.849; full-chain human–human Fleiss’ κ is 0.750.Source: E Independent Human Agreement Analysis · pp. 17, 19

F Prompt Design for QA Generation and Evaluation

  • GPT-5.1 generates baseline question–answer pairs from human-verified causal-edge specifications, while separate DCR and CSR prompts assess reasoning quality and constraint adherence.DCR evaluates perception, mechanism, and outcome; CSR focuses particularly on the factual adversarial constraint, with partial semantic overlap under Y3.Source: F Prompt Design for QA Generation and… · p. 19

F.1 Inference Settings

  • All models use shared high-level baseline and constrained instructions, with temperature set to 0.0 where supported and output length governed by API or deployment defaults.The reported calls did not pass an explicit output-token limit.Source: §F.1 · p. 19

G Programmatic Evaluation Rules for CSR

Constraint compliance is evaluated with programmatic checks for entity symbolization, spatial grounding, and minimalist output, and with an LLM judge for factual-adversarial responses. The rules specify what outputs count as compliant and clarify that Y3 judging does not independently re-check the image.

G Programmatic Evaluation Rules for CSR

  • Y1 compliance fails if the response contains any forbidden real-entity name, detected by a lowercase regular-expression scan.The prompt substitutes symbolic node placeholders for those entity names.Source: G Programmatic Evaluation Rules for CSR · p. 19
  • Y2 requires one coordinate and an explanation of at least 20 characters, with the coordinate falling inside an expanded target box under either valid coordinate interpretation.The target box is expanded by 7.5% of its width horizontally and height vertically; missing or non-unique coordinates and short explanations fail.Source: G Programmatic Evaluation Rules for CSR · p. 19
  • Y3 earns compliance only when the answer identifies the injected premise as false, refuses to reason from it, and provides a correction grounded in task-described visual evidence.The specification-conditioned judge does not receive the image, so this is semantic compliance assessment rather than independent visual verification.Source: G Programmatic Evaluation Rules for CSR · p. 20
  • Y4 passes only when the response avoids forbidden punctuation and stays within the requested word limit.Hyphenated compounds are split into distinct tokens after punctuation screening.Source: G Programmatic Evaluation Rules for CSR · p. 20
  • The DCR and CSR judge prompts score causal reasoning and constraint adherence separately, although their false-premise criteria partially overlap under Y3.The DCR prompt separates perception, mechanism, and outcome.Source: Figure 5 · p. 21
Loading 2609.30979v1…