PaperlayerView exact revision on arXiv

Source-linked AI summary

CauSight: Learning to Supersense for Visual Causal Discovery

Yize Zhang, Meiqi Chen, Sirui Chen, Bo Peng, Yanxi Zhang, Tianyu Li, Chaochao Lu

arXiv:2512.01827v1cs.CV

TL;DR

Visual causal discovery asks models to infer how visual entities affect one another, a capability that scene graphs and current VLMs handle poorly. The paper builds VCG-32K and develops CauSight using synthesized reasoning trajectories and causal-reward reinforcement learning. CauSight reports a 21.0% absolute graph-recall gain over GPT-4.15 and improvements across in-domain and cross-domain settings.

  • Problem

    Visual causal discovery requires grounding visual concepts in language while reasoning about causal relations, which conventional scene graphs do not represent and current VLMs struggle to infer.

    Source: §1 · p. 2

  • Method

    The paper introduces VCG-32K and trains CauSight with Tree-of-Causal-Thought reasoning trajectories and reinforcement learning guided by a graph-based causal reward.

    Source: §1 · p. 2, §4.1 · p. 4, §4.2 · p. 5

  • Results

    21.0% absolute gain in graph recall over GPT-4.15, with strong performance on in-domain and cross-domain subsets and consistent F1 improvement.

    Source: §5.2 · p. 6

  • Takeaways & Limitations

    Causal graphs represent cause-and-effect chains that scene graphs omit, making them relevant to downstream reasoning and intervention tasks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Causal thinking enables humans to understand not just what is seen, but why it happens. To replicate this capability in modern AI systems, we introduce the task of visual causal discovery. It requires models to infer cause-and-effect relations among visual entities across diverse scenarios instead of merely perceiving their presence. To this end, we first construct the Visual Causal Graph dataset (VCG-32K), a large-scale collection of over 32,000 images annotated with entity-level causal graphs, and further develop CauSight, a novel vision-language model to perform visual causal discovery through causally aware reasoning. Our training recipe integrates three components: (1) training data curation from VCG-32K, (2) Tree-of-Causal-Thought (ToCT) for synthesizing reasoning trajectories, and (3) reinforcement learning with a designed causal reward to refine the reasoning policy. Experiments show that CauSight outperforms GPT-4.1 on visual causal discovery, achieving over a threefold performance boost (21% absolute gain). Our code, model, and dataset are fully open-sourced at project page: https://github.com/OpenCausaLab/CauSight.

1. Introduction

The paper frames visual causal discovery as reasoning about how entities affect one another, beyond recognizing objects or their spatial relations. It introduces VCG-32K and CauSight to support this task and reports improved performance and cross-dataset generalization.

1. Introduction

  • Causal graphs are presented as supporting logical, safe, and precise downstream decisions, with robust generalization under changing conditions.The paper cites causal graphs as a basis for these potential benefits.Source: §1 · p. 1
  • Visual causal discovery grounds visual concepts in language by explicitly reasoning about causal relations between entities.A causal relation is described through counterfactual reasoning: without the cause, the effect would not sustain its current state.Source: §1 · p. 2
  • Scene graphs encode spatial or associative relations, whereas causal graphs represent cause-and-effect chains relevant to reasoning and intervention.The introduction argues that scene graphs do not model how changing one entity affects another.Source: §1 · p. 2
  • VCG-32K provides over 32,000 images annotated with entity-level causal graphs, while CauSight generates causal graphs across visual scenarios through causally aware reasoning.The training recipe combines data curation, ToCT-generated reasoning trajectories, and reinforcement learning with a causal reward.Source: §1 · p. 2
  • CauSight exceeds its base model by 27.4% (≈8.2×) and GPT-4.1 by 21.0% (≈3.1×), with strong cross-dataset generalization.The comparison is reported as absolute gains; the passage does not specify the metric.Source: §1 · p. 2

2. Task Formulation: Visual Causal Discovery

Visual causal discovery constructs a graph from one image, with entities represented by category labels and bounding boxes and directed, predicate-labeled edges representing causal relations. The paper defines those relations through interventions and evaluates predicted graph structure.

2. Task Formulation: Visual Causal Discovery

  • A visual causal graph is constructed from a single image and comprises entities and their causal relationships.Each entity is represented by a category label and a bounding box.Source: §2 · p. 2
  • A directed edge from one entity to another exists when intervening on the first changes the state of the second.The formulation represents intervention as removing the entity; the entity state can include presence or sitting.Source: §2 · p. 3
  • Evaluation ignores entity and predicate labels, matches entities to ground truth with Hungarian matching and GIoU, and checks whether edges have the correct causal direction.An edge counts as correct when the ground-truth graph contains it with the same direction.Source: §2 · p. 3

3. The VCG-32K Dataset

VCG-32K extends visual-understanding datasets with refined entity boxes, explicitly annotated causal relationships, and mechanism labels. It is built through a two-stage annotation process on MS-COCO and Objects365 images.

3. The VCG-32K Dataset

  • VCG-32K adds accurate entity localization, explicit causal relations, and mechanism types that explain why a relation exists.These annotations are intended to guide causally aware reasoning beyond perception-level visual understanding.Source: §3 · p. 3
  • The two-stage pipeline refines bounding boxes first, then labels causal relationships between entities.Annotators may modify, delete, or add boxes; relationship labels require direct contact and that removing one entity would change another's current state.Source: §3 · p. 3, Figure 2 · p. 3
  • The dataset uses images from MS-COCO and Objects365, with 50 trained annotators and quality checks reporting average annotation accuracy above 95%.Ten reviewers randomly sample items and correct errors.Source: §3 · p. 3
  • VCG-32K contains 32,256 images, 299,262 entities, 2,287 entity categories, and 185,321 causal relationships.The dataset averages 5.75 relationships per image.Source: §3 · p. 3

4. Method

CauSight’s method combines Tree-of-Causal-Thought trajectories to initialize its reasoning policy with reinforcement learning that further optimizes visual causal discovery using a designed causal reward.

4. Method

  • Tree-of-Causal-Thought expands a reasoning path into a tree to select high-quality demonstrations, followed by reinforcement learning to optimize CauSight’s reasoning policy.The training pipeline uses the selected trajectories to initialize the policy.Source: §4 · p. 3, Figure 4 · p. 4

4.1. Tree-of-Causal-Thought

  • ToCT loops region selection, entity recognition, and causality orientation; MCTS expands trajectories through selection, expansion, simulation, and backpropagation.Leaf states are evaluated against ground-truth causal graphs, and search returns a greedy path through the highest-value nodes.Source: §4.1 · pp. 4–5
  • Only ToCT trajectories that outperform vanilla one-step reasoning are retained for supervised fine-tuning of the reasoning policy.This filtering is intended to avoid training on suboptimal reasoning patterns.Source: §4.1 · p. 5

4.2. Reinforce with Causal Reward

  • Reinforcement learning starts from the SFT-trained policy and uses GRPO, which compares sampled outputs for one input without a separate value function.Relative comparisons across rollouts are used to capture differences in graph-reconstruction quality while reducing computational overhead.Source: §4.2 · p. 5
  • The causal reward combines Recall and Precision for visual causal discovery with Format compliance for faithful structural generation.Recall encourages alignment with the ground-truth graph, Precision suppresses false positives, and Format checks compliance with the output schema.Source: §4.2 · p. 5

5. Experiments

Across VCG-32K’s in-domain and cross-domain benchmarks, CauSight outperforms baselines; further experiments examine reasoning and detection, training components, and broader capabilities.

5.1. Setup

  • Evaluation uses 350 VCG-32K samples each from MS-COCO and Objects365, with MS-COCO training enabling cross-dataset testing.The ground-truth-label SFT baseline is distinct from the policy model SFTed on ToCT-induced trajectories.Source: §5.1 · p. 5

5.2. Results on VCG-32K

  • CauSight gains 21.0% absolute graph recall over GPT-4.1 and outperforms all baselines in-domain and cross-domain.On VCG-32K-365, it improves graph recall by 16.0% and F1 by 17.3% over the SFT variant.Source: Table 1 · p. 6, §5.2 · p. 6

5.3. Reasoning and Detection

  • CauSight reaches 34.2% graph recall with 8.0% reasoning-induced loss, balancing causal discovery performance with detection stability.GPT-4.1 has the highest stability, while direct SFT severely degrades detection capability; reasoning loss and RSI show opposing trends across models.Source: Table 2 · p. 6, §5.3 · p. 7, Figure 5 · p. 7

5.5. General Capability Assessment

  • CauSight largely preserves the base model’s generalization across Math-V, MathVista, and BLINK, while direct SFT drops over 20% on math reasoning tasks.Figure 6 reports model accuracy in each benchmark cell.Source: Figure 6 · p. 7, §5.5 · p. 8

5.6. Qualitative Comparison

  • CauSight produces high-fidelity, structurally consistent causal relations and attribute-based reasoning without category-label supervision.The qualitative comparison includes a corrected ground-truth label from “chopping board” to “table.”Source: Figure 7 · p. 8, §5.6 · p. 8

5.4. Ablation Study on Training Recipe

  • Across both benchmarks, CauSight improves graph recall by 27.4% over the base model, 21.1% over the model without RL, and 6.7% over the model without ToCT.On the cross-domain benchmark, SFT on ToCT trajectories before RL yields a 10.1% graph-recall increase over RL alone.Source: §5.4 · p. 7, Table 3 · p. 8

6. Related Work

Prior causal-learning work either scales poorly beyond low-dimensional settings, uses mostly synthetic or interactive data, or does not learn explicit causal structures from visual observations. Existing visual QA and causal benchmarks lack explicit structures and rely on shallow event reasoning.

6. Related Work

  • Classical structural causal-model approaches work in low-dimensional settings but scale poorly, while newer causal-learning methods largely use interactive data or synthetic domains.Graph-based and differentiable DAG methods improve scalability but rarely model explicit causal structures from visual observations.Source: §6 · p. 8
  • Visual QA and causal benchmarks touch on causality but lack explicit structures and rely on shallow event reasoning.Text-based LLM causal evaluations cover commonsense, extraction, and multi-step reasoning, while visual grounding remains limited.Source: §6 · p. 8
  • The paper addresses these gaps by learning explicit causal graphs from images for fine-grained, entity-level reasoning in vision-language contexts.Source: §6 · p. 9

7. Conclusions

The work introduces CauSight, a vision-language model for constructing causal graphs across diverse visual scenes, and VCG-32K, a large-scale dataset intended to support research in visual causal discovery.

7. Conclusions

  • CauSight is introduced as a vision-language model that constructs causal graphs across diverse visual scenes.Source: §7 · p. 9
  • The work formulates visual causal discovery as a task and proposes VCG-32K as a large-scale dataset for further research.Source: §7 · p. 9
  • The authors argue that causal understanding is pivotal for AI systems to generalize to complex environments and unseen tasks.Source: §7 · p. 9

A. Synthesized Trajectories

After excluding zero-recall cases, ToCT improves mean recall over vanilla one-step reasoning; the filtering procedure retains trajectories where ToCT strictly outperforms the vanilla approach on the same image.

A. Synthesized Trajectories

  • ZERO denotes samples where both ToCT and vanilla have 0 recall, treated as relatively extreme cases such as excessively difficult samples.Source: A. Synthesized Trajectories · p. 13
  • 3,631 of 6,000 trajectories remain after filtering for strict ToCT improvement over vanilla on the same image.Filtered trajectories have mean recall 0.50, versus 0.21 for their vanilla counterparts.Source: A. Synthesized Trajectories · p. 13
  • 0.42 mean recall for ToCT without ZERO, a 13% absolute gain over vanilla; Table 4 reports the comparison across conditions.About 50% of samples had an improvement greater than 0.1; the scatter plot covers 6,000 samples.Source: Table 4 · p. 13, A. Synthesized Trajectories · p. 13

B. Dataset

VCG-32K annotations refine candidate entity boxes and label directed causal relationships between entity pairs, with junior annotations reviewed by senior annotators.

B. Dataset

  • Images are sampled from MS-COCO and Objects365 based on entity counts and diversity of associated labels.Source: B. Dataset · p. 13
  • Annotators review pre-annotated boxes, correcting or deleting those with excessive empty space or truncation, overly broad or factually incorrect labels, or entities lacking direct contact or interaction with another scene entity.They add an unannotated entity only when it engages in direct physical contact or force interaction with at least one other scene entity.Source: B. Dataset · p. 13
  • Annotators label causal attributes and directions for every ordered entity pair, with a relationship requiring contact and maintenance of the target's current state.Removing the putative cause must make the target lose its current state.Source: B. Dataset · p. 13
  • After training to 95% accuracy on a pre-annotated validation set, 50 junior annotators labeled data and 10 senior annotators reviewed both annotation stages.The example encodes entities with IDs and represents directed relations using categories such as carry_on and support.Source: B. Dataset · pp. 13–14

C. Qualitative Comparisons

The qualitative cases contrast CauSight with the base and SFT models, show its structured reasoning path, and present prompts for region selection, entity recognition, and causal decisions.

C. Qualitative Comparisons

  • Black solid edges mark correct causal edges, dashed black edges mark correct relations without precise box matches, and gray solid edges mark incorrect edges.Source: C. Qualitative Comparisons · p. 14
  • The base model has zero recall in the provided cases, while SFT improves slightly but hallucinates in complex scenes and over-reasons in simpler ones.Source: C. Qualitative Comparisons · p. 14
  • CauSight captures essential causal structure, achieving perfect recall in simpler cases and identifying main causal relations in complex cases despite imperfect box matches.Source: C. Qualitative Comparisons · p. 14
  • CauSight’s illustrated reasoning path proceeds through region focus, entity recognition, and causal-relationship inference.Source: C. Qualitative Comparisons · p. 14, Figure 9 · p. 15

Region Selection

  • Region selection asks the model to choose an unexplored, worthwhile region and return its name and bounding box, or end the trace when coverage is sufficient.Source: Region Selection · p. 22

Entity Recognition

  • Entity recognition identifies potentially correlated entity pairs, such as through spatial dependence, support, grasping, placement, or inclusion, and outputs their bounding boxes.Source: Entity Recognition · p. 22

Causality Orientation

  • Causality orientation keeps only pairs where one entity directly contacts and maintains the other’s state, and removing the first would change that state.Source: Causality Orientation · p. 23

E2E Visual Causal Discovery

Related papers

Loading 2512.01827v1…