PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal Reasoning Meets Visual Representation Learning: A Prospective Study

Yang Liu, Yushen Wei, Hong Yan, Guanbin Li, Liang Lin

arXiv:2204.12037v8cs.CVcs.AIcs.LGcs.MM

TL;DR

Visual representation learning methods often fit data distributions and capture spurious correlations rather than essential causal relations, raising challenges for generalization and cognitive ability. This paper surveys causal reasoning for visual representation learning, including foundational concepts, methods, and datasets, and discusses future challenges and research directions. It concludes that the field remains at an early stage, with limitations in task coverage, evaluation benchmarks, and multimodal causal analysis.

  • Problem

    Visual learning methods often fit data distributions and capture spurious correlations while neglecting essential causal relations in multimodal knowledge, alongside concerns about generalization and cognitive abilities.

    Source: §1 · p. 1

  • Method

    The paper systematically reviews causal reasoning for visual representation learning, presents foundational causal concepts, and discusses related works, datasets, challenges, and research directions.

    Source: §1 · p. 2

  • Results

    The survey synthesizes causal visual representation learning methods, datasets, challenges, and future directions, and identifies the area as still in its infancy.

    Source: §5.4 · p. 13, §9 · p. 20

  • Takeaways & Limitations

    The paper proposes directions including improved causal relation modeling, intervention-distribution approximation, counterfactual synthesis, and large-scale benchmarks with evaluation pipelines.

    Source: §8 · p. 19

    Current work covers only several computer-vision tasks, lacks causal-reasoning-specific large-scale benchmarks and pipelines, and often studies visual or linguistic causality separately.

    Source: §5.4 · pp. 13–14

Abstract

from arXiv · show

Visual representation learning is ubiquitous in various real-world applications, including visual comprehension, video understanding, multi-modal analysis, human-computer interaction, and urban computing. Due to the emergence of huge amounts of multi-modal heterogeneous spatial/temporal/spatial-temporal data in big data era, the lack of interpretability, robustness, and out-of-distribution generalization are becoming the challenges of the existing visual models. The majority of the existing methods tend to fit the original data/variable distributions and ignore the essential causal relations behind the multi-modal knowledge, which lacks unified guidance and analysis about why modern visual representation learning methods easily collapse into data bias and have limited generalization and cognitive abilities. Inspired by the strong inference ability of human-level agents, recent years have therefore witnessed great effort in developing causal reasoning paradigms to realize robust representation and model learning with good cognitive ability. In this paper, we conduct a comprehensive review of existing causal reasoning methods for visual representation learning, covering fundamental theories, models, and datasets. The limitations of current methods and datasets are also discussed. Moreover, we propose some prospective challenges, opportunities, and future research directions for benchmarking causal reasoning algorithms in visual representation learning. This paper aims to provide a comprehensive overview of this emerging field, attract attention, encourage discussions, bring to the forefront the urgency of developing novel causal reasoning methods, publicly available benchmarks, and consensus-building standards for reliable visual representation learning and related real-world applications more efficiently.

1 Introduction

Visual learning methods often fit data distributions and capture spurious correlations rather than essential causal relations, limiting generalization and cognitive ability. This paper surveys causal reasoning for visual representation learning, its foundations and applications, and future directions.

1 Introduction

  • The paper’s overview spans heterogeneous modalities—including images, videos, text, audio, and multisensor data—and tasks such as visual comprehension and video understanding.The introduction also lists visual-linguistic analysis and multimodal fusion among example tasks.Source: §1 · p. 1
  • Existing methods rely heavily on fitting data distributions and may capture spurious correlations instead of essential causal relations in multimodal knowledge.The paper connects this concern to limits in generalization and cognitive abilities.Source: §1 · p. 1
  • The paper offers a systematic survey focused on causal reasoning’s relation to visual representation learning, alongside related works, datasets, insights, challenges, and opportunities.It distinguishes this focus from earlier surveys of general representation-learning tasks.Source: §1 · p. 2
  • The paper presents causality concepts, structural causal models, the independent causal mechanism principle, causal inference, and causal intervention.It also proposes potential research directions for causal visual representation learning, described as first to the authors’ knowledge.Source: §1 · p. 2
  • The review systematically organizes existing work around efforts to apply causal reasoning to visual representation learning and aims to clarify why and how these methods may help.It also aims to inspire future research and studies.Source: §1 · p. 2

2 Preliminaries

Causal reasoning distinguishes observing associations from asking what an intervention would change or what would have happened under an unrealized condition. The paper presents these as three levels: association, intervention, and counterfactual.

2.1 Causal Learning and Reasoning

  • Association concerns statistical relationships between variables, while causal learning aims to discover causal relationships beyond those statistical relations.The paper notes that statistical learning can perform well with sufficient i.i.d. data but may perform poorly when the i.i.d. assumptions do not hold.Source: §2.1 · p. 3
  • Intervention asks about an outcome after a specified treatment, which cannot be answered from data associations alone.The paper contrasts this with simply observing that people who go to the gym may not be stronger than professional athletes.Source: §2.1 · p. 3
  • Counterfactual reasoning compares outcomes under the same condition when the question’s antecedent is unrealized.Its typical form asks what would have happened under a condition that did not occur.Source: §2.1 · p. 3

2.2 Structural Causal Models

Structural causal models represent variables as nodes in a causal graph and express each variable through a function of its causes. The graph and joint distribution together represent causal relations and effects.

2.2 Structural Causal Models

  • A structural causal model represents variables as vertices in a directed acyclic graph describing their causal relations.Each variable is written as the outcome of a function.Source: §2.2 · p. 3
  • Each variable’s structural function depends on its graph parents and unmeasured factors such as noise.The function gives a mathematical form for the effect of the variable’s direct causes.Source: §2.2 · p. 3
  • After decomposition and graphical modeling, the causal graph and joint distribution represent a dataset’s causal relations and effects.The paper calls the decomposition of joint distributions a product decomposition.Source: §2.2 · p. 3

2.3 Independent Causal Mechanism

The independent causal mechanism principle treats a system’s causal generative process as autonomous modules. Changing or learning about one mechanism does not affect or inform the others, supporting the possibility of transfer across domains with shared modules.

2.3 Independent Causal Mechanism

  • The independent causal mechanism principle treats a system’s causal generative process as autonomous modules that do not inform or influence each other.In probabilistic terms, each variable’s conditional distribution given its causes is its mechanism.Source: §2.3 · p. 4
  • Changing one mechanism does not change any other mechanism.The paper states this independence for interventions on conditional distributions given their causes.Source: §2.3 · p. 4
  • Knowing one mechanism does not provide information about another, and this independence suggests knowledge can transfer across domains with the same modules.The paper relates this transfer possibility to interventions that leave other mechanisms unaffected.Source: §2.3 · p. 4

2.4 Causal Inference

Causal inference estimates how treatments shift outcomes, including outcomes that cannot be observed under the treatment not taken. It uses observational data to estimate these effects, though such data are often incomplete in real-world settings.

2.4 Causal Inference

  • Causal inference estimates the outcome shift or effect of different treatments.The paper illustrates treatment with medicine and recovery as the outcome.Source: §2.4 · p. 4
  • A counterfactual outcome is the potential outcome of an action that was not taken.If treatment A = 1 is taken, Y(A = 0) is counterfactual.Source: §2.4 · p. 4
  • Causal inference estimates treatment effects from observational data, which may be incomplete because of cost and moral concerns.When treatment is applied, the no-treatment outcome may not be available.Source: §2.4 · p. 4

2.5 Causal Intervention

Causal intervention estimates effects by modifying a graphical model and manipulating conditional probabilities. Back-door adjustment uses an observed confounder, while front-door adjustment can identify effects when that confounder is unobserved and an observed mediator is available.

2.5 Causal Intervention

  • Causal intervention modifies the graphical model and manipulates conditional probabilities to capture intervention effects and use dataset causal relations to improve model performance and generalization.The section reviews back-door and front-door adjustment strategies.Source: §2.5 · p. 4

2.5.1 Back-door Adjustment

  • Back-door adjustment blocks the path X ← Z → Y by deleting Z → X, replacing P(z|X) with P(z) to estimate P(Y|do(X)).Fig. 2 shows the edge deletion used to block the back-door path.Source: §2.5.1 · pp. 4–5, Figure 2 · p. 5

2.5.2 Front-door Adjustment

  • Front-door adjustment estimates the total effect by summing over mediator W after applying the adjustment formulation.The formulation follows from applying the preceding equations to the effects through W.Source: §2.5.2 · p. 5
  • Front-door adjustment can bypass an unobserved confounder Z when an observed mediator W lies on the path X → W → Y.It applies the do-operator at W and X; Fig. 3 depicts the case where Z is unobserved.Source: Figure 3 · p. 5, §2.5.2 · p. 5

2.5.3 Back-door or Front-door?

  • Back-door adjustment requires identifying the confounder in advance, which is difficult when visual or linguistic confounders are not observable or disentangled.The paper presents front-door intervention as a feasible way to calculate P(Y|do(X)) in this setting.Source: §2.5.3 · p. 5

3 Causality-aware Feature Learning

The section reviews three causal feature-learning paradigms: embedding structural causal models, applying interventions or counterfactual inference, and selecting features using Markov boundaries. It describes their approaches and notes that much existing work targets general rather than specifically visual feature learning.

3 Causality-aware Feature Learning

  • Structural causal model embedding includes RELIC [98] for self-supervised learning and DEAR [99] for causal controllable generation.RELIC achieved competitive robustness and out-of-distribution generalization on ImageNet; DEAR was evaluated on synthesized and real datasets.Source: §3 · p. 6
  • Intervention- and counterfactual-based feature learning includes methods for deconfounded visual grounding, weakly supervised segmentation, long-tail classification, and visual context learning.The cited examples are Huang et al. [64], Zhang et al., Tang et al., and Chen et al..Source: §3 · p. 6
  • Markov-boundary feature selection seeks a subset of predictive features, using local causal relations to identify features associated with a class variable.The section discusses approaches addressing incorrect conditional-independence tests, multiple interventional datasets, and missing data.Source: §3 · p. 6
  • Causal feature selection can improve explanation and robustness, but most existing methods address general feature learning rather than visual representation learning specifically.The paper connects causal relations and the ICM principle with transferring the class-variable mechanism across settings or environments.Source: §3 · p. 6

4 Visual Representation Learning: State-of-the-art

This section surveys representative visual-learning tasks—including object detection, scene graph generation, visual grounding, visual commonsense reasoning, action detection and recognition, and visual question answering—and discusses their task-specific challenges.

Object detection

  • Object detection follows either a region-proposal pipeline or a unified framework that directly predicts categories and locations.Video salient-object detection extends image methods with temporal cues, including optical flow and sequential feature evolution.Source: §4 · p. 7

Video salient-object detection

  • Non-objects, occlusion, motion blur, and lens movement make it difficult for video salient-object detectors to focus on the target area.Source: §4 · p. 7

Scene graph generation

  • Scene graph generation builds graphs of objects and relations, but relationship predictions face conditional-distribution bias and methods remain far from practical knowledge bases.Approaches use facts alone or add priors; many use GNNs or relevant prior information.Source: §4 · pp. 7–8

Visual grounding

  • Visual grounding locates image targets from descriptions, with fully supervised, weakly supervised, and unsupervised approaches, but dataset-specific visual and linguistic biases limit transfer and generalization.Source: §4 · p. 8

Visual commonsense reasoning

  • Visual commonsense reasoning has question-answering and answer-justification subtasks, with relatively few studies using specialized architectures such as R2C, CCN, and HGL.Source: §4 · p. 8
  • VCR performance is generally attributed to large pretrained models and external knowledge, while retrieval, integration, and interpretable reasoning remain challenges.Source: §4 · p. 9

Action detection and recognition

  • Action detection and recognition methods use CNNs, video transformers, or skeleton-based models, yet scene and object reliance can impede transfer across datasets.Models may focus on static appearance or background rather than true motion.Source: §4 · p. 9

Visual question answering

  • VQA models use attention to connect visual and linguistic information, while VideoQA must additionally reason over temporal relations and remains vulnerable to modality bias and spurious correlations.Source: §4 · pp. 9–10

5 Causality-aware Visual Representation Learning

The section reviews causal visual representation learning across visual understanding, robustness, and visual question answering, then discusses the field’s current scope and open challenges. It presents causal reasoning as an approach to addressing challenges including limited interpretability, poor generalization, and reliance on data correlations.

5 Causality-aware Visual Representation Learning

  • Causal visual representation learning is organized around visual understanding, visual robustness, and visual question answering, an emerging research topic with work appearing since the 2020s.The section frames causal reasoning as a potential alternative to correlation reliance and poor generalization.Source: §5 · p. 10

5.1 Causal Visual Understanding

  • Causal visual understanding methods use interventions and counterfactual inference to reduce confounding and bias across recognition, grounding, segmentation, and scene-graph tasks.Examples include adversarial training that strengthens mediator effects while suppressing confounder effects, and back-door adjustment to remove confounders.Source: §5.1 · pp. 10–11

5.2 Causal Visual Robustness

  • Causal robustness methods address confounding in few-shot learning, adversarial defense, class-incremental learning, and cross-domain transfer, including through back-door adjustment and counterfactual samples.Other approaches steer generative models to produce intervened distributions or use representations to simulate causal factors.Source: §5.2 · p. 12

5.3 Causal Visual Question Answering

  • Causal visual question answering methods target language and multimodal bias using confounder sets, front-door adjustment, counterfactual QA pairs, and causal-effect-based inference.The motivating example is a model predicting “red” for a green apple based on the question–answer correlation; balancing data may only partly mitigate the bias.Source: §5.3 · pp. 12–13

5.4 Comparisons and discussions

  • The field remains limited to several computer-vision tasks, lacks purpose-built large-scale causal-reasoning benchmarks and pipelines, and often studies visual or linguistic causality separately.The authors identify applying causal knowledge to prompt learning and analyzing the relation between causal reasoning and visual representation learning as directions for further work.Source: §5.4 · pp. 13–14

6 Related Causal Datasets

This section reviews image and video question-answering datasets intended to assess causal reasoning. It notes that superficial dataset correlations can permit high scores without demonstrating reasoning, and identifies the need for benchmarks designed to evaluate causal reasoning directly.

6 Related Causal Datasets

  • Existing QA datasets can reward superficial correlations rather than reasoning, motivating benchmarks that evaluate models’ causal-reasoning capability.The paper gives VQA v1.0 as an example: answering “yes” to “Do you see a...” can achieve nearly 90% accuracy.Source: §6 · p. 14

6.1 Image Question Answering

  • Image-QA datasets include complementary or re-split QA pairs, semantically edited images, and human–machine adversarial collection, but large-scale benchmarks for fair causal-reasoning evaluation remain lacking.The authors suggest incorporating causal concepts such as confounders and interventions when building benchmarks.Source: §6.1 · p. 15

6.2 Video Question Answering

  • Video-QA benchmarks span diagnostic synthetic datasets and realistic videos, with tasks involving counterfactuals, physical properties, event forecasting, and causal and temporal action reasoning.CLEVRER includes diagnostic annotations and counterfactuals; SUTD-TrafficQA includes six reasoning tasks, while NExT-QA targets causal and temporal actions in daily activities.Source: §6.2 · pp. 15–17

7 Extensive Applications

The section surveys potential applications of causal reasoning in image and video analysis, explainable AI, recommendation, human–computer interaction, and crowd intelligence. It describes how these applications address data bias, interpretability, and heterogeneous spatial-temporal information.

Image/video analysis

  • In image and video analysis, causal intervention is proposed to reduce observation bias and learn features robust to different data distributions.The section also identifies weakly supervised object localization, detection, and grounding as vulnerable to ambiguous, confounded associations.Source: §7 · p. 18

Explainable artificial intelligence

  • Causal interpretability work is presented as a route to explaining machine decisions, with mediation analysis used to disentangle effects contributing to COVID-19 case fatality rates.Explainable treatment-effect estimation is described as a way to estimate drug effects and inform prescribing.Source: §7 · p. 18

Recommendation system

  • Recommendation systems trained on biased samples may fail to reflect users’ mental states and emotions; counterfactual mirror users and unbiased similarity measures are described for alleviating survival bias.The stated goals include personalized and reliable recommendations and reducing dissatisfaction and low user activity.Source: §7 · p. 18

Human-computer dialog and interaction

  • For human–computer interaction, causal reasoning is presented as a way to reduce reliance on superficial correlations and model heterogeneous spatial-temporal information for reliable, explainable decisions.In robotics, prior knowledge is proposed as a candidate causal structure when relevant environmental features are not known in advance.Source: §7 · pp. 18–19

Crowd intelligence analysis

  • Crowd intelligence analysis covers multiple subjects and their interactions; urban computing uses heterogeneous traffic and geographic data to address congestion, energy consumption, and pollution.The section notes that such data contains strong causal relations and argues that discovering complex, entangled relations is beneficial to urban computing and crowd intelligence analysis.Source: §7 · p. 19

8 More Detailed Discussions

Causal reasoning for visual representation learning remains in its infancy, with open directions spanning causal modeling, intervention approximation, counterfactual synthesis, and evaluation benchmarks.

8 More Detailed Discussions

  • The authors identify four future directions: more reasonable causal relation modeling, more precise intervention-distribution approximation, more proper counterfactual synthesis, and large-scale benchmarks with evaluation pipelines.They describe the field as still in its infancy, with many issues unsolved.Source: §8 · p. 19

8.1 More Reasonable Causal Relation Modeling

  • Real-world visual data are unstructured, causal effects may be unobserved, and VQA methods differ in whether visual and language features are modeled jointly or separately.The cited methods focus on causal discovery in either the visual or linguistic modality without considering both; the authors call for deeper analysis and more comprehensive causal relations.Source: §8.1 · p. 19

8.2 More Precise Approximation of Intervention Distributions

  • Current intervention approximations often define confounders as average object features, which may poorly represent confounders in complex heterogeneous visual data.The authors identify more accurate confounder approximation as a key direction for causal intervention methods.Source: §8.2 · p. 20

8.3 More Proper Counterfactual Synthesising Process

  • Counterfactual methods commonly embed inference in training, but entangled, heterogeneous visual data make bias difficult to understand and model.The authors identify proper counterfactual synthesis as a potential direction for debiasing visual representations.Source: §8.3 · p. 20

8.4 Large-scale Benchmarks and Evaluation Pipeline

  • Existing evaluation datasets are largely traditional correlation-learning datasets, while causal datasets discussed in the paper are limited in scale and lack comprehensive evaluation standards.The authors call for large-scale benchmarks and pipelines tailored to specific visual representation learning tasks.Source: §8.4 · p. 20

9 Conclusions

The paper surveys causal reasoning for visual representation learning, covering foundational concepts, tasks, related works, and datasets. It also discusses challenges and future directions for further exploration.

9 Conclusions

  • The survey presents basic concepts including causality, structural causal models, independent causal mechanisms, causal inference, and causal intervention.Source: §9 · p. 20
  • It reviews visual understanding, action detection and recognition, and visual question answering, alongside related causal visual-learning works and datasets.Source: §9 · p. 20
  • The paper discusses existing challenges and potential future directions, and expresses the aim of encouraging work on causal methods, public benchmarks, and consensus-building standards.Source: §9 · p. 20
Loading 2204.12037v8…