PaperlayerView exact revision on arXiv

Source-linked AI summary

Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering

Yang Liu, Guanbin Li, Liang Lin

arXiv:2207.12647v8cs.CVcs.AI

TL;DR

Event-level visual question answering must handle temporality, causality, and dynamics while avoiding spurious visual-linguistic correlations. CMCIR uses cross-modal causal interventions and specialized reasoning modules, achieving effective and robust performance across four event-level datasets.

  • Problem

    Existing event-level visual question answering methods may rely on spurious visual or linguistic correlations and oversimplify reasoning over complex temporal and causal events.

  • Method

    CMCIR uses front-door and back-door causal interventions within causality-aware visual-linguistic reasoning, alongside spatial-temporal interaction modeling and adaptive feature fusion.

  • Results

    Experiments on SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA demonstrate CMCIR’s effectiveness in discovering visual-linguistic causal structures and achieving robust event-level visual question answering.

  • Takeaways & Limitations

    CMCIR provides a cross-modal causal reasoning framework for event-level visual question answering rather than eliminating only visual or linguistic bias.

  • Takeaways & Limitations

    The paper identifies future needs for more comprehensive causal discovery, especially for question-critical temporal scene elements, and quantitative analysis of spurious correlations.

Abstract

from arXiv · show

Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes that fail to capture event temporality, causality, and dynamics spanning over the video. In this work, to address the task of event-level visual question answering, we propose a framework for cross-modal causal relational reasoning. In particular, a set of causal intervention operations is introduced to discover the underlying causal structures across visual and linguistic modalities. Our framework, named Cross-Modal Causal RelatIonal Reasoning (CMCIR), involves three modules: i) Causality-aware Visual-Linguistic Reasoning (CVLR) module for collaboratively disentangling the visual and linguistic spurious correlations via front-door and back-door causal interventions; ii) Spatial-Temporal Transformer (STT) module for capturing the fine-grained interactions between visual and linguistic semantics; iii) Visual-Linguistic Feature Fusion (VLFF) module for learning the global semantic-aware visual-linguistic representations adaptively. Extensive experiments on four event-level datasets demonstrate the superiority of our CMCIR in discovering visual-linguistic causal structures and achieving robust event-level visual question answering. The datasets, code, and models are available at https://github.com/HCPLab-SYSU/CMCIR.

1 INTRODUCTION

Event-level VQA requires reasoning over temporal, causal, spatial, and linguistic dependencies rather than relying on visual content correlations. CMCIR addresses these challenges by discovering cross-modal causal structures through intervention-based reasoning and adaptive feature fusion.

  • Motivation: The framework targets event-level reasoning because correlating visual contents alone cannot recover hidden spatial-temporal and causal dependencies needed for counterfactual inference.Existing VQA methods often focus on simple events and overlook deeper causality, temporal relations, and linguistic interactions.
  • Contribution: CMCIR’s CVLR module mitigates visual and linguistic confounder biases using linguistic back-door intervention and visual front-door intervention through LGCAM.LGCAM aggregates local and global visual representations, while linguistic semantic relations guide intervention within the linguistic modality.
  • Contribution: CMCIR is proposed as a causality-aware framework for discovering true cross-modal causal structures and improving robustness in event-level visual question answering.The paper identifies this as the first effort to discover cross-modal causal structures for event-level VQA.
  • Contribution: The Spatial-Temporal Transformer models fine-grained multimodal co-occurrences among linguistic semantics, spatial representations, and temporal representations.It is designed to capture interactions between visual and linguistic knowledge across spatial and temporal dimensions.
  • Contribution: The Visual-Linguistic Feature Fusion module adaptively combines causality-aware features into global semantic-aware representations using hierarchical linguistic semantic relations.Experiments on SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA show CMCIR’s effectiveness for causal-structure discovery and event-level VQA.

2 RELATED WORKS

Event-level VideoQA is more challenging than ImageQA because it requires spatial-temporal and visual-linguistic relational reasoning. Prior causal visual-question-answering methods address individual modality biases, leaving cross-modal causality in complex event-level video understanding insufficiently explored.

  • VideoQA: Event-level VideoQA extends ImageQA with temporal complexity, requiring spatial-temporal and visual-linguistic relations to infer answers.Prior VideoQA relational reasoning includes attention mechanisms guided by questions to exploit appearance and motion knowledge.
  • Event-Level VideoQA: CMCIR targets complex event-level VideoQA by performing cross-modal causal relational reasoning over spatial-temporal and linguistic content.The framework differs from Eclipse, which focuses on computation-efficient and dynamic reasoning for urban traffic events, by aiming to uncover causal structures behind visual-linguistic modalities.
  • Relational Reasoning: Relational reasoning has also supported action recognition and spatial-temporal grounding through spatiotemporal aggregation and dynamic object-interaction modeling,,,,,.These approaches model event relations beyond visual question answering, including person-centered spatiotemporal context and object-object interactions.
  • Causal Inference: Causal inference can mitigate spurious correlations and disentangle model effects for better generalization across vision and vision-language tasks,,,,.Counterfactual and causal methods have been applied to visual explanations, scene graphs, image recognition, video analysis, and vision-language tasks,,,,,,,,,,.
  • Causal Inference: Existing causal VQA methods eliminate either linguistic or visual bias with structured causal models, but do not discover cross-modal causality in challenging video or event-level settings,,, [59].Most causal visual tasks remain relatively simple, while recent VQA methods such as CVL, Counterfactual VQA, CATT, IGV, and VCD address only one modality’s bias.

3 METHODOLOGY

CMCIR is an event-level VideoQA architecture that models video-question answering with Pearl’s structural causal model to remove visual-linguistic confounding. It combines causal interventions with semantic parsing, visual feature extraction, and fine-grained spatial-temporal reasoning.

  • Visual Representation: The framework divides each L-frame video into N equal clips and represents each clip with frame-wise appearance and clip-level motion features mapped into a shared d-dimensional space.Swin-L extracts appearance features and Video Swin-B extracts motion features before linear feature transformation.
  • Causality-aware Visual-Linguistic Reasoning: CMCIR models video-question answering as the causal process {V, Q} → X → A and uses do(V, Q) to block confounder-induced back-door paths and learn the true causal effect.Visual and linguistic confounders can create spurious correlations between videos, words, and answers; intervention cuts the link from {V, Q} to their parents.
  • Causality-aware Visual-Linguistic Reasoning: The causal visual features are enhanced by a Local-Global Causal Attention Module that jointly estimates mediator and visual representations from local-local and local-global visual features.The module increases the representation ability of causality-aware visual features through unified attention.
  • Spatial-Temporal Transformer: The Spatial-Temporal Transformer captures multi-level, fine-grained interactions between question semantics and appearance-motion information through QA, QM, AS, and MS submodules.These submodules model question-appearance, question-motion, appearance-semantics, and motion-semantics interactions for event-level reasoning.

4 EXPERIMENTS

The experiments evaluate CMCIR against state-of-the-art methods and examine its components, hyper-parameter sensitivity, and causal-reasoning visualizations.

  • 4 EXPERIMENTS: CMCIR is evaluated against state-of-the-art methods to assess its performance.
  • 4 EXPERIMENTS: Ablation studies assess the effectiveness of CMCIR and its components, while parameter sensitivity analysis examines hyper-parameter effects.
  • 4 EXPERIMENTS: Visualization analysis is used to validate CMCIR’s ability to perform causal reasoning.

4.1 Datasets

CMCIR is evaluated on the event-level SUTD-TrafficQA dataset and three real-world benchmarks: TGIF-QA, MSVD-QA, and MSRVTT-QA. SUTD-TrafficQA supports six reasoning tasks spanning understanding, forecasting, reverse and counterfactual inference, introspection, and attribution.

  • Dataset overview: CMCIR is evaluated on SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA.SUTD-TrafficQA is an event-level urban dataset, while the other three are benchmark real-world datasets.
  • SUTD-TrafficQA: SUTD-TrafficQA contains 62,535 QA pairs and 10,090 traffic-scene videos across six challenging reasoning tasks.The tasks are basic understanding, event forecasting, reverse reasoning, counterfactual inference, introspection, and attribution analysis.
  • TGIF-QA: TGIF-QA contains 165K QA pairs from 72K animated GIFs and covers repetition count, repeating action, state transition, and frame QA.Repetition count is a counting task; repeating action and state transition are multiple-choice tasks, while FrameQA is open-ended with a predefined answer set.
  • MSVD-QA: MSVD-QA contains 50,505 algorithm-generated QA pairs and 1,970 approximately 10-second trimmed clips spanning What, Who, How, When, and Where questions.The dataset is created from the Microsoft Research Video Description Corpus.
  • MSRVTT-QA: MSRVTT-QA contains 10,000 approximately 15-second trimmed clips and 243,680 automatically generated QA pairs across What, Who, How, When, and Where question types.The dataset contains more complex scenes constructed from MSRVTT.

4.2 Implementation Details

The implementation standardizes video segmentation across datasets and extracts appearance, motion, and linguistic features using pretrained models.

  • Implementation Details: Videos are divided into 8 clips for SUTD-TrafficQA and TGIF-QA, and 24 clips for the longer MSVD-QA and MSRVTT-QA videos, following.
  • Implementation Details: Frame-level appearance features come from ImageNet-22K-pretrained Swin-L, while clip-level motion features use Kinetics-600-pretrained video Swin-B.
  • Implementation Details: Question representations use pretrained 300-dimensional GloVe word features.

4.3 Comparison With State-of-the-Art Methods

CMCIR achieves state-of-the-art performance across six SUTD-TrafficQA reasoning tasks and three additional event-level benchmarks. Its gains are strongest on overall accuracy and most question types, while How and Where remain affected by dataset bias.

  • 4.3 Comparison With State-of-the-Art Methods: CMCIR achieves the best performance across six SUTD-TrafficQA reasoning tasks, improving Eclipse by 1.53% overall.The tasks are basic understanding, event forecasting, reverse reasoning, counterfactual inference, introspection, and attribution analysis.
  • 4.3 Comparison With State-of-the-Art Methods: CMCIR achieves the best performance on TGIF-QA Action and FrameQA, with relatively high performance on Transition and Count.The comparison is reported in Table 5.
  • 4.3 Comparison With State-of-the-Art Methods: 43.7% overall accuracy on MSVD-QA gives CMCIR a 4.7% improvement over DualVGR, while it leads on What, Who, and When.CMCIR outperforms nearly all comparison methods by a significant margin, although GMIN and CASSG perform marginally better on some types.
  • 4.3 Comparison With State-of-the-Art Methods: 38.9% accuracy makes CMCIR the best method on MSRVTT-QA, leading on What, Who, and When despite ASTG being the strongest comparator.CASSG performs better on How and GMIN on Where.
  • 4.3 Comparison With State-of-the-Art Methods: CMCIR improves by +0.3%∼0.9% on TGIF-QA and +1.3% on MSRVTT-QA, while How and Where degrade under data bias and limited samples.The reported gains are +1.53% on SUTD-TrafficQA and +4.7% on MSVD-QA; results across Tables 5–7 support generalization across urban traffic and real-world scenes.

4.4 Ablation Studies

Ablation experiments evaluate CMCIR’s designed components across four event-level VQA datasets. CMCIR outperforms all six variants, while removing HSRP, either causal intervention, SGE, or ALFF reduces performance.

  • 4.4 Ablation Studies: CMCIR achieves the best performance across all datasets and tasks compared with six ablated variants.The evaluation covers SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA.
  • 4.4 Ablation Studies: Removing HSRP significantly reduces performance, showing that hierarchical semantic-role parsing strengthens question-semantic representation.Without HSRP, the model uses only the original question as its linguistic representation.
  • 4.4 Ablation Studies: Removing either LBCI or VFCI lowers performance, indicating that both linguistic back-door and visual front-door interventions help discover causal structures and learn causality-aware representations.The w/o LBCI variant retains only VFCI, whereas w/o VFCI retains only LBCI.
  • 4.4 Ablation Studies: Removing SGE or ALFF also lowers performance relative to CMCIR, although both variants outperform the intervention- and CVLR-removal variants.SGE supports semantic graph embedding, while ALFF adaptively fuses linguistic and visual features.

4.5 Parameter Sensitivity

This section analyzes CMCIR’s sensitivity to key architectural hyperparameters across four event-level QA datasets and tests generalization across visual appearance and motion features on three datasets. Using ResNet-101 and ResNetXt-101 achieves 40.3% on MSVD-QA and 37.0% on MSRVTT-QA, with competitive or superior comparison performance as reported.

  • Parameter Sensitivity: CMCIR sensitivity is evaluated for MMA heads, MTB layers, GCN layers, and hidden-state dimension across SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA.The results are presented in Table 9.
  • Feature Generalization: CMCIR generalization is evaluated with different visual appearance and motion features on SUTD-TrafficQA, MSVD-QA, and MSRVTT-QA.The comparisons are presented in Table 10.
  • Feature Generalization: 40.3% with ResNet-101 and ResNetXt-101 on MSVD-QA outperforms other comparison methods.
  • Feature Generalization: 37.0% with ResNet-101 and ResNetXt-101 on MSRVTT-QA achieves competitive performance compared with other methods.

4.6 The Evidence of Reducing Spurious Correlations

The section evaluates CVLR as an orthogonal module for reducing spurious correlations in non-causal VideoQA models. Applied to Co-Mem, HGA, and HCRN, CVLR yields especially favorable gains on descriptive-question benchmarks, while fine-grained visual-linguistic interactions strengthen the proposed backbone’s robustness.

  • Experimental Setup: CVLR is applied to the non-causal VideoQA models Co-Mem, HGA, and HCRN to test its ability to reduce spurious correlations.The evaluation assumes that VideoQA datasets contain spurious correlations introduced during dataset construction and treats CVLR as orthogonal to the backbone.
  • Results: On MSVD-QA and MSRVTT-QA, CVLR gains are more favorable with Co-Mem, HGA, and HCRN than with the proposed backbone.The proposed backbone’s fine-grained interactions between linguistic semantics and spatial-temporal representations provide greater robustness, particularly for descriptive questions.
  • Results: The proposed backbone achieves stronger performance on benchmarks focused on descriptive questions, including MSVD-QA and MSRVTT-QA.Its advantage is attributed to fine-grained interactions between linguistic semantics and spatial-temporal representations.

4.7 Qualitative Results

Figure 9 qualitatively shows that CMCIR performs robust spatial-temporal visual-linguistic causal reasoning by focusing on question-relevant regions and reducing spurious correlations. It also generalizes across question types, handles introspective and counterfactual reasoning, and reveals failure cases involving visual ambiguity.

  • Generalization ability: CMCIR generalizes across question types by remaining sensitive to questions and capturing dominant spatial-temporal video content.Figure 9(a)-(b) illustrates robust and reliable spatial-temporal relational reasoning across different question types.
  • Reliable reasoning: CMCIR focuses on question-relevant visual regions despite distracting dominant scenes, mitigating ambiguity in video-question-answer reasoning.Causal relational learning helps identify the correct spatial-temporal content when dominant scenes correlate strongly with question semantics.
  • Introspective and counterfactual learning: For introspective and counterfactual questions, CMCIR determines whether attended scenes reflect answer logic while exploring causal, logical, and spatial-temporal structures.This capability is attributed to robust visual-linguistic causal reasoning that disentangles visual-linguistic spurious correlations.
  • Additional failure cases: Failure cases arise when CMCIR confuses “suv” with a green “traffic plate” or cannot distinguish visually similar “rainy” and “snowy” conditions.The first error reflects resemblance between the traffic plate and truck despite only a white suv appearing; the second reflects similar video appearances.

5 CONCLUSION

CMCIR addresses cross-modal spurious correlations in event-level visual question answering by discovering visual-linguistic causal structures through front-door and back-door interventions. Experiments across four datasets demonstrate its effectiveness and robustness, while future work targets temporal causal discovery.

  • Conclusion: Extensive experiments on SUTD-TrafficQA, TGIF-QA, MSVD-QA, and MSRVTT-QA demonstrate CMCIR’s effectiveness in robust event-level visual question answering.The evaluation covers one event-level urban dataset and three benchmark real-world datasets.
  • Conclusion: CMCIR applies front-door and back-door causal intervention modules to discover cross-modal causal structures rather than eliminating only linguistic or visual bias.This design targets spurious correlations across visual and linguistic modalities.
  • Conclusion: Future work will pursue comprehensive causal discovery of question-critical scene elements, especially through finer-grained temporal consistency in videos.The stated goal is to achieve better causality in event-level visual question answering.
Loading 2207.12647v8…