PaperlayerView exact revision on arXiv

Source-linked AI summary

Causal-EVC: Breaking Emotional Spurious Causality via Spatiotemporal Grounding and Counterfactual Intervention

Cheng Ye, Weidong Chen, Peipei Song, Zhendong Mao

arXiv:2609.36776v1cs.CV

TL;DR

Emotional video captioning methods can rely on background correlations, and existing evaluations do not establish whether models reason from genuine emotional causes. This paper constructs EVC-CauseGround and proposes Causal-EVC, which combines cause localization with counterfactual emotion reasoning. Causal-EVC reports improvements across caption-generation and cause-grounding metrics, including on the causal-faithfulness evaluation.

  • Problem

    Existing emotional video captioning methods can exploit background confounders, while current benchmarks do not quantify genuine causal reasoning.

    Source: §1 · pp. 1–2

  • Method

    The paper builds EVC-CauseGround with dense causal annotations and proposes Causal-EVC, combining motion-guided localization with counterfactual emotion deduction.

    Source: §1 · p. 2

  • Results

    Causal-EVC improves caption-generation and cause-grounding performance across reported benchmarks, including +17.3% mIoU and +11.8% emotion accuracy over zero-shot Qwen-2.5-VL-7B.

    Source: §1 · pp. 2–3, §4.1 · p. 7

  • Takeaways & Limitations

    The Causal-Faithfulness Subset evaluates whether models perform genuine causal reasoning or exploit contextual confounders.

    Source: §1 · p. 2

Abstract

from arXiv · show

Emotional Video Captioning aims to generate factually accurate and emotionally empathetic descriptions. While recent methods have recognized the importance of visual causes to guide emotion perception and caption generation, they fundamentally rely on simple attention matching, which inevitably suffers from {causal redundancy and spurious correlations} in co-occurrence bias (e.g., misclassifying ``sadness'' as ``joy'' on a sunny beach), leading to severe shortcut learning from confusing backgrounds. Furthermore, existing evaluations fail to verify whether models have genuinely mastered causal reasoning or merely exploited background confounders. To address these limitations, we first construct {EVC-CauseGround}, a comprehensive benchmark with dense spatio-temporal causal annotations. Crucially, it introduces a carefully selected {Causal-Faithfulness Subset} to explicitly quantify genuine emotion-cause attribution. Second, we propose {Causal-EVC}, an emotion-grounding captioning framework, which introduces a Motion-guided Causal Spatiotemporal Localization module to precisely decouple causal triggers from background confounders. Besides, we introduce an Interpretable Sparse Emotion Routing module. By synthesizing counterfactual representations and formulating a novel counterfactual contrastive objective, we enforce the model to anchor its emotion predictions strictly on authentic causal triggers instead of confusing background. Extensive experiments show that Causal-EVC not only achieves the best performance on semantic metrics but also exhibits significant advantages in the causal-faithfulness subset, which demonstrates that our model could mine emotional cues from genuine visual causes and mitigate co-occurrence bias for interpretable multimodal emotion understanding.

1 INTRODUCTION

Existing emotional video captioning methods can rely on background cues rather than fine-grained emotional causes, while current benchmarks do not measure genuine causal reasoning. The paper introduces EVC-CauseGround and Causal-EVC to address these gaps, reporting gains in cause grounding and emotion accuracy over a zero-shot baseline.

1 INTRODUCTION

  • Existing methods can exploit backgrounds aligned with emotion labels or be misled by conflicting backgrounds, while existing benchmarks do not test genuine causal reasoning.The paper describes these patterns as causal redundancy and spurious correlations, respectively.Source: §1 · pp. 1–2
  • EVC-CauseGround adds dense spatiotemporal causal annotations and the ST-IoU metric to evaluate genuine emotion-cause attribution.The benchmark is introduced as a way to evaluate causal attribution capabilities.Source: §1 · p. 2
  • Causal-EVC combines motion-guided cause localization with counterfactual emotion deduction and objectives that constrain cause extraction and emotion mining.The stated aim is to direct emotion reasoning toward true triggers rather than spurious background correlations.Source: §1 · p. 2
  • +17.3% mIoU and +11.8% emotion accuracy over zero-shot Qwen-2.5-VL-7B are reported for Causal-EVC.The introduction presents these gains as evidence of improved cause grounding and emotional causal reasoning.Source: §1 · pp. 2–3

2 EVC-CAUSEGROUND: A VISUAL-CAUSE GROUNDING EMOTIONAL DESCRIPTIVE BENCHMARK

EVC-CauseGround extends emotional video descriptions with fine-grained spatiotemporal cause annotations and a curated subset for evaluating genuine causal reasoning. Its construction combines in-the-wild videos, automated cause annotation, and human verification.

2 EVC-CAUSEGROUND: A VISUAL-CAUSE GROUNDING EMOTIONAL DESCRIPTIVE BENCHMARK

  • EVC-CauseGround unifies fine-grained spatiotemporal causal tubes and a curated counterfactual subset for authentic emotional cause reasoning.The benchmark extends conventional emotion descriptions with fine-grained emotional-cause annotations.Source: §2 · p. 3
  • The benchmark uses 2,000 videos from EmVidCap and MAFW, split into 1,500 training and 500 evaluation samples.The collection includes 1,523 EmVidCap videos and 477 MAFW videos, with videos shorter than five seconds excluded.Source: §2 · p. 3
  • An MLLM-led, tracker-assisted pipeline samples 30 frames, estimates a causal window, identifies a peak-motion frame, and tracks the cause across that span.The causal span is constrained to at most 30% of the video frames.Source: §2 · p. 3
  • Human annotators check boundaries and use masked-background emotion guesses to label samples as Causal-Redundant or Causal-Faithfulness.The resulting Causal-Faithfulness Subset contains 176 videos for evaluating genuine reasoning versus contextual-confounder exploitation.Source: §2 · pp. 3–4
  • ST-IoU scores temporal overlap first and assigns zero when temporal IoU is zero, then evaluates spatial overlap on selected frames.The paper also reports mIoU, R@0.3, and R@0.5 for causal localization.Source: §2 · p. 4

3 CAUSAL-EVC: A CAUSE-GROUNDING VIDEO CAPTIONING FRAMEWORK

Causal-EVC encodes video and emotion information, then uses motion-guided causal localization and sparse emotion routing to ground emotional captions in visual causes. Counterfactual features and a contrastive objective encourage emotion mining from causal features rather than background shortcuts.

3 CAUSAL-EVC: A CAUSE-GROUNDING VIDEO CAPTIONING FRAMEWORK

  • Video frames and emotion words are encoded as semantic features that support cause mining and caption generation.The framework downsamples the video, extracts visual features with Qwen-2.5-VL, and uses a pretrained emotion-word embedding.Source: §3 · p. 4

3.1 MOTION-GUIDED CAUSAL SPATIOTEMPORAL LOCALIZATION

  • Adjacent-frame pixel similarities form a motion prior that separates temporal action peaks from moving regions and emphasizes movement peaks in temporal attention.A global motion baseline helps account for camera shake and perspective changes.Source: §3.1 · p. 5
  • The localization module combines motion with salient regions, predicts causal masks, and regresses normalized temporal boundaries and spatial coordinates.Spatial loss is calculated only on frames where the genuine causal action occurs.Source: §3.1 · pp. 5–6

3.2 INTERPRETABLE SPARSE EMOTION ROUTING

  • Sparse emotion routing uses VAD-enriched emotion prototypes and retains only each visual token’s top-K prototype matches.The visual cause mask filters features before they query the emotion prototypes.Source: §3.2 · p. 6
  • Counterfactual alignment contrasts emotion features from causal regions with features from their masked-background complement, while the decoder receives video, cause, and emotion features.An NCE loss encourages emotion features to reflect genuine causes rather than shortcut learning from backgrounds.Source: §3.2 · pp. 6–7

4 RESULTS AND DISCUSSION

Causal-EVC leads the reported captioning and cause-grounding comparisons. Ablations and visualizations attribute its results to motion-guided localization, sparse emotion routing, and counterfactual objectives.

4.1 MAIN RESULTS

  • On EVC-MSVD, Causal-EVC improves +4.2% Accsw over MM-ECPE and +15.7% BFS over SAGML; on EVC-VE, it improves +2.4% Accc and +3.0% BLEU-4 over SAGML.Source: §4.1 · p. 7
  • +67.2% R@0.3 over VTimeLLM and +52.1% mIoU over Gemini-2.5-Flash on EVC-CauseGround.The prose reports these comparisons against specialized visual-grounding models and general MLLMs, respectively.Source: Table 2 · p. 7, §4.1 · p. 7

4.2 ABLATION STUDIES

  • Combining MCSL and ISER yields the best ablation performance, with +14.7% Accsw and +12.8% CFS improvements.Individually, MCSL improves BLEU-4/CFS by +7.5%/+9.1%, while ISER improves Accsw by +13.2%.Source: §4.2 · p. 8, Table 5 · p. 8
  • Motion dynamics alone degrade MCSL performance, while motion-guided spatial and temporal localization produces a substantial performance surge.The authors attribute the difference to pixel-wise motion being semantically weak and vulnerable to environmental disturbance.Source: §4.2 · p. 8, Table 3 · p. 8
  • Top-K routing uses K = 10 as its default trade-off between affective expressiveness and noise suppression.K = 1 undermines fault tolerance and emotional diversity; K = 50 introduces redundant prototypes and background noise.Source: Table 6 · p. 8, §4.2 · p. 8
  • Lcf improves emotion accuracy through counterfactual contrastive learning, while Ltemp and Lspa improve semantic metrics through tube-supervised cause grounding.The combined objectives support end-to-end training for both emotional description and causal localization.Source: §4.2 · p. 8, Table 4 · p. 8

4.3 QUALITATIVE RESULTS

  • In the t-SNE visualization, Causal-EVC’s counterfactual representations are distributed across latent space, unlike MM-ECPE’s emotion-cluster-entangled representations.The authors interpret this as evidence that Lcf severs spurious environmental-context learning.Source: Figure 7 · p. 9, §4.3 · p. 9

5 CONCLUSION

The conclusion presents Causal-EVC as a cause-grounding captioning framework and EVC-CauseGround as a benchmark for authentic emotion-cause reasoning. Its modules target causal localization and background-driven shortcut learning, and experiments report strong text-generation and causal-faithfulness performance.

  • Causal-EVC achieves the paper’s best performance in both text generation and causal faithfulness.Source: §5 · p. 9
  • EVC-CauseGround combines dense spatio-temporal causal tubes with a curated Causal-Faithfulness Subset and a metric for emotion-cause reasoning.Source: §5 · p. 9
  • Motion-guided localization decouples foreground triggers into a 3D mask, while sparse prototype routing and a counterfactual contrastive objective penalize shortcut learning from background representations.Source: §5 · p. 9

AI USE STATEMENT

The paper discloses generative-AI assistance and reviews textual and multimodal emotion-cause extraction alongside emotional video captioning. It distinguishes its approach by focusing on visual causes of emotion.

AI USE STATEMENT

  • Generative AI assisted with language polishing, portions of experiment-code writing and debugging, and some dataset-construction steps; the authors say reported results and figures were produced by the authors and assisted outputs were tested and verified.Source: AI USE STATEMENT · p. 10

A.1 EMOTION-CAUSE PAIR EXTRACTION

  • Textual emotion-cause pair extraction research spans documents, articles, stories, microblogs, and conversations, with methods using graphs, causal discourse knowledge, and emotion-oriented prediction.Source: §A.1 · p. 15
  • Prior multimodal ECPE generally keeps the task in text while using images and audio as auxiliary information; this work instead studies visual causes and textual emotions.Source: §A.1 · p. 15

A.2 EMOTIONAL VIDEO CAPTIONING

  • Prior emotional video captioning methods use separate factual and emotional prediction, contextual or tree-structured emotion learning, multimodal context, adaptive decoding, or prompting and alignment losses.Source: §A.2 · p. 15
  • The authors identify visual emotional causes as an overlooked factor and present their work as the first to address them for emotion perception and caption generation.Source: §A.2 · p. 15

B THEORETICAL ANALYSIS: FROM THE INFORMATION THEORY PERSPECTIVE

The analysis argues that standard video–emotion alignment can reward background correlations, then presents counterfactual intervention as a way to focus emotion prediction on true causes. Its objective treats the cause as positive and the decoupled background as a hard negative, with the stated goal of reducing spurious information.

B THEORETICAL ANALYSIS: FROM THE INFORMATION THEORY PERSPECTIVE

  • Maximizing video–emotion mutual information can also maximize background leakage because I(V;Y) decomposes into cause information and I(U;Y|C).The paper attributes Contextual Leakage to confounders that co-occur with emotions independently of true causes.Source: B THEORETICAL ANALYSIS: FROM THE… · p. 16
  • The framework aims to maximize I(C;Y) while minimizing I(U;Y), separating causes and confounders with a causal mask.It defines C = V ⊙ Mc and U = V ⊙ (1 − Mc), then uses the cause as a positive sample and the confounder as a same-video hard negative.Source: B THEORETICAL ANALYSIS: FROM THE… · p. 16
  • The counterfactual objective penalizes predictive information in the background and is stated to drive I(U;E) toward zero.The analysis says minimizing the loss forces the causal mask to absorb emotion-relevant semantics into C.Source: B THEORETICAL ANALYSIS: FROM THE… · p. 16
  • The intervention is described as severing the backdoor path V → U → Y and anchoring prediction to P(Y | do(C)).The paper presents this as a theoretical guarantee of causal faithfulness relative to traditional positive alignment.Source: B THEORETICAL ANALYSIS: FROM THE… · p. 16

C.1 COMPARISON WITH EXISTING BENCHMARK

EVC-CauseGround combines emotion labels and captions with dense spatiotemporal visual-cause annotations and a counterfactual subset. The paper positions this combination as enabling evaluation of both caption generation and emotional cause reasoning.

C.1 COMPARISON WITH EXISTING BENCHMARK

  • The benchmark is presented as the first to combine fine-grained spatiotemporal causal tubes with a counterfactual subset for emotional videos.The table caption likewise identifies this pairing as distinguishing EVC-CauseGround across dataset domains.Source: Table A.1 · p. 17
  • EVC-CauseGround unifies emotion labels, rich captions, dense spatiotemporal visual-cause annotations, and a curated counterfactual subset.The paper describes it as a benchmark for evaluating caption quality alongside authentic emotional cause reasoning.Source: §C.1 · pp. 16–17, Table A.1 · p. 17
  • Traditional emotion datasets lack causal-trigger annotations, while emotion-cause pair datasets omit comprehensive captions and fine-grained visual localization.The latter datasets are described as restricting causal annotation to coarse utterance-level spans.Source: §C.1 · p. 17

C.2 DETAILS OF DATA SOURCES

The benchmark draws on EmVidCap-L and selected MAFW videos to obtain longer, varied scenes with dynamic foreground actions and complex backgrounds. The resulting 2,000 videos are split into training and evaluation sets, retaining their original emotion labels and captions.

C.2 DETAILS OF DATA SOURCES

  • The benchmark combines 1,523 EmVidCap-L videos with 477 selected MAFW videos, yielding 2,000 videos split into 1,500 for training and 500 for evaluation.Selected videos retain emotion labels and descriptive captions from their source datasets.Source: §C.2 · p. 17
  • EmVidCap-S is excluded because its typically sub-five-second, monotonous videos leave little meaningful scope for fine-grained temporal cause annotation.EmVidCap-L instead provides 1,523 user-generated videos averaging approximately 100 seconds.Source: §C.2 · p. 17
  • The 477 MAFW videos were selected for prominent foreground dynamic actions alongside complex backgrounds, expanding scene diversity.They were chosen from MAFW’s 10,045 clips, which include unconstrained environments from internet platforms.Source: §C.2 · p. 17

C.3 SPATIOTEMPORAL VISUAL CAUSE ANNOTATION

The annotation pipeline combines MLLM reasoning with optical-flow-guided localization and zero-shot tracking to produce spatiotemporal causal tubes. Its stages constrain temporal selection, localize a peak-motion frame, and propagate a box across the event.

C.3 SPATIOTEMPORAL VISUAL CAUSE ANNOTATION

  • The pipeline uses an MLLM-led, tracker-assisted coarse-to-fine process to generate spatiotemporal causal tubes while addressing temporal hallucination and spatial drift.It comprises temporal span grounding, peak-motion spatial grounding, and dense tube generation.Source: §C.3 · p. 18
  • Temporal grounding samples 30 frames into a labeled 6×5 grid and constrains the causal span to at most 30% of frames.The MLLM uses the ground-truth emotion label and caption to infer the triggering window; the limit corresponds to at most nine frames.Source: §C.3 · p. 18
  • Spatial grounding uses optical flow to find the peak-motion frame, then prompts the MLLM for a static causal-agent or action box.The paper separates temporal and spatial grounding to reduce bounding-box hallucination and jitter.Source: §C.3 · p. 18
  • SAM-3 bidirectionally tracks the peak-frame box across the localized event to create a continuous causal tube.Tracking proceeds from the peak frame toward both temporal boundaries.Source: §C.3 · p. 18

D EXPERIMENTAL SETTINGS

Experiments use two public EVC benchmarks and report emotion, semantic, and hybrid caption metrics. The paper compares traditional captioning methods with emotional captioning methods.

D EXPERIMENTAL SETTINGS

  • EVC-MSVD and EVC-VE provide the training and testing video and sentence splits used in the experiments.EVC-MSVD contains 240/134 videos and 8,169/4,611 sentences; EVC-VE contains 1,141/382 videos and 19,398/6,527 sentences.Source: D EXPERIMENTAL SETTINGS · p. 18
  • Evaluation combines emotion word accuracy Accsw and emotion sentence accuracy Accc with BLEU, METEOR, ROUGE, CIDEr, BFS, and CFS.BFS and CFS combine emotion evaluation with BLEU and CIDEr, respectively.Source: D EXPERIMENTAL SETTINGS · p. 18
  • Videos are sampled at 30 frames and decoded with Qwen-2.5-VL-7B using LoRA, with a 32,128-word vocabulary and 15-token maximum captions.The implementation uses 224 × 224 center-cropped frames, batch size 4, and beam search size 4.Source: D EXPERIMENTAL SETTINGS · pp. 18–19

E BASELINES

  • Traditional captioning baselines include SA-LSTM, CANet, and VideoBLIP, covering reconstruction, contextual attention, and video-text in-context learning.The paper groups these methods as Traditional Captioning Methods.Source: E BASELINES · p. 19
  • Emotional captioning baselines include FT, EPAN, and VEIN, which model visual emotion cues, hierarchical emotion perception, and multimodal emotional context.The paper groups these methods as Emotional Captioning Methods.Source: E BASELINES · p. 19
  • Additional emotional-captioning baselines include DCGN, MM-ECPE, HEART, GLM-EER, and SAGML, addressing emotion changes, cause pairs, temporal structure, and affective representations.The listed methods also include global-local memory and a soft affective heterogeneous graph.Source: E BASELINES · p. 19

F HUMAN EVALUATION

Human evaluation rates emotional descriptions on emotion accuracy, relevance, coherence, and usability. Causal-EVC is reported ahead of EVC state-of-the-art methods on all four metrics and also outperforms LVLMs in human evaluations.

F HUMAN EVALUATION

  • Twenty English-proficient participants rated 50 video-caption pairs from 1 to 10 for emotion accuracy, relevance, coherence, and usability.Usability asks how useful a description would be for someone, especially a blind person, to understand the video.Source: F HUMAN EVALUATION · p. 20
  • Causal-EVC is significantly ahead of EVC state-of-the-art methods on all four human-evaluation metrics and outperforms LVLMs in human evaluations.The comparison is reported in Table A.2; no numerical scores are provided in the supplied passages.Source: F HUMAN EVALUATION · p. 20, Table A.2 · p. 20

Related papers

Loading 2609.36776v1…