PaperlayerView exact revision on arXiv

Source-linked AI summary

Interventional Video Grounding with Dual Contrastive Learning

Guoshun Nan, Rui Qiao, Yao Xiao, Jun Liu, Sicong Leng, Hao Zhang, Wei Lu

arXiv:2106.11013v2cs.CVcs.CL

TL;DR

Video-grounding models may learn spurious language–video correlations from dataset selection bias. The paper introduces SCM-based interventional video grounding with confounder approximation and dual contrastive learning, reporting effectiveness across three benchmarks and state-of-the-art performance on Charades-STA.

  • Problem

    Video-grounding models may learn spurious correlations between textual and visual features because of dataset selection bias.

  • Method

    The paper combines SCM-based backdoor adjustment with do-calculus and latent-confounder approximation, plus dual mutual-information contrastive learning.

  • Results

    IVG-DCL is effective across three standard benchmarks and achieves state-of-the-art performance on Charades-STA.

  • Takeaways & Limitations

    The results support combining causal intervention and dual contrastive learning for video grounding under dataset-induced spurious correlations.

Abstract

from arXiv · show

Video grounding aims to localize a moment from an untrimmed video for a given textual query. Existing approaches focus more on the alignment of visual and language stimuli with various likelihood-based matching or regression strategies, i.e., P(Y|X). Consequently, these models may suffer from spurious correlations between the language and video features due to the selection bias of the dataset. 1) To uncover the causality behind the model and data, we first propose a novel paradigm from the perspective of the causal inference, i.e., interventional video grounding (IVG) that leverages backdoor adjustment to deconfound the selection bias based on structured causal model (SCM) and do-calculus P(Y|do(X)). Then, we present a simple yet effective method to approximate the unobserved confounder as it cannot be directly sampled from the dataset. 2) Meanwhile, we introduce a dual contrastive learning approach (DCL) to better align the text and video by maximizing the mutual information (MI) between query and video clips, and the MI between start/end frames of a target moment and the others within a video to learn more informative visual representations. Experiments on three standard benchmarks show the effectiveness of our approaches. Our code is available on GitHub: https://github.com/nanguoshun/IVG.

1. Introduction

Video grounding localizes a temporal span for a textual query, but dataset selection bias can create spurious language–video correlations. The paper introduces interventional video grounding and dual contrastive learning to address these issues.

  • Task and motivation: Video grounding localizes temporal boundaries in untrimmed video for a textual description involving complex actions, roles, and objects.The task retrieves the relevant segment from video content using the query’s multimodal interactions.
  • Task and motivation: Dataset imbalance can associate shared objects such as “people” and “vacuum” more strongly with holding than fixing actions.This selection bias may cause models to rely on spurious correlations between textual and visual features.
  • Proposed approach: Interventional video grounding applies SCM-based backdoor adjustment and do-calculus P(Y |do(X)) to deconfound these correlations.The paper also proposes approximating the unobserved confounder, which cannot be directly sampled from the dataset.
  • Proposed approach: Dual contrastive learning maximizes mutual information to learn more informative representations and better align textual and video features.The method is introduced as an unsupervised representation-learning component.
  • Evaluation: Experiments use quantitative and qualitative analyses on three benchmark datasets to evaluate the proposed approaches.The introduction reports findings from these analyses without specifying their numerical results here.

2. Related Work

Prior video-grounding methods use ranking, regression, or reinforcement learning, while related work motivates causal inference and mutual-information-based contrastive learning. The paper positions its approach as causal debiasing with improved text–video alignment.

  • Video grounding: Video-grounding methods include ranking, regression, and reinforcement learning approaches for locating target moments.Ranking methods may require proposal generation and matching, whereas reinforcement learning progressively localizes moments.
  • Causal inference: Unlike these approaches, the paper advances video grounding by mitigating spurious correlations between video and language features through causal inference.This distinguishes the proposed perspective from conventional matching and sequential-localization formulations.
  • Causal inference: Causal inference has been applied across vision and learning tasks to alleviate spurious correlations and disentangle desired model effects.The related-work discussion lists applications including visual question answering, recognition, segmentation, and policy evaluation.
  • Contrastive learning: Contrastive learning trains representations by contrasting positive and negative pairs, while mutual information quantifies dependence between random variables.The paper relates DCL to mutual-information-based unsupervised representation learning.
  • Contrastive learning: The IVG-DCL architecture contains VV-CL and QV-CL modules whose contrastive losses support better text–video alignment.These modules are part of the paper’s proposed architecture.

3.1. Overview

IVG-DCL encodes video and query inputs, applies two contrastive modules, fuses contextualized features for cross-modal interaction, and predicts temporal boundaries.

  • Architecture: Two encoders produce contextualized visual and textual representations from the input video and query.These representations are subsequently processed by the contrastive modules and fusion network.
  • Architecture: VV-CL and QV-CL learn representations with contrastive losses Lvv and Lqv.The modules operate before cross-modal fusion in the described architecture.
  • Architecture: A context-query attention fusion module captures cross-modal interactions between visual and textual features.The fused representations support subsequent temporal-boundary prediction.

3.2. Feature Encoder:

The feature encoder extracts video and query features, projects them into a shared dimension, and contextualizes them for subsequent contrastive and grounding operations.

  • Feature extraction: A feature extractor converts video frames into T video features, while query words are represented with pretrained word embeddings.Video features have dimension d_v, and query features have dimension d_w.
  • Feature extraction: Temporal start and end boundaries are converted into indices over the extracted video features for learning.The conversion uses the video duration and target start and end times.
  • Feature encoding: DCL treats clips inside the target boundaries as positive samples and clips outside them as negative samples for mutual-information-based contrastive learning.This alignment objective uses contextualized video and query representations.
  • Feature encoding: Two linear layers project video and query features into the same dimension before contextual encoding.The encoder then uses convolution, multi-head attention, and feed-forward layers to generate contextualized representations.

3.3. Contrastive Learning Module

The dual contrastive learning module aligns textual queries with target video clips and strengthens visual representations by maximizing mutual information across query-video and boundary-video relationships.

  • VQ Contrastive Learning: DCL treats clips inside target boundaries as positive samples and clips outside them as negative samples for aligning textual and visual representations.The contrastive objective uses mutual-information maximization to compute the loss.
  • VQ Contrastive Learning: The VQ-CL module maximizes mutual information between the pooled query representation q and video representation V′.Because direct MI estimation is intractable for continuous random variables, the method optimizes a Jensen-Shannon MI lower-bound estimator.
  • VV Contrastive Learning: The VV-CL module models mutual information between target-moment start and end boundary representations and other video clips.It follows the same estimation procedure to form the video-video contrastive loss.
  • Contrastive Learning Objective: DCL trains a better encoder by jointly maximizing the query-video and video-video mutual-information objectives.The two contrastive components provide complementary representation-learning signals.

3.4. Fusion Module

The fusion module captures interactions between video and text representations using context-query attention before downstream grounding prediction.

  • Fusion Module: Context-query attention takes visual representation V′ and textual representation Q′ as inputs to model their bi-modal interaction.The resulting fusion representation is denoted X.
  • Fusion Module: The implementation follows VSLNet and uses CQA as the interaction function for combining video and query features.The supplied passages identify FFN as a single feed-forward layer.

3.5. Interventional Causal Model

The interventional causal model uses Pearl’s SCM and backdoor adjustment to remove selection-bias-induced correlations, while approximating unobserved confounders from caption vocabulary.

  • Structural Causal Model: Pearl’s SCM represents variables as nodes and causal dependencies as directed edges linking query-video inputs, representations, mediators, confounders, and predicted boundaries.The model uses X for multi-modal representations, D as a mediator, and S and E for target start and end boundaries.
  • Causal Intervention: Conventional prediction P(S,E|Q,V) includes both the direct query-video path and a spurious backdoor path through the confounder Z.The backdoor path is {Q,V} ← Z → D → {S,E}.
  • Causal Intervention: The intervention do(Q,V) fixes query and video values and removes their dependency on parent confounder Z, enabling backdoor adjustment.The resulting objective is expressed through do-calculus rather than ordinary likelihood conditioning.
  • Latent Confounder Approximation: Because Z is unobserved, the method approximates its distribution using caption-derived vocabularies of roles, actions, and objects.This approach avoids directly sampling the unavailable dataset-selection process.
  • Latent Confounder Approximation: The approximated confounder representation is mapped to the representation space with a linear projection and word-embedding layer, then used in deconfounded prediction.NWGM is applied to approximate the softmax-based interventional prediction and obtain target start and end indices.

3.6. Training Objectives

Training combines the grounding loss with dual contrastive losses, while DCL is omitted during inference because it only supports representation learning and requires moment annotations.

  • Training Objective: The overall training loss combines grounding and dual contrastive objectives weighted by α and β.The supplied passage identifies α and β as the weights for the two contrastive losses.
  • Inference: DCL is ignored during inference because it facilitates representation learning and requires moment annotations to identify contrastive samples.Thus, the contrastive components contribute during training rather than inference.

4. Experiments

Experiments on three benchmark datasets evaluate IVG-DCL against existing baselines, causal alternatives, ablations, and sensitivity settings. IVG-DCL achieves strong temporal-localization results, while analyses attribute gains to causal intervention and dual contrastive learning.

  • Performance Comparisons: IVG-DCL achieves state-of-the-art performance on Charades-STA, improving over VSLNet by 3.33 points at IoU=0.3 and 2.87 points in mIoU.It also exceeds VSLBase by 9.27 at IoU=0.5 and 6.09 in mIoU.
  • Performance Comparisons: IVG-DCL performs best on TACoS at IoU=0.1, IoU=0.3, IoU=0.5, and mIoU, including a 9.23-point gain over VSLNet at IoU=0.3.It also exceeds 2D-TAN and DRN by 3.75 and 5.9 at IoU=0.5, respectively.
  • Performance Comparisons: IVG-DCL achieves state-of-the-art results in most ActivityNet Caption settings, but its performance gain is smaller than on Charades-STA and TACoS.The paper notes that ActivityNet Caption has more complex activities and longer target moments, averaging 36.18s versus 8.22s and 5.45s.
  • Ablation Study: Removing IVG decreases performance by 4 points on average, while removing DCL decreases it by 2 points and removing both decreases it by 10 points on average.Removing either QV-CL or VV-CL decreases performance by 1 point on average.
  • Sensitivity Analysis: The best loss-weight combination is {α = 0.1, β = 0.01}; increasing both weights reduces performance, and {α = 1, β = 1} lowers it by 5 points on average.These results indicate that IVG has greater importance than DCL in the multi-task learning setting.
  • Qualitative Analysis: In a Charades-STA case study, IVG-DCL distinguishes “holding” from “fixing” despite 208 versus 35 training instances, whereas VSLNet mislocalizes the less frequent query.The case study is presented as evidence that the model mitigates spurious text-video correlations.

5. Conclusion

The paper proposes IVG-DCL to address spurious correlations between query and video features through causal inference. Experiments on three standard benchmarks show the effectiveness of the proposed method.

  • IVG-DCL targets spurious correlations between query and video features using causal inference.
  • Experiments on three standard benchmarks show the effectiveness of IVG-DCL.
Loading 2106.11013v2…