PaperlayerView exact revision on arXiv

Source-linked AI summary

Deconfounded Video Moment Retrieval with Causal Intervention

Xun Yang, Fuli Feng, Wei Ji, Meng Wang, Tat-Seng Chua

arXiv:2106.01534v1cs.CV

TL;DR

VMR models can exploit temporal-location biases rather than visual content, limiting generalization. The paper builds a causal model and introduces DCM, which uses feature disentangling and intervention to remove location confounding; experiments report improved retrieval performance and generalizability, with explicit OOD limitations for some moment-length cases.

  • Problem

    Existing VMR models can exploit temporal-location biases instead of learning cross-modal matching, and robust matching against these biases has not been studied.

  • Method

    DCM builds a causal model, disentangles moment representations, and applies causal intervention based on backdoor adjustment to remove moment-location confounding.

  • Results

    DCM achieves higher retrieval performance and strong generalizability on IID and OOD testing sets across ActivityNet-Captions, Charades-STA, and DiDeMo.

  • Takeaways & Limitations

    The deconfounding algorithm can remove harmful location bias while retaining its useful temporal-context effect and can be integrated with existing VMR methods.

  • Takeaways & Limitations

    OOD evaluation excludes long moments from ActivityNet-Captions because moment length itself creates a prediction prior; very long targets can be retrieved by returning the whole video.

Abstract

from arXiv · show

We tackle the task of video moment retrieval (VMR), which aims to localize a specific moment in a video according to a textual query. Existing methods primarily model the matching relationship between query and moment by complex cross-modal interactions. Despite their effectiveness, current models mostly exploit dataset biases while ignoring the video content, thus leading to poor generalizability. We argue that the issue is caused by the hidden confounder in VMR, {i.e., temporal location of moments}, that spuriously correlates the model input and prediction. How to design robust matching models against the temporal location biases is crucial but, as far as we know, has not been studied yet for VMR. To fill the research gap, we propose a causality-inspired VMR framework that builds structural causal model to capture the true effect of query and video content on the prediction. Specifically, we develop a Deconfounded Cross-modal Matching (DCM) method to remove the confounding effects of moment location. It first disentangles moment representation to infer the core feature of visual content, and then applies causal intervention on the disentangled multimodal input based on backdoor adjustment, which forces the model to fairly incorporate each possible location of the target into consideration. Extensive experiments clearly show that our approach can achieve significant improvement over the state-of-the-art methods in terms of both accuracy and generalization (Codes: \color{blue}{\url{https://github.com/Xun-Yang/Causal_Video_Moment_Retrieval}}

1 INTRODUCTION

Video Moment Retrieval localizes a video moment from a textual query, but existing models can exploit temporal-location biases instead of learning video-query matching. The paper frames this as a hidden-confounding problem and proposes causal deconfounding evaluated under both IID and changed location distributions.

  • Task and motivation: VMR retrieves a specific video moment according to a textual query by modeling multimodal content and cross-modal relationships.Existing systems commonly generate diverse moment candidates and rank them by query-moment matching scores.
  • Task and motivation: Temporal-location biases can make models rely on annotation patterns while overlooking the true effect of video content, harming generalizability.The paper identifies long-tailed annotations and correlations between query wording and moment locations as two bias sources.
  • Task and motivation: Temporal location has both harmful and useful effects: it can mislead predictions, yet it can also help model temporal context for queries with temporal language.The stated goal is to retain the useful effect while removing the harmful effect for robust matching.
  • Causal framing: The proposed causal graph treats query, video moment, moment location, and prediction as interacting variables, with location acting as a hidden confounder.Traditional prediction uses P(Y|Q,V), while the paper argues that location spuriously associates the video moment with prediction.
  • Proposed direction: DCM applies causal intervention to multimodal inputs to remove moment-location confounding and improve robustness.The framework is evaluated on IID and OOD testing sets across ActivityNet-Captions, Charades-STA, and DiDeMo.

2 CAUSAL VIEW OF VMR

The causal view models query, video moment, moment location, and prediction to expose how location can create spurious video-prediction correlations. VMR is formulated as learning a cross-modal matching function over query-moment pairs while removing this confounding influence.

  • Problem formulation: VMR represents a query and video as query-moment pairs with relevance scores, then ranks candidate moments using a learned matching function.Candidates are generated with diverse temporal durations, and candidates with y≥0.5 are usually treated as positive.
  • Problem formulation: The matching function models intra-modality and inter-modality feature interactions and maps each query-moment pair to a real-valued score.Inference returns the candidate with the highest estimated matching score.
  • Causal graph: The causal graph contains query Q, video moment V, moment location L, and prediction Y, with directed links representing causal interactions.Traditional VMR models use Q→Y and V→Y and predict with P(Y|Q,V).
  • Causal graph: Moment location L is a confounder because it influences prediction and the video-related variable, creating a spurious correlation between V and Y.The location-to-prediction path is associated with annotation frequency and location-specific dataset priors.
  • Causal objective: The paper therefore proposes deconfounding to remove the backdoor-path effect of moment location so visual content contributes more directly to prediction.This follows the paper’s diagnosis that location confounds V and Y and causes visual content to be overlooked.

3 METHODOLOGY

DCM deconfounds video moment retrieval by separating visual content from temporal location and intervening over possible locations before matching queries with moment candidates. It is implemented within cross-modal matching networks to reduce location-related confounding in prediction.

  • Deconfounded Cross-modal Matching: The resulting DCM matching function predicts a query–candidate matching score from disentangled multimodal representations and ranks candidates by that score.The implementation can use existing matching architectures, including CMI and TCN, with intervened moment representations.
  • Moment Representation Disentangling: DCM first disentangles each moment representation into independent content and location latent vectors.The content vector represents visual information, while the location vector is supervised using the moment’s original temporal coordinates.
  • Moment Representation Disentangling: A reconstruction constraint makes the location representation approximate positional features derived from the moment’s start and end timestamps.The positional embedding is nonlearnable and supervises the location branch during training.
  • Moment Representation Disentangling: An independence constraint reduces correlation between content and location, weakening the feature-level link between video moments and temporal location.The disentangling module is paired with intervention at the prediction level.
  • Causal Intervention: Causal intervention estimates prediction by averaging over candidate location features, forcing the query to interact with possible target locations under a location prior.The method uses sampled moment candidates to approximate the location confounder set and assumes equal opportunity for each location when setting the prior.
  • Deconfounded Cross-modal Matching: The model operates on sampled moment tensors indexed by candidate start and end coordinates and applies cross-modal networks to aggregate temporal dependencies.CMI uses convolution over the intervened moment tensor, while TCN uses a multi-layer 2D convolutional network.

4 EXPERIMENTS

The experiments evaluate DCM against baseline and state-of-the-art VMR methods under both IID and OOD testing settings. The evaluation is designed to assess effectiveness and robustness to distribution changes.

  • Experimental Evaluation: The study compares DCM with baseline methods and state-of-the-art VMR methods.The stated evaluation uses extensive comparisons to assess DCM methods.
  • Experimental Evaluation: The evaluation includes both independent-and-identically-distributed and out-of-distribution testing settings.These settings are used to examine performance under standard testing and distribution changes.
  • Experimental Evaluation: The experiments assess the effectiveness of DCM methods across the selected comparison methods and testing conditions.The section frames the comparisons as tests of DCM effectiveness in both evaluation settings.

4.1 Datasets and Experimental Setting

Experiments use three public VMR datasets, strict rank-1 localization metrics, IID and OOD testing, and comparisons with baselines and recent SOTA methods. OOD testing shifts target timestamps by inserting a random clip at the beginning of each video.

  • Datasets: The evaluation covers ActivityNet-Captions, Charades-STA, and DiDeMo, which represent diverse daily activities, indoor activities, and videos from YFCC100M.The datasets contain approximately 20k ActivityNet-Captions videos and more than 10k DiDeMo videos; Charades-STA is a widely used VMR dataset.
  • Metrics: Performance is measured with rank-1 accuracy at specified IoU thresholds and mean IoU of the top-1 results.The study uses IoU thresholds of 0.5 and 0.7 for ActivityNet-Captions and Charades-STA, and 0.7 and 1.0 for DiDeMo.
  • Testing Settings: OOD testing inserts a randomly generated clip at each video’s beginning, changing video duration and shifting target timestamps accordingly.This protocol evaluates generalization beyond the original IID split.
  • Comparison Methods: The comparison includes CMI and TCN baselines, frequency-prior and Blind-TAN bias-based methods, and reported SOTA methods such as SCDM, 2D-TAN, DRN, LGI, and VSLNeT.DCM variants are formed by adding DCM to the CMI and TCN matching networks.
  • Implementation and Reporting: Table 2 reports R@1 percentages for SOTAs and baselines, with markers for relative improvements and statistical significance in mIoU comparisons.The table defines dagger markers for relative R@1 improvements and star markers for mIoU significance levels.

4.2 Overall Performance Comparison

DCM improves VMR performance across IID and OOD settings, with especially large gains under distribution shifts. Comparisons also reveal strong temporal-location biases and the importance of OOD evaluation.

  • Baseline comparison: OOD testing exposes strong temporal-location bias: Freq-Prior reaches nearly 30% mIoU on IID sets, while Blind-TAN reaches 45.3% R@1(IoU>0.5) on ANet-Cap but drops significantly OOD.These results support the necessity of OOD testing for evaluating generalization.
  • DCM performance: DCM consistently improves TCN and CMI across methods, datasets, backbones, and testing distributions.The gains are typically larger on OOD than IID testing.
  • DCM performance: 23.68% relative improvement in R@1(IoU>0.5) occurs on OOD testing versus 2.82% on IID testing for ANet-Cap.This comparison indicates stronger benefits under distribution changes in moment annotations.
  • Evaluation caveat: OOD evaluation excludes nearly 4K long moments from ANet-Cap because returning the whole video can achieve 100% R@1(IoU>0.5) when moments occupy at least half the video.This exclusion is intended to preserve OOD testing effectiveness.
  • Comparison with SOTA: DCM achieves new IID SOTA results for R@1(IoU>0.7), including 27.7% on ANet-Cap and 27.8% on DiDeMo.On OOD testing, DCM reports 27.4% relative improvement over LGI on ANet-Cap OOD-1 and 26.7% over LGI on Charades-STA OOD-2.

4.3 Study of DCM

The study analyzes DCM through feature disentangling, causal intervention, and performance across moment-length groups and qualitative distribution shifts. Results show that these components improve robustness, especially under OOD testing and for short moments.

  • 4.3.1 Effect of Feature Disentangling.: Removing feature disentangling causes clear performance drops across all settings.The authors attribute this to entangled representations being dominated by latent location factors and weakening generalization.
  • 4.3.1 Effect of Feature Disentangling.: Higher independence between moment content and location is substantially coupled with better OOD performance.Distance correlation and R@1 comparisons support the relationship between feature disentangling and generalization.
  • 4.3.2 Effect of Causal Intervention.: 15.2%, 19.3%, and 4.5% relative R@1(IoU>0.7) drops occur without causal intervention on the three datasets.On ANet-Cap, removing intervention raises IID performance from 27.7 to 29.5 but lowers OOD performance from 7.9 to 6.7.
  • 4.3.2 Effect of Causal Intervention.: Removing either causal intervention or counterfactual loss substantially reduces OOD performance.The findings identify counterfactual loss as a complement to representation intervention.
  • 4.3.3 On the Moment Length.: 168% relative improvement is observed for TCN+DCM over TCN on Charades-STA’s shortest-moment group.The group’s average moment duration is about 3 seconds, and improvements occur across all Charades-STA length groups.

5 RELATED WORK

Related work positions VMR as cross-modal retrieval and connects this paper to causal inference for removing dataset biases. The paper distinguishes DCM by targeting temporal-location confounding and evaluating generalizability beyond IID settings.

  • 5 RELATED WORK.: VMR retrieves short video segments from language queries by learning cross-modal matching between language and video.Existing methods commonly generate candidate moments and rank them using matching scores, with differing approaches to candidate sampling and temporal structure.
  • 5 RELATED WORK.: Prior VMR methods mainly differ in candidate generation and multimodal matching, while some have limited recall for short candidates.Traditional multi-scale sliding windows are contrasted with dense candidate enumeration that preserves temporal structure.
  • 5 RELATED WORK.: The paper differentiates its approach through counterfactual training, OOD evaluation, and analysis of temporal-location vulnerability.Unlike existing methods described here, its counterfactual framing asks what happens if the target moment does not exist in the video.
  • 5 RELATED WORK.: Causal inference has been applied to remove dataset biases in recommendation, visual dialog, segmentation, feature learning, video action localization, and scene graphs.This paper extends that line of work to VMR by identifying temporal location as a confounder.

6 CONCLUSION

The paper concludes that deconfounded cross-modal matching improves VMR generalizability by addressing temporal-location biases. It presents the approach as an initial step toward robust retrieval and identifies query intervention and other location-sensitive tasks as future directions.

  • 6 CONCLUSION.: DCM improves VMR generalizability by mitigating temporal-location biases while retaining useful location information.The method is presented as integrable with most existing VMR methods.
  • 6 CONCLUSION.: OOD evaluation shows that existing VMR models are sensitive to distribution changes in moment annotations.The conclusion links this sensitivity to hidden temporal-location biases affecting training and prediction.
  • 6 CONCLUSION.: Future work will explore intervention on user queries and apply deconfounding to visual grounding and temporal activity localization.These directions target more comprehensive intent understanding and other location-sensitive tasks.
Loading 2106.01534v1…