Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Zhu Zhang; Zhijie Lin; Zhou Zhao; Zhenxin Xiao

Source-linked AI summary

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Zhu Zhang, Zhijie Lin, Zhou Zhao, Zhenxin Xiao

arXiv:1906.02497v2Jun 6, 2019cs.IRcs.CVcs.MM

TL;DR

Query-based moment retrieval must localize a relevant moment in an untrimmed video, but prior methods often address only one aspect of the task. CMIN jointly models query syntax, long-range video dependencies, and video-query interaction, achieving state-of-the-art performance on ActivityCaptions and TACoS.

Problem
Existing moment-retrieval methods often focus on query representation, video context modeling, or multimodal fusion separately instead of developing a comprehensive system.
Method
CMIN combines syntactic GCNs, multi-head self-attention, and multi-stage cross-modal interaction to model query structure, long-range video context, and video-query relations.
Results
CMIN achieves state-of-the-art performance on the ActivityCaptions and TACoS datasets.
Takeaways & Limitations
The proposed network provides a comprehensive retrieval framework that considers multiple crucial factors of query-based moment retrieval.

Abstract

from arXiv · show

Query-based moment retrieval aims to localize the most relevant moment in an untrimmed video according to the given natural language query. Existing works often only focus on one aspect of this emerging task, such as the query representation learning, video context modeling or multi-modal fusion, thus fail to develop a comprehensive system for further performance improvement. In this paper, we introduce a novel Cross-Modal Interaction Network (CMIN) to consider multiple crucial factors for this challenging task, including (1) the syntactic structure of natural language queries; (2) long-range semantic dependencies in video context and (3) the sufficient cross-modal interaction. Specifically, we devise a syntactic GCN to leverage the syntactic structure of queries for fine-grained representation learning, propose a multi-head self-attention to capture long-range semantic dependencies from video context, and next employ a multi-stage cross-modal interaction to explore the potential relations of video and query contents. The extensive experiments demonstrate the effectiveness of our proposed method.

1 INTRODUCTION

Query-based moment retrieval localizes query-relevant temporal boundaries in untrimmed videos, where complex events and irrelevant content make precise localization difficult. CMIN addresses this challenge by jointly modeling query syntax, long-range video dependencies, and cross-modal relations.

Task and motivation: Query-based moment retrieval localizes the start and end boundaries of a query-relevant moment within an untrimmed video.Unlike whole-video retrieval, it provides fine-grained temporal localization and avoids manually searching for the relevant moment.
Task and motivation: Complex queries can describe successive actions and object interactions, requiring joint understanding of video and query contents through cross-modal interactions.The paper illustrates this with a query describing a man throwing a ball, hitting a boy, and landing it in a cup.
Research gap: Existing moment-retrieval methods often emphasize only query representation, video context modeling, or cross-modal fusion rather than a comprehensive system.CMIN therefore considers multiple crucial factors for high-quality retrieval.
Proposed approach: CMIN uses syntactic GCNs for fine-grained query representations and multi-head self-attention for long-range semantic dependencies in video context.The syntactic GCN exploits query dependency structure, while self-attention addresses interactions between distant frames.
Proposed approach: Its multi-stage cross-modal interaction combines attentive query aggregation, a cross gate, and low-rank bilinear fusion to learn video-query semantic representations.These stages extract frame-relevant query information, emphasize crucial contents, and model cross-modal relations.
Results: CMIN achieves state-of-the-art performance on the ActivityCaptions and TACoS datasets.The paper presents this as evidence of the proposed method’s effectiveness.

2 RELATED WORK

Prior work spans image/video retrieval, temporal action localization, and query-based moment retrieval, but these areas differ in task constraints and modeling assumptions. CMIN targets unconstrained query-based moment retrieval by jointly considering query syntax, long-range video context, and cross-modal interaction.

Image and video retrieval: Image and video retrieval selects media matching a natural-language query from candidate collections.Prior image-retrieval systems include visual-semantic alignment, visual concept discovery, and natural-language object retrieval approaches.
Temporal action localization: Temporal action localization detects action instances in untrimmed videos using methods such as segment-based 3D ConvNets and multi-stream bidirectional RNNs.Other approaches model action structure with structured temporal pyramids or adapt object-detection frameworks.
Temporal action localization: Temporal action localization methods remain limited to predefined action lists, whereas query-based moment retrieval introduces natural-language queries.This distinction separates query-driven retrieval from fixed-label action localization.
Query-based moment retrieval: Query-based moment retrieval detects moments depicting natural-language queries in untrimmed videos and has progressed from constrained settings to unconstrained ones.Earlier settings included fixed spatial prepositions, instruction videos, and ordering constraints; recent methods commonly use sliding-window frameworks.
CMIN: CMIN differs from previous methods by jointly modeling query syntax, long-range semantic dependencies in video context, and sufficient cross-modal interaction.These factors define the paper’s proposed cross-modal interaction network.

3 CROSS-MODAL INTERACTION NETWORKS

CMIN combines syntactic query modeling, long-range video-context modeling, and multi-stage cross-modal interaction to predict query-relevant moments. Its pipeline represents words and frames, then progressively aligns query clues with each frame and fuses their semantics.

Architecture: CMIN comprises syntactic GCN, multi-head self-attention, multi-stage cross-modal interaction, and moment retrieval components.The architecture is designed as a comprehensive system for query-based moment retrieval.
Problem Formulation: The task represents an untrimmed video as frame features and a natural-language query as word features, with annotated start and end boundaries for the target moment.Training learns cross-modal interaction networks to predict the target boundary.
Syntactic GCN Module: The query encoder uses pretrained GloVe features, a BiGRU for contextual word representations, and a syntactic GCN over dependency relations.The GCN passes information along directed, reverse, and self-loop dependency edges, using relation-specific transformations and residual connections.
Multi-Head Self-Attention Module: The video encoder applies multi-head self-attention so each frame can interact with adjacent and distant frames, followed by a BiGRU for contextual video semantics.Multiple attention heads aggregate information from different representation subspaces.
Multi-Stage Cross-Modal Interaction Module: The cross-modal module computes frame-word attention, aggregates query clues for each frame, applies cross-gated interaction, and learns fused cross-modal representations.The attention matrix records frame-word correlations before subsequent gating and low-rank bilinear fusion.

4 EXPERIMENTS

Experiments evaluate CMIN on ActivityCaption and TACoS using candidate-moment retrieval metrics, implementation settings, baseline comparisons, and ablations. CMIN outperforms prior baselines and its ablations, while TACoS remains more difficult and two GCN layers perform best.

4.1 Datasets: ActivityCaption contains 20,000 untrimmed videos with temporally annotated descriptions, whereas TACoS contains 127 cooking-scene videos with longer videos and shorter target moments.ActivityCaption uses val_1 for validation and val_2 for testing because test captions are unavailable.
4.2 Implementation Details: CMIN preprocesses videos into 500-dimensional frame features from overlapping 16-frame units and uniformly downsamples overlong sequences to 200 features.Features come from a pretrained 3D-ConvNet and are reduced from 4,096 dimensions using PCA.
4.2 Implementation Details: CMIN samples 1,400 candidate moments per ActivityCaption video and 800 per TACoS video using dataset-specific multi-scale windows.The window widths are [16, 32, 64, 96, 128, 160, 196] for ActivityCaption and [8, 16, 32, 64] for TACoS.
4.3 Evaluation Criteria: The evaluation metric R@n, IoU=m measures the percentage of queries whose top-n predictions include a moment with IoU above m.ActivityCaption uses m ∈ {0.3, 0.5, 0.7}, while TACoS uses m ∈ {0.1, 0.3, 0.5}.
4.4 Performance Comparisons: CMIN outperforms all prior state-of-the-art baselines on every reported criterion across both datasets, with especially large improvements on ActivityCaption.Context modeling and cross-modal interaction also outperform simpler approaches, while mean pooling in MCN may introduce noise.
4.4 Performance Comparisons: TACoS results and CMIN improvements are lower than ActivityCaption because longer videos, shorter targets, invariant scenes, and shorter queries make retrieval harder.The passage attributes these conditions to more candidate moments, weaker discrimination, and less precise query descriptions.
4.5 Ablation Study: CMIN(full) outperforms every ablation on both datasets, and all ablations still outperform all baselines.The ablations remove syntactic GCN, self-attention, cross gate, or low-rank bilinear fusion, supporting the usefulness of the comprehensive framework.
4.5 Ablation Study: Two syntactic GCN layers yield the best performance; fewer layers insufficiently leverage dependencies, while more layers cause over-smoothing.Over-smoothing makes word representations converge toward the same value.

5 CONCLUSION

The conclusion presents CMIN as a cross-modal interaction network for query-based moment retrieval. It combines syntactic query structure, long-range video dependencies, and fine-grained cross-modal interaction, with experiments on ActivityCaption and TACoS demonstrating effectiveness.

5 CONCLUSION: CMIN addresses three factors in query-based moment retrieval: syntactic query structure, long-range semantic dependencies in video context, and fine-grained cross-modal interaction.The network uses a syntactic GCN, multi-head self-attention, and multi-stage cross-modal interaction to explore video-query relations.
5 CONCLUSION: Experiments on ActivityCaption and TACoS demonstrate the effectiveness of the proposed method.The conclusion reports this outcome without specifying a numerical result.