Source-linked AI summary
Jointly Cross- and Self-Modal Graph Attention Network for Query-Based Moment Localization
Daizong Liu, Xiaoye Qu, Xiao-Yang Liu, Jianfeng Dong, Pan Zhou, Zichuan Xu
TL;DR
Query-based moment localization must identify a short query-matching segment within an untrimmed video despite extensive irrelevant content and fine-grained visual–linguistic details. CSMGAN uses hierarchical query encoding and iterative message passing over cross-modal and self-modal graphs, and it outperforms state-of-the-art methods on the reported datasets.
Problem
Query-based moment localization must ground the segment matching a sentence query in an untrimmed video, where most content is irrelevant and fine-grained details matter.
Method
CSMGAN uses a hierarchical word–phrase–sentence encoder and stacked cross-modal and self-modal graph message passing over frame and word nodes.
Results
CSMGAN outperforms state-of-the-art methods with clear margins on Activity Caption and TACoS, including 5.27% and 8.90% gains on strict Activity Caption metrics.
Takeaways & Limitations
Jointly modeling cross-modal relevance and within-modality relations provides the paper’s framework for more precise moment localization.
Abstract
from arXiv · showhide
Query-based moment localization is a new task that localizes the best matched segment in an untrimmed video according to a given sentence query. In this localization task, one should pay more attention to thoroughly mine visual and linguistic information. To this end, we propose a novel Cross- and Self-Modal Graph Attention Network (CSMGAN) that recasts this task as a process of iterative messages passing over a joint graph. Specifically, the joint graph consists of Cross-Modal interaction Graph (CMG) and Self-Modal relation Graph (SMG), where frames and words are represented as nodes, and the relations between cross- and self-modal node pairs are described by an attention mechanism. Through parametric message passing, CMG highlights relevant instances across video and sentence, and then SMG models the pairwise relation inside each modality for frame (word) correlating. With multiple layers of such a joint graph, our CSMGAN is able to effectively capture high-order interactions between two modalities, thus enabling a further precise localization. Besides, to better comprehend the contextual details in the query, we develop a hierarchical sentence encoder to enhance the query understanding. Extensive experiments on four public datasets demonstrate the effectiveness of our proposed model, and GCSMAN significantly outperforms the state-of-the-arts.
1 INTRODUCTION
Query-based moment localization seeks the segment matching a sentence in an untrimmed video, where irrelevant content and fine-grained boundaries make visual–linguistic integration difficult. CSMGAN jointly models cross-modal and self-modal relations with hierarchical query representations to improve localization.
- The task is challenging because most video content is irrelevant to the query while only a short segment matches, requiring fine-grained segment discrimination.Accurate localization depends on deeply incorporating video and sentence information.
- CSMGAN jointly models cross-modal and self-modal relations over video-frame and sentence-word nodes for query-based moment localization.Cross-modal edges pass information between words and frames, while self-modal edges capture relations within each modality.
- CSMGAN stacks graph layers to propagate messages recursively, correlating highlighted instances with sequential elements and capturing higher-level multimodal relationships.The joint graph uses heterogeneous attention for cross-modal edges and self-modal edges for within-modality context.
- The hierarchical sentence encoder represents queries at word, phrase, and sentence levels to capture fine-grained textual information.These multi-granular representations are merged into a more informative sentence-query representation.
- The work reports that CSMGAN outperforms state-of-the-art methods with clear margins on Activity Caption and TACoS datasets.
2 RELATED WORKS
Related work covers query-based localization in images and videos, emphasizing candidate-region or candidate-segment matching and increasingly fine-grained cross-modal interaction. CSMGAN extends this direction with successive cross- and self-modal graph message passing for higher-level multimodal interactions.
- Query-based moment localization in videos: Video moment localization matches text descriptions to the most relevant segment, using candidate sampling, feature fusion, or direct temporal-boundary prediction.
- Graph neural networks: CSMGAN represents frames and words as graph nodes, passing messages first across modalities and then within each modality through attention-weighted graph relations.
- Graph neural networks: Successive cross- and self-modal graph layers enable the model to capture higher-level interactions among multimodal representations.
3 THE PROPOSED CSMGAN FRAMEWORK
CSMGAN jointly models cross-modal and self-modal relations between video frames and query words through layered graph message passing. It also uses video self-attention and a hierarchical word-, phrase-, and sentence-level encoder to improve contextual representations for moment localization.
- 3.2 Video and Sentence Encoder: The video encoder combines self-attention for long-range frame dependencies with a bidirectional GRU for temporal context.The resulting frame representations provide contextual inputs to the joint graph.
- 3.2 Video and Sentence Encoder: The hierarchical sentence encoder extracts word-, phrase-, and sentence-level features to capture fine-grained query information.Word embeddings are transformed with convolutions using unigram, bigram, and trigram windows, max-pooled into phrase features, and integrated with sentence features.
- 3.3 Joint Graph Network: CSMGAN represents frames and words as nodes in cross-modal and self-modal graphs, enabling layered message passing for multimodal interaction.The CMG connects word-frame pairs, while the SMG models word-word and frame-frame relations; stacking graph layers captures higher-order relationships.
- 3.3.1 Cross-Modal Relation Graph: Cross-modal attention computes pairwise frame-word relations and aggregates neighboring modality features with edge-weighted messages.Linear projections place heterogeneous frame and word features in a joint latent space before attention is computed.
- 3.3.2 Self-Modal Relation Graph: Self-modal attention models within-modality relations using semantic information and temporal position, reducing attention to distant but similar nodes.This temporal encoding helps the model correlate relevant frames or words for localization.
4 EXPERIMENTS
Experiments on Activity Caption and TACoS evaluate CSMGAN against prior methods, ablations, graph-layer choices, and qualitative localization behavior. The results show stronger benchmark performance, benefits from the joint graph and hierarchical encoder, and best behavior with two graph layers.
- 4.4 Ablation Study: The full model outperforms every ablation on both datasets, while removing the joint graph produces the worst ablation performance.The ablation results identify the jointly modeled cross- and self-modal relations as especially important, with the hierarchical sentence structure also contributing significantly.
- 4.2 Implementation Details: Two graph layers yield the best result, whereas additional layers reduce performance through accumulated messages and over-smoothing.With too many layers, video and sentence representations converge toward the same value.
- 4.5 Qualitative Results: Qualitative examples show that the full model localizes more precise boundaries than CMIN and the w/o CSG and w/o HS variants.The w/o CSG variant has the coarsest boundaries, while w/o HS misses contextual sentence-guiding clues.
- 4.5 Qualitative Results: Cross-modal attention concentrates on query words relevant to a frame, while self-modal attention assigns higher context weights to related surrounding frames.Increasing graph depth sharpens relevant word weights until excessive depth causes over-smoothing.
5 CONCLUSION
The paper concludes that CSMGAN jointly models cross-modal and self-modal relations for query-based moment localization. It also uses hierarchical word-, phrase-, and sentence-level query representations, with experiments demonstrating effectiveness.
- 5 CONCLUSION: CSMGAN captures higher-level interactions by first highlighting cross-modal components and then modeling within-modality correlations for frames and words.Its hierarchical encoder provides contextual sentence understanding through word-, phrase-, and sentence-level representations.
A ADDITIONAL DATASETS
Additional experiments analyze CSMGAN on the Charades-STA and DiDeMo datasets.
- A ADDITIONAL DATASETS: The paper reports detailed performance analysis on Charades-STA and DiDeMo as additional datasets.
A.1 Charades-STA
Charades-STA is an indoor-activity dataset with semi-automatically generated temporal annotations and short videos.
- A.1 Charades-STA: Charades-STA contains 12,408 training and 3,720 testing moment-query pairs, with videos averaging 30 seconds.Its temporal annotations are generated through sentence decomposition, keyword matching, and human checking.
A.2 DiDeMo
DiDeMo is an open-world dataset for natural-language moment retrieval, containing annotated moment-query pairs across training, validation, and testing splits.
- DiDeMo contains 10,464 videos and 33,005, 4,180, and 4,021 annotated moment-query pairs in its training, validation, and testing sets, respectively.
- Videos are trimmed to at most 30 seconds and divided into six five-second segments, with moments spanning one or more consecutive segments.
- Each DiDeMo video yields 21 candidate moments for localization.
B MORE QUALITATIVE RESULTS
Qualitative results span Activity Caption, TACoS, Charades-STA, and DiDeMo, comparing the full model with module-removed variants and ground truth. Across these examples, the full CSMGAN provides the most precise localization, supported by its graph-based cross/self-modal modeling and hierarchical sentence structure.
- Across four datasets, the full model achieves the most precise qualitative localization compared with ground truth and variants without HS or CSG.The visualizations cover Activity Caption, TACoS, Charades-STA, and DiDeMo.
- CSG mines deep cross-modal interactions and relates instances within each modality, while HS improves sentence understanding for grounding.