Source-linked AI summary
Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Language
Songyang Zhang, Houwen Peng, Jianlong Fu, Jiebo Luo
TL;DR
Moment localization must retrieve a queried segment while accounting for relations among temporal moments, which existing methods neglect by treating candidates individually. The paper represents moments on a two-dimensional start–end map and uses 2D-TAN to encode adjacent temporal context and learn discriminative matching features. 2D-TAN achieves competitive or state-of-the-art performance across Charades-STA, ActivityNet Captions, and TACoS.
Problem
Existing moment-localization methods consider temporal candidates individually, neglecting dependencies that matter when target moments relate to other segments or overlap with semantically different moments.
Method
2D-TAN represents candidate moments on a two-dimensional map indexed by start and end time, then uses adjacent temporal context to learn discriminative video-language matching features.
Results
2D-TAN performs among the best across three benchmark datasets and criteria, ranking first or second in all reported cases.
Takeaways & Limitations
Modeling adjacent moment relations enables precise boundary localization while distinguishing visually similar moments with complex semantics.
Abstract
from arXiv · showhide
We address the problem of retrieving a specific moment from an untrimmed video by a query sentence. This is a challenging problem because a target moment may take place in relations to other temporal moments in the untrimmed video. Existing methods cannot tackle this challenge well since they consider temporal moments individually and neglect the temporal dependencies. In this paper, we model the temporal relations between video moments by a two-dimensional map, where one dimension indicates the starting time of a moment and the other indicates the end time. This 2D temporal map can cover diverse video moments with different lengths, while representing their adjacent relations. Based on the 2D map, we propose a Temporal Adjacent Network (2D-TAN), a single-shot framework for moment localization. It is capable of encoding the adjacent temporal relation, while learning discriminative features for matching video moments with referring expressions. We evaluate the proposed 2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art.
Introduction
Moment localization with natural language retrieves a queried segment from an untrimmed video, but existing candidate-by-candidate methods neglect temporal dependencies. 2D-TAN addresses this by representing moments and their adjacent relations on a two-dimensional temporal map.
- Moment localization retrieves a temporary segment from an untrimmed video using a natural-language query.
- Existing two-step methods select sliding-window candidates and match each candidate with the query independently.
- Ignoring temporal dependencies makes it difficult to localize moments related to other segments and distinguish overlapping moments with different semantics.
- 2D-TAN maps a candidate from time iτ to (j +1)τ at location (i, j), covering different moment lengths while representing adjacent relations.
- 2D-TAN directly performs moment localization without pre- or post-processing and is evaluated on three benchmark datasets.
Related Work
Related work addresses moment localization through visual-content context modeling and video-language alignment. Existing approaches use global context, attention, or sequential modeling, whereas 2D-TAN explicitly models context from moment representations with a 2D convolution network.
- Moment localization requires understanding video content and aligning video semantics with language.
- Visual Content Understanding: Visual-content methods integrate temporal context using global video representations concatenated with individual moment or semantic features.
- Video and Language Cross-Modality Alignment: Video-language alignment methods use attention to align relevant visual features with the query description.
- Video and Language Cross-Modality Alignment: Sequential approaches use recurrent neural networks to align video and language while capturing evolving frame-by-word interactions.
- Temporal Adjacent Network: Unlike RNN-based gradual aggregation from clip representations, 2D-TAN explicitly models context from moment representations through a 2D convolution network.
Our Approach
2D-TAN represents sampled video moments on a two-dimensional start–end map, then uses adjacent context to match candidates with a query sentence. It combines sparse candidate selection, pooled moment features, multimodal fusion, convolutional context modeling, and score prediction.
- Framework: The framework processes language representation, video representation, and moment localization in three steps.The query is encoded with word embeddings and a three-layer LSTM, while video clips are sampled and represented before matching.
- Video Representation: Each sampled clip is encoded by a pretrained CNN and fully connected layer, producing a dV-dimensional feature for candidate construction.The sampled clips are the basic elements for constructing moment candidates and their feature map.
- Video Representation: Moment candidates are max-pooled over their corresponding clip spans and reorganized into FM ∈ R^N×N×dV, indexed by start and end clips.The lower-triangular region where the start index exceeds the end index is invalid and is zero-padded.
- Candidate Selection: Sparse sampling selects densely spaced short moments and increasingly spaced long moments, reducing candidate count and subsequent matching cost.When N ≤ 16, all moments are enumerated; when N > 16, candidates satisfy G(a,b), including the displayed N = 64 example.
- Moment Localization: The model fuses the 2D temporal map with the sentence feature, applies convolutional Temporal Adjacent Network layers, and predicts scores for valid candidates.The fused map uses fully connected projections, Hadamard product, and normalization; convolution preserves map shape while expanding adjacent context.
- Loss Function: Training uses scaled IoU with thresholds tmin and tmax as supervision labels in a binary cross-entropy objective over valid candidates.Candidates below tmin receive 0, those above tmax receive 1, and intermediate IoU values are linearly scaled.
Experiment
2D-TAN is evaluated on three benchmark datasets against state-of-the-art methods using Rank n@m, with ablations examining sampling, receptive fields, and feature extraction. It performs among the best across datasets and criteria, while temporal-context modeling and larger receptive fields improve localization.
- Evaluation Setup: 2D-TAN is evaluated on Charades-STA, ActivityNet Captions, and TACoS against recently proposed state-of-the-art methods.The evaluation uses public benchmark datasets and compares sliding-window, RNN-based, GCN-based, and other approaches.
- Evaluation Setup: Rank n@m measures the percentage of queries with at least one correct moment among the top-n retrieved moments, using IoU threshold m.The reported settings vary by dataset, including thresholds from 0.1 to 0.7.
- Comparison to State-of-the-Art Methods: On TACoS, 2D-TAN surpasses ACL-K and TGN by more than 5 and 14 points in Rank1@0.5 and Rank5@0.5, respectively.On ActivityNet Captions, it also outperforms CMIN at IoU@0.5 and IoU@0.7.
- Comparison to State-of-the-Art Methods: 2D-TAN models dependencies among overlapping candidates with a 2D temporal map, improving over sliding-window methods that match candidates independently.The paper attributes the improvement to increased context information and better distinction between similar moments.
- Comparison to State-of-the-Art Methods: RNN-based methods outperform sliding-window approaches but remain inferior to 2D-TAN, which explicitly exploits context through a 2D temporal map.The comparison contrasts recurrently updated context with explicit temporal-map modeling.
- Comparison to State-of-the-Art Methods: On Charades-STA, 2D-TAN performs better than MAN at IoU@0.7 but is slightly inferior at IoU@0.5.MAN models relations between moment pairs with a graph convolutional network, whereas 2D-TAN uses a 2D convolution network.
- Ablation Study: Increasing sampled clips from 16 to 64 improves Rank5@0.7 from 57.79 to 60.18, while the sampling upper bound remains below 100%.The upper bound is limited because sampled clips can introduce errors.
- Ablation Study: 2D-TAN achieves results comparable to CMIN with 136 candidates versus CMIN’s 1400, and further improves with 1200 candidates and stacked convolutions.The comparison supports gains from context modeling rather than dense candidate sampling alone.
Conclusion
2D-TAN retrieves moments on a two-dimensional temporal map that incorporates adjacent candidates as context. The method encodes temporal relations, learns discriminative matching features, and achieves competitive performance on three benchmark datasets.
- 2D-TAN retrieves a moment on a two-dimensional temporal map using adjacent moment candidates as temporal context.
- The model encodes adjacent temporal relations while learning discriminative features to match video moments with referring expressions.
- 2D-TAN achieves competitive performance against state-of-the-art methods on three benchmark datasets.
- The authors propose extending the model to temporal action localization and video re-localization.