Source-linked AI summary
Proposal-free Temporal Moment Localization of a Natural-Language Query in Video using Guided Attention
Cristian Rodriguez-Opazo, Edison Marrese-Taylor, Fatemeh Sadat Saleh, Hongdong Li, Stephen Gould
TL;DR
The paper addresses temporal localization of a natural-language query in a long untrimmed video, where the goal is to identify the matching moment’s start and end. It proposes an end-to-end proposal-free model using dynamic filtering, guided attention, and soft labels for annotation uncertainty. Experiments report state-of-the-art performance on three challenging datasets.
Problem
Temporal moment localization must identify the start and end of a query-matching segment in an untrimmed video, while existing approaches rely on manually trimmed content or predefined proposals.
Method
The model directly predicts temporal boundaries using a dynamic filter for text-video matching, guided attention, and soft labels that model annotation uncertainty.
Results
The approach achieves state-of-the-art performance on Charades-STA, TACoS, and ActivityNet Captions, with ablations supporting its attention guidance and soft-label components.
Takeaways & Limitations
Proposal-free localization can directly estimate query boundaries while accounting for attention to relevant video features and uncertainty in boundary annotations.
Takeaways & Limitations
The method can confuse visually similar activities, such as opening a door and leaving a room, suggesting a need for stronger spatio-temporal features or contextual reasoning.
Abstract
from arXiv · showhide
This paper studies the problem of temporal moment localization in a long untrimmed video using natural language as the query. Given an untrimmed video and a sentence as the query, the goal is to determine the starting, and the ending, of the relevant visual moment in the video, that corresponds to the query sentence. While previous works have tackled this task by a propose-and-rank approach, we introduce a more efficient, end-to-end trainable, and {\em proposal-free approach} that relies on three key components: a dynamic filter to transfer language information to the visual domain, a new loss function to guide our model to attend the most relevant parts of the video, and soft labels to model annotation uncertainty. We evaluate our method on two benchmark datasets, Charades-STA and ActivityNet-Captions. Experimental results show that our approach outperforms state-of-the-art methods on both datasets.
1. Introduction
Temporal moment localization identifies the start and end of a video segment matching a natural-language query in an untrimmed video. The paper proposes a proposal-free, end-to-end approach using guided attention and modeling annotation subjectivity.
- Task motivation: Existing video-language tasks often require informative segments to be manually trimmed and aligned with textual descriptions.This motivates methods that can localize relevant content directly in untrimmed videos.
- Task motivation: Natural-language temporal localization identifies the start and end points of the video segment that best matches a query in an untrimmed video.The task is illustrated as finding the temporal moment corresponding to a sentence.
- Limitations of prior approaches: Proposal-and-rank methods generate candidate clips before classification, restricting localization through predefined candidates and making variable-length videos harder to handle.The paper positions proposal-free prediction as a way to avoid candidate generation and ranking.
- Proposed approach: The proposed model directly predicts start and end times using a dynamic filter for text-video matching and an attention mechanism focused on the segment of interest.The approach is described as not relying on candidate generation or ranking.
- Annotation uncertainty: The model also incorporates annotation subjectivity because activity boundaries have relatively low inter-annotator agreement.Experiments evaluate the method and its attention guidance and uncertainty modeling through ablation analysis on three datasets.
2. Related Work
Prior temporal localization methods either restrict queries to predefined action categories or generate and rank candidate clips. Language-driven methods broaden query specification, while later approaches increasingly reduce or eliminate proposal generation.
- Temporal action localization: Traditional temporal action localization is restricted to a predefined list of actions, limiting coverage of complex activity queries.A fixed label space makes it difficult to retain the objects, properties, and relations expressed in user queries.
- Language-driven moment localization: Language-driven moment localization determines the start and end times of the video segment best corresponding to a natural-language query.Natural-language queries can describe open-set activities, objects, properties, and relations.
- Proposal-based methods: Early language-driven methods learned shared video-language embeddings or generated temporal sliding-window proposals that were subsequently ranked.These approaches used ranking, alignment, or regression objectives for candidate clips.
- Query-guided proposals: Later methods reduced proposal counts with query-dependent clips or modeled temporal relationships among moments using latent variables and graph structures.These methods aimed to skip unlikely clips or address semantic and structural misalignment.
- Proposal-free localization: Other approaches directly predicted temporal coordinates or used reinforcement learning to dynamically observe video sequences instead of scanning many proposals.Cross-modal interactions and gated attention were also used to align textual and visual representations.
3. Proposed Approach
The proposed approach directly localizes a queried moment in an untrimmed video by predicting its start and end positions without generating candidate proposals. It combines multimodal encoding, query-conditioned attention, guided attention loss, and uncertainty-aware endpoint distributions.
- The model treats each annotated video–sentence–moment tuple as an independent training example and predicts the most likely start and end positions for a new query.
- The modular architecture contains sentence and video encoders, a dynamic filter, and a localization layer that predicts the moment boundaries.
- Proposal-based localization is computationally expensive and can miss temporal dependencies by treating candidate segments independently.
- Video Encoder: The video encoder maps input frames to visual feature vectors that capture high-level spatiotemporal semantics, with the feature sequence length potentially differing from the frame count.
- Language Encoder: The language encoder maps query words to semantic embeddings using a bidirectional GRU over pretrained GloVe vectors, then mean-pools the hidden states.
- Guided Attention: The dynamic filter transforms the sentence representation into a query-dependent filter and uses softmax-normalized inner products to scale video features by their relevance to the query.
- Guided Attention: A guided attention loss encourages features inside the annotated temporal boundaries to receive greater attention, supporting the model’s focus on relevant video content.
- Localization Layer: The localization layer contextualizes attended features with a bidirectional GRU and predicts categorical start and end distributions using separate fully connected layers.
4. Experiments
Experiments evaluate the proposal-free model on three temporal localization datasets, test guided attention and soft-labeling through ablations, and compare performance with prior methods. The method generally improves localization, while qualitative examples expose failures when visually similar actions are difficult to distinguish.
- Datasets: Experiments use Charades-STA, TACoS, and ActivityNet Captions, with TACoS presenting long videos and highly variable, often very short query segments.TACoS descriptions include atomic queries, with 8.4% shorter than 1.6 seconds.
- Evaluation: Evaluation reports temporal IoU accuracy at multiple α thresholds and mean tIoU on ActivityNet Captions.A prediction is correct when its overlap with the gold segment exceeds the selected α threshold.
- Ablation study: Soft-labeling with KL distribution matching significantly improves retrieval and full-query localization over the likelihood-based NLL loss.The ablation models subjectivity in the labeling process rather than treating the annotated boundaries as fully fixed.
- Ablation study: Adding the attention loss consistently improves performance with and without soft labels, and combining both techniques produces a synergistic gain.The results support guiding the model toward features within the query segment.
- Comparison to the state of the art: 48.22 mean tIoU on Charades-STA accompanies accuracies of 33.74 at α = 0.7 and 9.68 at α = 0.9, outperforming all baselines by a large margin.These values are reported for the proposed method in the Charades-STA comparison.
- Comparison to the state of the art: The method outperforms prior methods at α = 0.7 on TACoS and at α = 0.1 and 0.7 on ActivityNet Captions, while exceeding prior methods in ActivityNet mean tIoU but not ABLR at α = 0.3 and 0.5.The TACoS result is reported despite its longer videos and greater segment-duration variability.
- Qualitative results: A successful Charades-STA example reaches 98.28% tIoU with peak attention of 0.83 at 16.27 seconds, whereas opening and leaving a room are confused in a failure case.The failure is attributed to similar features for opening the door and leaving the room; richer spatio-temporal features or deeper context reasoning are suggested.
5. Conclusion
The paper presents an end-to-end architecture for temporal localization of natural-language queries, using guided attention and label-uncertainty modeling to improve localization.
- The architecture uses guided attention to focus localization on more generalizable video features.
- The approach models annotation uncertainty with soft labels and reports substantial performance gains.
- The method achieves state-of-the-art performance on three challenging datasets.