Source-linked AI summary
Weakly Supervised Video Moment Retrieval From Text Queries
Niluthpol Chowdhury Mithun, Sujoy Paul, Amit K. Roy-Chowdhury
TL;DR
The paper addresses the costly, non-scalable need for temporal boundary annotations in text-to-video moment retrieval. It learns joint video-text representations with latent frame–sentence alignment through Text-Guided Attention, then uses that attention to retrieve moments. On two benchmark datasets, the weakly supervised approach performs comparably to several fully supervised methods, though it struggles when relevant visual content occupies only small frame regions or queries are ambiguous.
Problem
Text-to-video moment retrieval commonly requires costly and non-scalable temporal boundary annotations for each text description.
Method
A joint visual-semantic embedding framework uses Text-Guided Attention to learn latent alignment between video frames and video-level sentence descriptions.
Results
The weakly supervised approach performs comparably to fully supervised methods on two benchmark datasets.
Takeaways & Limitations
Using video-level sentences instead of temporal boundaries makes the task more realistic and allows learning from easily available web data with less manual annotation effort.
Takeaways & Limitations
Performance suffers when important visual content occupies only small frame regions, and ambiguous queries can produce overly long or temporally shifted moments.
Abstract
from arXiv · showhide
There have been a few recent methods proposed in text to video moment retrieval using natural language queries, but requiring full supervision during training. However, acquiring a large number of training videos with temporal boundary annotations for each text description is extremely time-consuming and often not scalable. In order to cope with this issue, in this work, we introduce the problem of learning from weak labels for the task of text to video moment retrieval. The weak nature of the supervision is because, during training, we only have access to the video-text pairs rather than the temporal extent of the video to which different text descriptions relate. We propose a joint visual-semantic embedding based framework that learns the notion of relevant segments from video using only video-level sentence descriptions. Specifically, our main idea is to utilize latent alignment between video frames and sentence descriptions using Text-Guided Attention (TGA). TGA is then used during the test phase to retrieve relevant moments. Experiments on two benchmark datasets demonstrate that our method achieves comparable performance to state-of-the-art fully supervised approaches.
1. Introduction
The paper introduces weakly supervised text-to-video moment retrieval using only video-level descriptions, avoiding temporal boundary annotations during training. Its Text-Guided Attention framework learns latent frame–text alignment and performs comparably to supervised approaches on two benchmark datasets.
- Motivation: Temporal boundary annotations for text descriptions are tedious, ambiguous, and difficult to scale for video moment retrieval.Existing methods require exact temporal locations, while marking moment boundaries can introduce ambiguity in training data.
- Problem: The paper asks whether video moments can be localized from natural-language queries using only video-level textual annotations without temporal boundaries.The weakly supervised formulation is motivated by the lower effort required to obtain descriptions than precise frame-wise annotations.
- Approach: Text-Guided Attention estimates which temporal locations correspond to a query, pools relevant video features, and supports moment localization during testing.The attention-weighted feature is trained to reduce the distance between the text-dependent video representation and its text vector.
- Approach: The proposed framework learns a joint visual-semantic embedding from video-level descriptions by using latent alignment between video frames and sentences.Frame-wise CNN features and text features are projected into a shared space for cross-modal learning.
- Results: Experiments on DiDeMo and Charades-STA show that the weakly supervised approach performs reasonably well compared with supervised baselines.The reported evaluation covers two benchmark datasets and compares the proposed method with supervised approaches.
2. Related Works
Prior work spans joint visual-semantic retrieval, temporal activity localization, and fully supervised text-to-video moment retrieval. This paper differs by retrieving sentence-specific moments without temporal ordering information or dense temporal annotations.
- Image/Video Retrieval using Text Queries: Video-text retrieval methods commonly learn joint visual-semantic spaces where similarity reflects semantic closeness between videos and text.The proposed framework also uses joint video-text embeddings but additionally learns alignment between individual video frames and descriptions.
- Activity Localization: Temporal activity localization includes fully supervised and weakly supervised approaches, but categorical methods rely on predefined activity classes.Text-to-video moment retrieval instead uses natural-language descriptions to identify relevant video portions.
- Text to Video Moment Retrieval: Most prior text-to-video moment retrieval methods require temporal annotations linking natural-language descriptions to video boundaries.Collecting these dense text–temporal-boundary tuples is tedious because moment start and end locations can be ambiguous.
- Text to Video Moment Retrieval: The proposed setting reduces data-collection cost and may enable training on larger video-text captioning datasets.The related-work discussion contrasts this with methods requiring temporally annotated training videos.
- Weakly Supervised Alignment: Unlike prior weak video-text alignment work, this task retrieves a portion of the video for a sentence without assuming temporal ordering among sentences.The framework can handle multiple sentence descriptions for different temporal portions during training and testing.
3. Approach
The approach learns sentence-specific relevant video regions without temporal boundary annotations by combining Text-Guided Attention with a joint video-text embedding. Similarity-based temporal attention pools video features for each sentence, and a ranking loss trains matching video-text representations to be closer than non-matching pairs.
- Approach: The framework learns latent alignment between video frames and sentence descriptions using a joint visual-semantic embedding model.Training uses video-level descriptions rather than moment-wise temporal annotations.
- Approach: Video and text are encoded by modality-specific networks and projected into a shared joint embedding space.Video features come from pre-trained CNNs, while sentences are encoded with GRUs; the embedding layers learn the shared representations.
- Text-Guided Attention: Text-Guided Attention computes cosine similarity between a sentence and each temporal video feature, then applies temporal softmax to identify relevant locations.The resulting attention values are expected to be high at sentence-relevant locations and low elsewhere.
- Text-Guided Attention: The attention weights produce a weighted temporal average, yielding a sentence-specific video feature for each description.This pooled feature represents the video content associated with one sentence and is computed separately for other sentences.
- Training Joint Embedding: The video-text ranking loss encourages matching projected video and text embeddings to be more similar than non-matching pairs in both retrieval directions.The scoring function uses cosine similarity, with a margin-based hinge formulation.
- Text-Guided Attention: At test time, Text-Guided Attention is used to localize the video moments relevant to a given sentence query.Figure 3 summarizes the computation of attention and sentence-wise video features used by the framework.
4. Experiments
Experiments on Charades-STA and DiDeMo evaluate whether weakly supervised moment retrieval can approach fully supervised methods using video-level descriptions. Across both datasets, the approach performs comparably to several supervised baselines, while qualitative cases reveal failures with ambiguous queries and small visual content.
- Experimental Setup: Experiments compare the weakly supervised approach with supervised and baseline methods on Charades-STA and DiDeMo.The study presents results on two benchmark datasets and compares performance against different supervised baselines.
- Evaluation Protocol: Charades-STA evaluation reports R@1, R@5, and R@10 at IoU thresholds 0.3, 0.5, and 0.7 using sliding windows of 128 and 256 frames.Candidate segments are ranked by Text-Guided Attention scores.
- Charades-STA Results: The method performs significantly better than VSA-RNN and VSA-STV, with minimum absolute improvements of 6.6% in R@5 and 2.6% in R@1.It is also comparable to CTRL on R@5 and shows reasonable performance relative to EFRC.
- DiDeMo Results: On DiDeMo, the weakly supervised approach performs comparably to several fully supervised methods across the reported evaluation results.Its R@1 accuracy and mIoU are nearly similar to supervised approaches, although R@5 is slightly lower; performance is also comparable to CCA and Txt-Obj-Retrieval.
- Qualitative Results: Qualitative examples show successful retrieval for several queries, including two different moments in the same video, but failures occur for ambiguous queries and moments containing small visual objects.Examples include shifted or overly long predictions when relevant content occupies only a small portion of the frames.
5. Conclusions
The paper introduces weakly supervised text-to-video moment retrieval, learning temporal relevance without sentence-level temporal boundaries. Despite weaker supervision, the method performs comparably to several fully supervised methods.
- The paper introduces the novel problem of weakly supervised text-to-video moment retrieval.
- An attention mechanism learns temporal relevance using only video-level sentences, without temporal boundaries for sentence descriptions.
- The weakly supervised formulation is more realistic because it avoids requiring temporal boundaries or temporal sentence ordering.
- The task can learn from easily available web data requiring minimal acquisition effort compared with manual annotations.
- The method performs comparably to several fully supervised methods despite using weak supervision.