Source-linked AI summary
Weakly-Supervised Video Moment Retrieval via Semantic Completion Network
Zhijie Lin, Zhou Zhao, Zhu Zhang, Qi Wang, Huasheng Liu
TL;DR
Video moment retrieval requires locating query-relevant temporal boundaries, while full boundary annotation is costly and time-consuming. The paper introduces a weakly supervised Semantic Completion Network using context-aware proposal scoring, exploration-aware top-K selection, and semantic completion feedback. Experiments on ActivityCaptions and Charades-STA demonstrate its effectiveness, including substantial improvements over weakly supervised methods and comparable results to other fully supervised methods.
Problem
Video moment retrieval commonly relies on costly full temporal-boundary annotations, motivating training from coarse video-level annotations.
Method
The Semantic Completion Network scores proposals in one pass, selects top-K candidates through exploitation and exploration, and refines scores using masked-query reconstruction rewards.
Results
Experiments on ActivityCaptions and Charades-STA show substantial improvements over weakly supervised methods and results comparable to other fully supervised methods.
Takeaways & Limitations
The method can exploit video-query alignment to localize semantically relevant moments without full temporal-boundary annotations.
Abstract
from arXiv · showhide
Video moment retrieval is to search the moment that is most relevant to the given natural language query. Existing methods are mostly trained in a fully-supervised setting, which requires the full annotations of temporal boundary for each query. However, manually labeling the annotations is actually time-consuming and expensive. In this paper, we propose a novel weakly-supervised moment retrieval framework requiring only coarse video-level annotations for training. Specifically, we devise a proposal generation module that aggregates the context information to generate and score all candidate proposals in one single pass. We then devise an algorithm that considers both exploitation and exploration to select top-K proposals. Next, we build a semantic completion module to measure the semantic similarity between the selected proposals and query, compute reward and provide feedbacks to the proposal generation module for scoring refinement. Experiments on the ActivityCaptions and Charades-STA demonstrate the effectiveness of our proposed method.
Introduction
Video moment retrieval seeks temporal boundaries matching natural-language queries, but full boundary annotation is costly. The paper therefore proposes a weakly supervised framework using coarse video-level annotations, context-aware proposal scoring, exploration-aware selection, and semantic completion.
- Motivation: Video moment retrieval locates the temporal boundary of the moment most related to a natural-language query.The task requires understanding both video and query.
- Motivation: Weak supervision uses coarse video-level descriptions instead of manually marking temporal boundaries for training.Existing latent attention can focus on small discriminative regions rather than complete relevant regions.
- Framework: The proposal generation module scores all candidate proposals in one pass using cross-modal video-query representations and context.This differs from treating candidate proposals separately and supports scoring refinement through a rank loss.
- Framework: An exploitation-exploration algorithm selects top-K proposals from densely sampled candidates rather than relying only on high-confidence proposals.Random selection with decaying probability encourages exploration of potentially good proposals.
- Framework: The semantic completion module masks important query words and predicts them from proposal visual context to estimate semantic similarity.The framework includes this module as feedback for proposal scoring refinement.
- Evidence: Experiments on ActivityCaptions and Charades-STA demonstrate the effectiveness of the proposed weakly supervised framework.The framework requires only coarse annotations for training.
Related Work
Related work spans cross-modal image/video retrieval, temporal action detection, and weakly supervised moment retrieval. Unlike selecting whole videos or detecting categorized actions, moment retrieval targets a query-relevant temporal segment while reducing dependence on dense boundary labels.
- Image/Video Retrieval: Image and video retrieval methods learn joint visual-semantic spaces to select relevant items from candidate image or video sets.Moment retrieval instead targets a relevant temporal moment within a video.
- Temporal Action Detection: Temporal action detection identifies action categories and their temporal boundaries in untrimmed videos.Related approaches still require substantial manually labeled temporal annotations.
- Weakly Supervised Retrieval: Weakly supervised moment retrieval has used temporal ordering, dual captioning-retrieval cycles, and latent visual-text alignment as supervision.These approaches reduce reliance on full temporal annotations but use different forms of weak supervision.
Approach
The approach generates and scores candidate moments using cross-modal context, selects proposals through exploration and exploitation, and refines their scores with semantic completion feedback. The network is trained from reconstruction and ranking objectives, with shared encoder-decoder parameters.
- Proposal Generation: The proposal generation module uses cross-modal video-query representations to score candidate proposals at all time steps in a single pass.Candidate proposals are predefined at multiple ratios, and a fully connected layer with sigmoid nonlinearity produces their confidence scores.
- Proposal Selection: The selection algorithm combines exploitation and exploration to choose top-K proposals from densely sampled candidates.It randomly selects a proposal with probability p or chooses the highest-scoring proposal with probability 1 −p, then applies NMS to remove highly overlapping proposals.
- Proposal Selection: The exploration probability decays as training proceeds, gradually shifting selection toward high-confidence proposals.The decay is controlled by hyper-parameters λ1 and λ2 and the number of parameter updates.
- Semantic Completion: The semantic completion module masks important query words and predicts them from each selected proposal’s visual context to estimate semantic similarity.The module extracts proposal video features, encodes the masked query, and predicts masked words from cross-modal semantic representations.
- Training Objectives: Reconstruction loss trains masked-word prediction, while reward-based rank loss refines proposal confidence scores and the combined losses train the network end to end.Rewards favor proposals with lower reconstruction loss, and the multi-task loss balances reconstruction and ranking objectives with hyper-parameter β.
- Network Design: Bi-directional Transformer encoders and decoders share parameters, reducing the number of parameters and saving memory.The shared architecture also supports knowledge sharing across the visual and textual tasks.
Experiments
Experiments evaluate SCN on ActivityCaptions and Charades-STA using R@n, IoU=m, comparisons with supervised and weakly supervised baselines, ablations, and qualitative examples. SCN outperforms several baselines and its components improve training and retrieval quality.
- Evaluation Setup: SCN uses public video-moment-retrieval datasets, with ActivityCaptions providing 17,031 training description-moment pairs.ActivityCaptions contains untrimmed videos with multiple natural-language descriptions and temporal annotations.
- Evaluation Setup: Experiments evaluate SCN on ActivityCaptions and Charades-STA using R@n, IoU=m, with dataset-specific IoU thresholds.ActivityCaptions uses m ∈ {0.1, 0.3, 0.5}; Charades-STA uses m ∈ {0.3, 0.5, 0.7}.
- Overall Results: SCN substantially outperforms Random, supervised visual-embedding methods VSA-RNN and VSA-STV, and weakly supervised WS-DEC and TGA on both datasets.The authors also report results comparable to other fully supervised methods despite using only coarse annotations.
- Overall Results: VSA-RNN and VSA-STV perform worst on both datasets, indicating limitations of learning a common visual-semantic space for high-quality retrieval.The comparison is reported against SCN and other methods in the experiments.
- Ablation Study: SCN(full) reaches its best results faster than SCN(w/o. reward), supporting reward feedback for training the proposal generation module.The authors attribute the difference to one-hot supervision rejecting reasonable proposals when semantic feedback is inaccurate early in training.
- Ablation Study: SCN(full) performs better than SCN(w/o. mask) and slightly better than SCN(w/o. share), supporting masking, semantic completion, and parameter sharing.Parameter sharing also greatly reduces the number of parameters.
- Qualitative Analysis: Qualitative examples show top-ranked proposals covering discriminative query-relevant content, while higher-IoU proposals receive lower reconstruction loss.These observations support proposal scoring and semantic completion as complementary components.
Conclusion
SCN addresses weakly supervised video moment retrieval without manually labeled temporal boundaries. It combines proposal scoring, exploration-aware selection, semantic similarity estimation, and multi-task training, with effectiveness demonstrated on two datasets.
- SCN performs video moment retrieval without manually labeled start and end times, using only weak supervision.
- The proposal generation module scores all candidate proposals in a single pass using context information.
- An efficient proposal-selection algorithm balances exploitation and exploration when selecting proposals.
- SCN uses semantic completion for similarity estimation and a multi-task loss for training.
- Experiments on ActivityCaptions and Charades-STA demonstrate the effectiveness of the method and its proposal-selection and rank-loss components.