Source-linked AI summary
ExCL: Extractive Clip Localization Using Natural Language Descriptions
Soham Ghosh, Anuva Agarwal, Zarana Parekh, Alexander Hauptmann
TL;DR
Clip Localization requires selecting the relevant temporal span for a natural-language query, while prior ranking-based approaches are restrictive or complex. ExCL uses an extractive formulation to predict clip boundaries directly, and author-reported results show stronger performance on two benchmarks and comparable performance on a third. The study excludes datasets whose coarse fixed-size moments reduce the task to ranking predetermined candidates.
Problem
Clip Localization requires identifying one relevant video clip for a query, but prior fixed-candidate ranking approaches are restrictive for videos with substantial length variation.
Method
ExCL formulates clip localization as multimodal extractive prediction, including direct start and end time estimation from SoftMax distributions.
Results
ExCL significantly outperforms prior work on TACoS and ActivityNet and performs comparably on Charades-STA.
Takeaways & Limitations
The extractive formulation works better empirically across three different benchmark datasets and supports a modular model design.
Takeaways & Limitations
The evaluation excludes DiDeMo/TEMPO because their coarse fixed-size moments reduce the task essentially to ranking a fixed set of candidates.
Abstract
from arXiv · showhide
The task of retrieving clips within videos based on a given natural language query requires cross-modal reasoning over multiple frames. Prior approaches such as sliding window classifiers are inefficient, while text-clip similarity driven ranking-based approaches such as segment proposal networks are far more complicated. In order to select the most relevant video clip corresponding to the given text description, we propose a novel extractive approach that predicts the start and end frames by leveraging cross-modal interactions between the text and video - this removes the need to retrieve and re-rank multiple proposal segments. Using recurrent networks we encode the two modalities into a joint representation which is then used in different variants of start-end frame predictor networks. Through extensive experimentation and ablative analysis, we demonstrate that our simple and elegant approach significantly outperforms state of the art on two datasets and has comparable performance on a third.
1 Introduction
Clip Localization selects the relevant temporal span for a natural-language description, requiring cross-modal reasoning over video and text. ExCL replaces restrictive or complex candidate-ranking strategies with an extractive approach that predicts the target clip directly and performs strongly across benchmarks.
- Task: Clip Localization selects the relevant span of video frames corresponding to a natural-language description and requires cross-modal reasoning.The task also involves segmenting videos into action segments and measuring multimodal semantic similarity.
- Limitations of prior work: Because each query has one most relevant clip, ranking uniformly sampled fixed moments is restrictive and poorly suited to videos with varied lengths.These approaches use predefined candidate clips and are difficult to extend to videos with considerable length variance.
- Limitations of prior work: Sliding-window and proposal-based methods are inefficient or difficult to learn because coarse windows limit flexibility and similarity metrics depend on negative-sample choices.The cited approaches also require learned similarity-based ranking of candidate clips.
- ExCL: ExCL proposes a simple extractive approach that treats video as a passage and the target clip as an answer span in a multimodal analogue of machine comprehension.This reframes localization as direct extraction rather than ranking candidate moments.
- Results: ExCL significantly outperforms prior work on TACoS and ActivityNet while performing comparably on Charades-STA.The approach is also described as flexible and modular, allowing different encoders and attention models.
2 Approach
ExCL combines bidirectional text and video encoders with span predictors that estimate start and end frames, using either classification or regression objectives. Its predictor variants model cross-modal interactions with MLPs or recurrent LSTMs, while regression normalizes time and favors absolute error.
- Model architecture: ExCL uses modular text, video, and span-predictor components to produce start and end scores for each frame.The text encoder uses a bidirectional LSTM with GloVe embeddings, while the video encoder uses a bidirectional LSTM over I3D features.
- Training objectives: Classification normalizes start and end scores with SoftMax and trains them using negative log-likelihood over ground-truth frame indices.At inference, the predicted span maximizes the joint probability of valid start and end frames.
- Training objectives: Regression addresses discretized time prediction by modeling start and end times directly from SoftMax-derived probability distributions.The method computes expectations over the distributions and trains with regression losses.
- Training objectives: Absolute error with normalized times in [0, 1] yields better results than the other regression configuration.The compared regression losses include mean squared error and absolute error.
- Span Predictor Variants: The MLP predictor concatenates video features and sentence embeddings, whereas the Tied-LSTM captures recurrent cross-modal interactions before predicting start and end scores.The Tied-LSTM feeds recurrent hidden states together with the original inputs into an MLP.
- Span Predictor Variants: The Conditioned-LSTM conditions its end-frame predictor on the start predictor through a second bidirectional LSTM.The resulting hidden states generate start and end scores similarly to the tied-LSTM method.
3 Datasets
ExCL is evaluated on three datasets selected for variation in visual content, vocabulary, and query length. They range from challenging fine-grained cooking actions to diverse open-domain videos and short controlled activities.
- Dataset selection: The evaluation uses three datasets chosen for distinct visual, vocabulary, and query-length characteristics.The datasets are TACoS, ActivityNet Captions, and Charades-STA.
- MPII TACoS: TACoS contains temporally aligned cooking descriptions, often only a few seconds long, making small localization errors consequential.Its videos average five minutes, and 8.4% of queries are shorter than 1.6 seconds.
- ActivityNet Captions: ActivityNet Captions provides large-scale, open-domain videos with dense temporal captions and richer vocabulary.The dataset includes 10,024 training videos and 5,044 test videos, although 3,370 reported videos were unavailable for download.
- Charades-STA: Charades-STA contains short videos and sentence-level temporal annotations with lower query-length variance than TACoS and ActivityNet.Its average video length is 30 seconds, and query length variance is 3.7 seconds versus 39.5 and 78.1 seconds.
4 Experiments
The experiments evaluate ExCL across three datasets, training objectives, recurrent architectures, and span predictors. Results show that recurrent temporal context is important, while performance varies with dataset characteristics and predictor design.
- Experimental setup: ExCL is evaluated on TACoS, Charades-STA, and ActivityNet against current state-of-the-art baselines and multiple architectural variants.The evaluation includes classification and regression objectives, models with or without video LSTMs, and MLP, tied LSTM, and conditioned LSTM span predictors.
- Ablation results: Models without recurrent architectures perform significantly worse across all datasets, demonstrating the importance of LSTM-provided temporal context.Temporal context can come from either the video encoder or span predictors, depending on the model variant.
- Ablation results: A recurrent video encoder substantially improves performance even when paired with an MLP span predictor.Adding LSTM span predictors on top of the recurrent video encoder provides an additional performance boost.
- Training objectives: Classification does not yield a substantial performance gain over regression, indicating limited benefit from preserving the continuous nature of the labels.The study compares ExCL-clf and ExCL-reg across the benchmark settings shown in Table 1.
- Span predictors: Tied LSTM span predictors generally perform well across datasets, while differences from conditioned LSTMs are negligible.The benefit of tied predictors is more pronounced with the regression objective, possibly because conditioning is already captured in its formulation.
- Dataset comparison: Unlike prior reported differences between Charades-STA and ActivityNet, ExCL achieves similar performance on both, while TACoS remains more challenging.The authors hypothesize that prior models struggle with large query-length variability.
5 Conclusion
The paper concludes that ExCL provides an extractive alternative to ranking-driven clip localization. Its modular design and performance across three benchmarks support future generalizable models and extensions to longer, more complex videos.
- Main contribution: ExCL predicts clip spans directly from text queries instead of ranking candidate clips.The conclusion identifies this extractive formulation as the paper’s main contribution.
- Empirical findings: The approach works better empirically across three different benchmark datasets, with tied LSTM span predictors generally giving the best results.The conclusion contrasts this with the three different architectures previously serving as state of the art on those datasets.
- Future directions: The work lays a foundation for training a single generalizable model across datasets and possibly related tasks.This is presented as a next step enabled by the approach’s cross-dataset formulation.
- Future directions: ExCL’s modularity makes it straightforward to substitute encoder and span-predictor architectures.Future directions include temporal attention and extension to longer, more challenging videos such as movies.