Source-linked AI summary
TALL: Temporal Activity Localization via Language Query
Jiyang Gao, Chen Sun, Zhenheng Yang, Ram Nevatia
TL;DR
Predefined action lists struggle to represent complex activities in untrimmed videos, motivating temporal localization from natural-language queries. The paper introduces CTRL, which jointly models language and video clips while regressing action boundaries, and reports effective results on TACoS and Charades-STA. It further finds that non-parameterized offsets work better for temporal boundary regression.
Problem
Predefined action lists have difficulty covering complex combinations of actors, actions, objects, properties, and relations in untrimmed videos.
Method
CTRL jointly models language queries, candidate video clips, and temporal context to produce alignment scores and regress temporal action boundaries.
Results
CTRL is effective on TACoS and Charades-STA, and non-parameterized offsets outperform parameterized offsets for temporal boundary regression.
Takeaways & Limitations
Natural-language temporal localization provides a formulation for locating activities described by queries, including complex activities not captured by predefined labels.
Takeaways & Limitations
The method faces common hard cases involving long query sentences and videos containing similar activities with different objects.
Abstract
from arXiv · showhide
This paper focuses on temporal localization of actions in untrimmed videos. Existing methods typically train classifiers for a pre-defined list of actions and apply them in a sliding window fashion. However, activities in the wild consist of a wide combination of actors, actions and objects; it is difficult to design a proper activity list that meets users' needs. We propose to localize activities by natural language queries. Temporal Activity Localization via Language (TALL) is challenging as it requires: (1) suitable design of text and video representations to allow cross-modal matching of actions and language queries; (2) ability to locate actions accurately given features from sliding windows of limited granularity. We propose a novel Cross-modal Temporal Regression Localizer (CTRL) to jointly model text query and video clips, output alignment scores and action boundary regression results for candidate clips. For evaluation, we adopt TaCoS dataset, and build a new dataset for this task on top of Charades by adding sentence temporal annotations, called Charades-STA. We also build complex sentence queries in Charades-STA for test. Experimental results show that CTRL outperforms previous methods significantly on both datasets.
1. Introduction
The paper formulates temporal activity localization as finding an activity’s start and end times in an untrimmed video from a natural-language query. It proposes CTRL to jointly model language and video, producing alignment scores and temporal boundary regressions, and evaluates the task on TACoS and Charades-STA.
- Natural-language queries can express combinations of actors, actions, objects, properties, and relations that predefined action lists may not cover.
- TALL determines the start and end times of the activity described by a query inside a temporally untrimmed video.
- The task is difficult because cross-modal representations must support language-video matching while sliding-window features provide limited temporal granularity.
- CTRL jointly models text queries, candidate video clips, and temporal context to produce alignment scores and location regression results.
- The authors add sentence temporal annotations to Charades as Charades-STA and evaluate the framework on TACoS and Charades-STA.
2. Related Work
Related work spans deep video action classification and temporal localization, sentence-based image/video retrieval, and object detection. TALL differs by using a single sentence query for temporal localization and by applying boundary regression to action intervals.
- Action classification and temporal localization: Video action recognition has used two-stream ConvNets, C3D, and 2D ConvNets combined with temporal LSTM or mean pooling.
- Action classification and temporal localization: Temporal action localization methods have trained C3D with localization loss, generated frame-wise scores with temporal LSTM, or combined person detection with bidirectional LSTMs.
- Sentence-based image/video retrieval: Sentence-based retrieval methods align sentence embeddings with visual features, including object proposals, semantic graphs, and visual concepts.
- Sentence-based image/video retrieval: In the paper’s setting, one sentence query is input and temporal ordering among multiple sentences is not used.
- Object detection: Object detection motivates boundary regression, and the paper compares parameterized and non-parameterized offsets for action boundaries.
3. Methods
CTRL localizes temporal activities from natural-language queries by jointly modeling sentence embeddings, video clips, and temporal context. Its cross-modal regression architecture predicts clip-query alignment and refines temporal boundaries.
- CTRL Architecture: CTRL comprises visual and sentence encoders, a multi-modal processing network, and a temporal regression network.The final network produces alignment scores and location offsets between sentence queries and video clips.
- Problem Formulation: The problem formulation represents videos as frames paired with sentence annotations containing natural-language descriptions and temporal start/end times.Given a sentence query, the task is to predict one or more corresponding temporal intervals.
- Visual Encoder: Video clips are generated with temporal sliding windows, and the visual encoder combines each central clip with separately pooled preceding and following context.Context clips are sampled around the central clip, with the resulting representation projected to the shared feature dimension.
- Sentence Encoder: Sentence encoders map descriptions into the same feature space as visual representations using either an LSTM or Skip-thought extractor.A linear transformation follows the sentence-level embedding extractor.
- Multi-modal Processing: CTRL fuses visual and sentence features through element-wise addition, multiplication, and concatenation followed by a fully connected layer.The resulting multi-modal representation is passed to the temporal localization regression networks.
- Temporal Regression and Training: Sibling alignment and regression outputs are jointly trained, with regression offsets refining candidate clip locations toward nearby ground-truth intervals.Training uses a combined alignment and regression loss; aligned examples satisfy IoU and nIoL criteria, while multi-scale windows are used for training.
4. Evaluation
The evaluation uses TACoS and Charades-STA, comparing CTRL variants with retrieval and predefined-label baselines under sentence-level temporal localization metrics. Results show that cross-modal modeling and temporal boundary regression improve localization, including on complex queries.
- Datasets: TACoS provides 127 videos and 17,344 sentence–video-clip pairs with natural-language temporal annotations for evaluation.The dataset includes fine-grained activity labels and crowd-sourced sentence descriptions with temporal locations.
- Datasets: Charades-STA adds sentence temporal annotations to Charades, with 13,898 training pairs, 4,233 test pairs, and 1,378 complex queries.Complex queries connect consecutive sub-sentences and are used only for testing.
- Evaluation metric: Evaluation uses R@n, IoU=m, the percentage of queries whose top-n predictions include a start–end pair exceeding the specified IoU threshold.Overall performance averages sentence-level recall across all queries.
- Baselines and variants: CTRL is compared with VSA-RNN, VSA-STV, predefined verb and object classifiers, and variants differing in alignment, overlap, and boundary-regression losses.The retrieval baselines use the same training samples and test sliding windows as CTRL.
- Results: Visual retrieval and predefined classifiers perform worse than CTRL, while adding object information to verb labels improves representation of sentence queries.The paper attributes retrieval differences to multilayer alignment and temporal-context modeling, and classifier limitations to imprecise predefined labels.
- Results: CTRL(reg-np) significantly outperforms CTRL(aln) and CTRL(loc), while non-parameterized regression performs better than parameterized regression on TACoS and Charades-STA.Regression refines coarse sliding-window localization, whose fixed length limits alignment precision.
5. Conclusion
The paper addresses Temporal Activity Localization via Language and proposes CTRL, which refines activity locations through temporal regression. Results on TACoS and Charades-STA demonstrate the method’s effectiveness, with non-parameterized offsets outperforming parameterized offsets for boundary regression.
- CTRL jointly models language queries and video clips to localize activities through temporal regression.
- Non-parameterized offsets outperform parameterized offsets for temporal boundary regression.
- Experimental results demonstrate CTRL’s effectiveness on TACoS and Charades-STA.
6. Supplementary Material
Supplementary experiments evaluate temporal regression on THUMOS-14 using controlled comparisons of classification and regression variants. Non-parameterized regression improves over SCNN and further improves when applied during proposal generation.
- The supplementary experiment directly compares the temporal regression method with prior state-of-the-art methods on THUMOS-14.The evaluation targets the traditional action detection task.
- The THUMOS-14 setup removes cross-modal processing and trains localization with classification and regression losses.The model is evaluated with mAP at tIoU=0.5 using C3D features.
- The ablation compares classification-only, parameterized regression, non-parameterized regression, and regression applied during proposal generation and detection.
- 19.8 to 20.5: applying temporal regression during proposal generation produces a further improvement over the detection-only regression variant.The non-parameterized regression method also outperforms SCNN.