Source-linked AI summary
MAC: Mining Activity Concepts for Language-based Temporal Localization
Runzhou Ge, Jiyang Gao, Kan Chen, Ram Nevatia
TL;DR
Language-based temporal localization must locate activities described by unrestricted, potentially complex queries in untrimmed videos, while prior methods do not explicitly model rich activity semantics. ACL mines activity concepts from verb-object query pairs and visual classifier distributions, combines them with actionness-enhanced alignment and boundary regression, and achieves state-of-the-art results on Charades-STA and TACoS, including gains exceeding 5%.
Problem
Language-based temporal localization must identify query activity boundaries in untrimmed videos without a predefined activity list, including queries that describe multiple activities.
Method
ACL mines language concepts from verb-object pairs and visual concepts from classifier probability distributions, then predicts alignment scores and regressed temporal boundaries while using sliding-window actionness scores.
Results
ACL achieves state-of-the-art performance on both Charades-STA and TACoS, with reported gains exceeding 5% on each dataset.
Takeaways & Limitations
Mining activity concepts from both modalities and incorporating actionness scores improves language-based temporal localization alignment.
Abstract
from arXiv · showhide
We address the problem of language-based temporal localization in untrimmed videos. Compared to temporal localization with fixed categories, this problem is more challenging as the language-based queries not only have no pre-defined activity list but also may contain complex descriptions. Previous methods address the problem by considering features from video sliding windows and language queries and learning a subspace to encode their correlation, which ignore rich semantic cues about activities in videos and queries. We propose to mine activity concepts from both video and language modalities by applying the actionness score enhanced Activity Concepts based Localizer (ACL). Specifically, the novel ACL encodes the semantic concepts from verb-obj pairs in language queries and leverages activity classifiers' prediction scores to encode visual concepts. Besides, ACL also has the capability to regress sliding windows as localization results. Experiments show that ACL significantly outperforms state-of-the-arts under the widely used metric, with more than 5% increase on both Charades-STA and TACoS datasets.
1. Introduction
Language-based temporal localization must identify query activities in untrimmed videos without a fixed activity vocabulary, despite potentially complex queries containing multiple activities. MAC addresses these challenges by mining activity concepts from language and video and enhancing localization with actionness scores.
- Natural language-based temporal localization predicts activity start and end times in untrimmed videos from a natural language query.
- Unlike fixed-label localization, queries are unrestricted by a predefined activity list and may describe multiple activities.The paper gives “person begin opening the refrigerator to find more food” as an example.
- Previous methods learn video-query correlations from whole sentences and clips without explicitly modeling activity information in either modality.They commonly extract CNN video features and LSTM query embeddings, then match them in a shared space or with an MLP.
- MAC represents language activity concepts with verb-object pairs and visual activity concepts with pretrained classifier probability distributions.The query activity need not belong to the predefined label set; the distribution is used as an additional feature vector.
- ACL processes activity concepts and conventional video-query features separately, then predicts alignment scores and regressed temporal boundaries with a two-layer MLP.The model also incorporates an actionness score for each sliding window, improving alignment accuracy.
- More than 5% improvement is reported on both Charades-STA and TACoS, with state-of-the-art performance on both datasets.
2. Related Work
Related work covers language-based localization, concept discovery, and temporal activity proposal generation. MAC extends concept-based approaches to temporal localization by using concepts from both video and language modalities.
- Localization by Language: Language-based video localization methods match natural language queries to temporal segments, including approaches that regress boundary offsets and refine clip locations.CTRL jointly models queries and video clips while producing matching scores and boundary offsets.
- Localization by Language: Image-based localization methods address mentioned-object localization through query reconstruction or regression, including weakly and fully supervised settings.Visual consistency has also been used to improve weakly supervised localization.
- Concept Discovery: Concept discovery mines concepts from image-sentence corpora for retrieval and weakly supervised action classification.MAC applies concepts from both video and language modalities to temporal activity localization.
- Temporal Activity Proposal Generation: Temporal activity proposal generation produces high-recall action-containing proposals for later detection and localization stages.Prior methods use boundary regression, snippet-level actionness, temporal grouping, or complementary filtering.
3. Methods
ACL processes videos into clip features, aligns them with language queries, mines activity concepts from both modalities, and uses actionness-enhanced localization to score and refine temporal windows.
- ACL comprises video preprocessing, correlation-based alignment, activity-concept mining, actionness scoring, and temporal localization.The framework combines these stages into one pipeline.
- Video pre-processing: Video clips are formed from sliding windows over unit-level features, pooling a central clip with pre- and post-context clips.Average pooling and vector concatenation produce the clip-level visual feature, which is projected through a fully connected layer.
- Correlation-based alignment: The query encoder embeds the language query, while multimodal processing combines visual and textual representations through multiplication, addition, and concatenation.The resulting representation is passed to an MLP to obtain an alignment score.
- Activity concepts mining: Semantic activity concepts come from lemmatized verb-object pairs, while visual concepts come from pooled classifier outputs representing predefined activity-label distributions.Only the central clip supplies the clip-level visual activity concept to reduce contextual activity noise.
- Activity concepts mining: Activity concepts from videos and queries are linearly projected and multimodally processed to model their cross-domain interactions.The resulting activity-concept representation has dimension 4d_a.
- Actionness score enhanced localization: The actionness generator assigns each sliding-window candidate a confidence score indicating its likelihood of containing meaningful activities.The generator is a separately trained two-layer MLP.
- Actionness score enhanced localization: ACL combines pre-alignment scores with actionness scores for final alignment confidence and predicts start and end offsets to refine clip boundaries.The localization network uses both ordinary and activity-concept multimodal outputs and is trained with pre-alignment and regression losses.
4. Evaluation
The evaluation on Charades-STA and TACoS examines dataset characteristics, actionness scoring, activity-concept variants, and comparisons with prior methods. ACL and its actionness-enhanced variants achieve strong localization results across both datasets.
- Datasets: Charades-STA and TACoS provide evaluation settings for language-based temporal localization, with TACoS focused on cooking activities.Charades-STA was built from everyday activities, while TACoS contains cooking-domain activities with crowd-sourced descriptions aligned to video clips.
- Actionness Score: Actionness scoring improves localization: Prop+Score raises R@1, IoU = 0.5 performance by about 2%, while Swin+Score also improves performance.Swin+Score multiplies each alignment score by its sliding-window actionness score; Prop+Score applies actionness scores to refined proposals.
- Actionness Score: Swin+Score achieves higher average recall than pure sliding windows at all evaluated frequencies under AR-F on Charades-STA.The authors use Swin+Score as the default method because its actionness score helps identify more related sliding windows.
- Activity Concept Variants: ACL exceeds all four activity-concept system variants on Charades-STA, while Activity only also outperforms the previous method.The variants include Activity only, w/o SAC, w/o VAC, and concatenation; SAC denotes Semantic Activity Concepts and VAC denotes Visual Activity Concepts.
- State-of-the-art Comparison: On Charades-STA, ACL with Sports-1M activity labels exceeds the state-of-the-art method by over 3% under R@1, IoU = 0.5, and ACL-K gains over 5% using Kinetics labels.ACL-K denotes ACL using Kinetics activity labels.
- TACoS Results: On TACoS, ACL performs best among the activity-concept variants, while ACL-K exceeds the best previous method by 5% and Swin+Score alone improves performance by about 2%.The TACoS results also report that replacing activity concepts with activity labels does not bring improvements.
5. Conclusion
The paper presents ACL, an actionness score enhanced localizer that mines activity concepts from videos and sentence queries to localize query activities.
- ACL mines activity concepts from both videos and natural language queries for language-based temporal localization.