Source-linked AI summary
Context-aware Biaffine Localizing Network for Temporal Sentence Grounding
Daizong Liu, Xiaoye Qu, Jianfeng Dong, Pan Zhou, Yu Cheng, Wei Wei, Zichuan Xu, Yulai Xie
TL;DR
Temporal sentence grounding must locate a sentence-matching segment in an untrimmed video, while prior proposal-ranking and boundary-regression approaches have limitations. CBLN jointly scores all start-end pairs with biaffine localization, enriches them using local and global contexts, and applies multimodal self-attention for query-guided video features. Experiments on three public datasets report significant improvements over state-of-the-art methods.
Problem
Temporal sentence grounding seeks the start and end boundaries of a sentence-matching segment in an untrimmed video, but prior methods rely on proposals or do not jointly consider start and end features.
Method
CBLN jointly scores all possible start-end frame pairs with biaffine localization, using multimodal self-attention and multi-scale local-global contexts.
Results
CBLN significantly outperforms state-of-the-art methods on ActivityNet Captions, TACoS, and Charades-STA.
Takeaways & Limitations
The experiments demonstrate the effectiveness of the proposed biaffine localization framework for temporal sentence grounding.
Abstract
from arXiv · showhide
This paper addresses the problem of temporal sentence grounding (TSG), which aims to identify the temporal boundary of a specific segment from an untrimmed video by a sentence query. Previous works either compare pre-defined candidate segments with the query and select the best one by ranking, or directly regress the boundary timestamps of the target segment. In this paper, we propose a novel localization framework that scores all pairs of start and end indices within the video simultaneously with a biaffine mechanism. In particular, we present a Context-aware Biaffine Localizing Network (CBLN) which incorporates both local and global contexts into features of each start/end position for biaffine-based localization. The local contexts from the adjacent frames help distinguish the visually similar appearance, and the global contexts from the entire video contribute to reasoning the temporal relation. Besides, we also develop a multi-modal self-attention module to provide fine-grained query-guided video representation for this biaffine strategy. Extensive experiments show that our CBLN significantly outperforms state-of-the-arts on three public datasets (ActivityNet Captions, TACoS, and Charades-STA), demonstrating the effectiveness of the proposed localization framework.
1. Introduction
Temporal sentence grounding identifies a query-matching segment's boundaries in an untrimmed video, requiring multimodal interaction and contextual reasoning. CBLN addresses limitations of proposal-based and independent-boundary methods by jointly scoring start-end pairs with contextual features.
- Temporal sentence grounding determines the start and end timestamps of a video segment matching a sentence description.
- TSG must model complex vision-language interactions and contextual information for semantic alignment.
- Proposal-ranking methods depend on proposal quality and can disrupt videos' intrinsic temporal structure and global context.
- Independent start and end predictions can confuse timestamps when segments share starting actions but differ in ending actions.
- CBLN jointly scores every possible start-end frame pair with a biaffine mechanism instead of predicting boundaries independently.
- Its multi-context localization module combines multi-scale local and global contexts, while multimodal self-attention produces fine-grained query-guided video features.
- Experiments on ActivityNet Captions, TACoS, and Charades-STA report significant improvements over state-of-the-art methods.
2. Related Work
Related TSG methods either rank query-matched proposals or directly regress temporal boundaries. CBLN instead scores all possible start-end pairs simultaneously and enriches frame-wise interactions with explicit local and global context.
- Early TSG systems generate candidate proposals and rank them by similarity to the language query.
- Proposal-based approaches are constrained by the quality of their generated proposals.
- Later methods directly regress target boundaries, including start-end timestamps or frame-wise distances to boundaries.
- CBLN builds all possible segments from frame pairs and selects the segment with the highest biaffine score.
- Unlike 2D-TAN's segment-level max pooling, CBLN preserves fine-grained frame-wise cross-modal interaction and explicitly models local-global relations.
3. Proposed Method
CBLN localizes temporal sentence grounding segments by scoring every start–end frame pair with biaffine interactions, enriched by context-aware frame features and query-guided video representations.
- Biaffine Localization: Biaffine localization scores all possible start–end frame pairs simultaneously and selects the segment with the highest matching score.Separate start and end representations are produced before biaffine scoring yields a T × T matrix of segment scores; sigmoid outputs indicate query-match probabilities.
- Feature Extractor: CBLN represents a video frame-by-frame and encodes video and query sequences with positional information and bidirectional recurrent networks.The video encoder uses C3D features, positional encoding, and a bidirectional GRU; the query encoder uses GloVe embeddings, positional encoding, and a bidirectional GRU.
- Context-aware Biaffine Localization: Local–global aggregation produces local-guided global and global-guided local contexts, which are concatenated with video representations for biaffine localization.The aggregation re-weights local–global feature pairs, models temporal relations with modified non-local blocks, and projects the combined contexts for each frame.
- Multi-Modal Self Attention: Multi-modal self-attention concatenates each query word with all video frames to capture long-range word-guided dependencies.The model uses one module per query word and average-pools their outputs into a fine-grained query-guided video representation.
- Training: CBLN is trained using scaled Intersection over Union values as supervision for candidate segments.The supplied method description states that each segment’s IoU score is computed for training supervision.
4. Experiments
Experiments evaluate CBLN against prior methods across three datasets and analyze its components, context modeling, aggregation, efficiency, and qualitative grounding behavior.
- Comparisons with state-of-the-arts: CBLN outperforms prior methods on ActivityNet Captions, including DRN by 3.24% for R@1, IoU=0.7 and 13.11% for R@5, IoU=0.7.Compared with 2D-TAN, CBLN improves all reported metrics by 1.06% to 6.89%.
- Comparisons with state-of-the-arts: On TACoS, CBLN achieves the highest R@1 and R@5 scores at IoU=0.5, outperforming 2D-TAN and DRN by a great margin.The dataset is challenging because cooking activities occur in the same kitchen scene with slightly varied objects.
- Comparisons with state-of-the-arts: On Charades-STA, CBLN achieves the highest results across all evaluation metrics.The comparison reports results across different visual feature settings, including the same VGG features.
- Ablation Studies: MMSA improves baseline* by 1.48% in R@1 and 3.77% in R@5, IoU=0.7, while MCBL improves the two metrics by 3.74% and 7.19%.These ablations evaluate fine-grained query-video interaction and local-global context aggregation on ActivityNet Captions.
- Ablation Studies: Max-pooling performs best for global context selection, concatenation performs best for local context extraction, and multiple context scales generally outperform individual scales.The selected global scales are Kg = {1, 2, 4}; adding a fourth scale yields only marginal improvement at substantially higher GPU-memory cost.
- Ablation Studies: Removing stacked non-local blocks reduces R@1 by 2.55% and R@5, IoU=0.7 by 2.53%, while replacing them with concatenation and a linear layer also lowers performance.The full model is less efficient than CMIN but outperforms it by a large margin; compared with 2D-TAN, it is both better and much more efficient.
5. Conclusion
CBLN reformulates temporal sentence grounding as simultaneous biaffine scoring of all start/end index pairs, enriched by multi-scale contexts and query-guided video representations. Experiments on three public datasets show significant improvements over state-of-the-art methods.
- 5. Conclusion: CBLN scores all start/end index pairs simultaneously with a biaffine mechanism for temporal segment localization.This reformulation is designed to model interactions between possible start and end frames.
- 5. Conclusion: Multi-scale local and global contexts enrich the representations of each start and end frame.The model also uses a multi-modal self-attention module to generate fine-grained query-guided video representations.
- 5. Conclusion: CBLN significantly improves over state-of-the-art methods on three public temporal sentence grounding datasets.The evaluated datasets are ActivityNet Captions, TACoS, and Charades-STA.