Source-linked AI summary
Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction
Jingwen Wang, Lin Ma, Wenhao Jiang
TL;DR
Temporally grounding language queries requires precise localization of the video segment matching a sentence, a limitation of prior fixed-window and anchor-based methods. CBP predicts anchors and semantic boundaries jointly while using contextual information to improve boundary detection. It achieves compelling performance across three public datasets, with reported gains on high-IoU localization metrics.
Problem
Prior video-grounding methods often overlook precise segment boundaries, while fixed windows and anchor-based approaches have limitations for accurate localization.
Method
CBP jointly predicts temporal anchors and semantic boundaries, modulating anchor predictions with boundary scores and integrating contextual information through self-attention.
Results
CBP achieves compelling performance on three public datasets and outperforms competitors with a clear margin.
Takeaways & Limitations
Boundary-aware prediction improves grounding precision, particularly at higher intersection-over-union thresholds.
Takeaways & Limitations
Experiments use the same settings for all baselines, including learning rate, segment sampling, NMS threshold, and other hyperparameters.
Abstract
from arXiv · showhide
The task of temporally grounding language queries in videos is to temporally localize the best matched video segment corresponding to a given language (sentence). It requires certain models to simultaneously perform visual and linguistic understandings. Previous work predominantly ignores the precision of segment localization. Sliding window based methods use predefined search window sizes, which suffer from redundant computation, while existing anchor-based approaches fail to yield precise localization. We address this issue by proposing an end-to-end boundary-aware model, which uses a lightweight branch to predict semantic boundaries corresponding to the given linguistic information. To better detect semantic boundaries, we propose to aggregate contextual information by explicitly modeling the relationship between the current element and its neighbors. The most confident segments are subsequently selected based on both anchor and boundary predictions at the testing stage. The proposed model, dubbed Contextual Boundary-aware Prediction (CBP), outperforms its competitors with a clear margin on three public datasets. All codes are available on https://github.com/JaywongWang/CBP .
1 Introduction
Temporally grounding language queries requires jointly understanding video and language while precisely localizing activity boundaries. CBP addresses this challenge with boundary prediction and contextual modeling alongside anchor predictions.
- Temporally grounding localizes the activity corresponding to a language query within a video.The task requires integrating video and sentence information to distinguish fine-grained segment details and localize the target segment.
- Sliding-window methods use predefined window sizes, which can miss varied-duration segments and produce inaccurate temporal boundaries.Their fixed search windows also introduce redundant computation.
- Anchor-based and threshold-based approaches can lose boundary information, limiting precise segment localization.Segments are labeled positive or negative according to overlap with the ground truth using a predefined threshold.
- CBP jointly predicts temporal anchors and semantic boundaries at each time step with small computational overhead.At prediction time, boundary scores modulate anchor predictions to produce boundary-aware grounding results.
- CBP adaptively integrates contextual information to detect semantic boundaries more accurately.Its self-attention module measures contributions from contextual elements using language-video interactions.
2 Related Work
Related work studies language grounding in images and videos. Image grounding retrieves regions matching expressions, while video grounding extends localization to temporal segments using multimodal representations and localization predictions.
- Grounding Language Queries in Images: Image grounding localizes the image region corresponding to a language query, often after generating candidate regions.Prior methods select regions using description reconstruction error or probabilities.
- Grounding Language Queries in Images: Image-grounding methods incorporate global context, other candidate regions, and relationships among regions and phrases.These contextual signals supplement direct region-phrase matching.
- Temporally Video Grounding: Temporally video grounding extracts the video segment corresponding to a language query and has expanded from constrained settings to general scenarios.Recent work jointly models video clips and text queries, then predicts alignment scores and location offsets.
3 Proposed Method
CBP integrates query-video interaction, contextual integration, and localization to jointly predict temporal anchors and semantic boundaries. Boundary-aware score fusion then ranks and suppresses candidate segments for precise grounding.
- Framework: CBP combines query-video interaction, contextual integration, and localization modules in an end-to-end architecture.The localization module contains separate anchor and boundary submodules.
- Query-Video Interaction: Match-LSTM integrates textual information, video motion, and long-term dependencies before producing query-video representations.Each video frame attends to different query words, and the interaction LSTM summarizes the integrated signals.
- Contextual Integration: Self-attention models relationships among query-video representations to collect contextual localization clues while preserving temporal dependencies.The contextual integration module operates after query-video integration and concatenates contextual and interaction representations.
- Localization Module: K predefined anchors produce confidence scores for segments ending at each time step and spanning different temporal durations.Anchor lengths are predefined, and each hidden state feeds K independent binary classifiers.
- Localization Module: A parallel boundary branch predicts whether each temporal position corresponds to a semantic activity boundary.The boundary score measures confidence that the LSTM is passing through a boundary marking an activity start or end.
- Boundary-aware Prediction: At inference, boundary scores modulate anchor scores, after which high-scoring candidates are globally ranked and redundant segments removed with NMS.NMS does not affect the top-1 result.
4 Experiments
CBP is evaluated against existing methods on three public datasets using standard recall and mIoU metrics, with controlled baseline settings. Experiments include benchmark comparisons, ablations, and qualitative analyses of boundary and contextual integration.
- Datasets and setup: Experiments use TACoS, Charades-STA, and ActivityNet Captions, with identical settings for all baselines.Controlled settings include learning rate, segment sampling, NMS threshold, and other hyperparameters.
- Datasets and setup: Evaluation reports R@N, IoU=θ and mIoU, measuring thresholded top-N recall and average top-1 IoU.R@N, IoU=θ is the percentage of queries whose top N results include a segment exceeding θ IoU; mIoU averages top-1 IoU.
- Benchmark comparison: 18.87% R@1, IoU=0.7 on TACoS surpasses the previous best result of 15.80%.Xu et al. achieves better recall for R@5, IoU=0.5 on TACoS.
- Benchmark comparison: 17.04% R@1, IoU=0.7 on ActivityNet Captions exceeds Xu et al. at 13.60% and TripNet at 13.93%.CBP surpasses TGN and Xu et al. on all reported metrics, while its relative advantage is smaller at lower IoUs.
- Ablation study: The boundary module substantially improves high-IoU performance on TACoS, and adding contextual integration makes CBP outperform the other tested configurations.Replacing the contextual or boundary modules with competing designs causes performance degeneration.
- Qualitative analysis: Qualitative examples show that combining boundary predictions with anchors improves localization, while contextual integration further recognizes desired segment boundaries.The highest context weight can correspond to a non-nearest segment, such as a precursor action or related visual content.
5 Conclusion
CBP addresses temporal language grounding with a single-stream, contextual boundary-aware model, whose effectiveness was demonstrated on three widely used datasets.
- CBP processes each video in a single pass using boundary prediction to improve temporal grounding precision.The model combines a single-stream architecture with a simple boundary-prediction idea.