Source-linked AI summary
Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos
Yitian Yuan, Lin Ma, Jingwen Wang, Wei Liu, Wenwu Zhu
TL;DR
Temporal sentence grounding must localize video content corresponding to a sentence, but prior methods largely overlook the sentence’s role in composing related activities over time. SCDM dynamically modulates hierarchical temporal convolutions with sentence semantics, and experiments on three public datasets show superior grounding performance over state-of-the-art methods.
Problem
Temporal sentence grounding requires locating a video segment matching a sentence, while existing methods neglect sentence guidance for correlating and composing described activities over time.
Method
SCDM dynamically modulates temporal feature maps in a hierarchical temporal convolutional network using sentence semantics, with scaling and shifting parameters guiding temporal composition.
Results
10.71% and 6.26% absolute improvements in R@1,IoU@0.7 over state-of-the-art methods are reported on Charades-STA and ActivityNet Captions, respectively.
Takeaways & Limitations
SCDM better associates and localizes sentence-related video contents by dynamically aligning sentence semantics with diverse video contents.
Abstract
from arXiv · showhide
Temporal sentence grounding in videos aims to detect and localize one target video segment, which semantically corresponds to a given sentence. Existing methods mainly tackle this task via matching and aligning semantics between a sentence and candidate video segments, while neglect the fact that the sentence information plays an important role in temporally correlating and composing the described contents in videos. In this paper, we propose a novel semantic conditioned dynamic modulation (SCDM) mechanism, which relies on the sentence semantics to modulate the temporal convolution operations for better correlating and composing the sentence related video contents over time. More importantly, the proposed SCDM performs dynamically with respect to the diverse video contents so as to establish a more precise matching relationship between sentence and video, thereby improving the temporal grounding accuracy. Extensive experiments on three public datasets demonstrate that our proposed model outperforms the state-of-the-arts with clear margins, illustrating the ability of SCDM to better associate and localize relevant video contents for temporal sentence grounding. Our code for this paper is available at https://github.com/yytzsy/SCDM .
1 Introduction
Temporal sentence grounding identifies the video segment semantically corresponding to a natural-language query. SCDM addresses the limitation that prior methods often match sentence and video representations without using sentence semantics to compose related activities over time.
- Temporal sentence grounding identifies the start and end timestamps of one video segment corresponding semantically to a natural sentence query.
- Existing approaches commonly sample candidate segments, fuse sentence and video representations, and evaluate their matching relationships.Other methods directly fuse sentence information with video clips before composing the fused features using an LSTM or ConvNet.
- Prior methods neglect the sentence’s role in correlating and composing distinct activities that together form one described event.For example, the query links walking across a room with reading on a sofa, helping determine the temporal boundaries of the whole event.
- SCDM uses sentence semantics to modulate temporal convolution operations, dynamically adjusting normalized feature maps to activate more relevant temporal composition.The mechanism adjusts scaling and shifting parameters with reference to sentence semantics.
- The model couples SCDM with a hierarchical temporal convolutional network to exploit sentence–video interactions across multiple temporal granularities.The stated contributions include dynamically correlating sentence-related contents and producing precise temporal boundary predictions.
2 Related Works
Related TSG methods primarily align sentence and video semantics through candidate-segment matching or clip-level fusion followed by temporal composition. The paper instead treats sentence information as a prior for dynamically composing related video contents across temporal granularities.
- Two-stage TSG methods sample candidate segments, fuse each with the sentence, and evaluate matching relationships through integrated features.Some incorporate temporal position features, location regression, memory attention, query-specific proposals, or activity concepts.
- The proposed model contains fully coupled multimodal fusion, semantic modulated temporal convolution, and position prediction components trained end-to-end.Position prediction outputs candidate-segment location offsets and overlap scores based on modulated features.
- Other approaches integrate sentence information with fine-grained video clips and predict boundaries by progressively merging the fusion feature sequence.Examples use frame-by-word interactions through Match-LSTM or graph-based relations among candidate segments.
- Unlike prior alignment-focused methods, the paper uses sentence information as a critical prior to dynamically compose and correlate video contents over time.The design targets diverse video contents represented at various temporal granularities.
3 The Proposed Model
The proposed model combines multimodal fusion, sentence-conditioned temporal convolution, and position prediction in an end-to-end architecture. It uses sentence semantics to dynamically modulate hierarchical temporal features and predicts segments across multiple temporal granularities.
- Overall architecture: The model has three coupled components: multimodal fusion, semantic modulated temporal convolution, and position prediction, trained end-to-end.The components fully couple together rather than operating as separate stages.
- Multimodal fusion: Each video clip interacts with the entire sentence, producing fused features that capture fine-grained sentence–video interactions.The global sentence representation is obtained by averaging word-level representations.
- Temporal convolution: A hierarchical temporal convolutional network produces multi-scale features so feature units correspond to video segments of different durations.With kernel size 3 and stride 2, each convolutional layer halves temporal resolution while expanding the receptive field.
- Semantic conditioned dynamic modulation: SCDM dynamically scales and shifts normalized temporal features using sentence guidance, allowing different temporal units to receive modulation based on their video content.Unlike conditional normalization with shared parameters across a batch or instance, SCDM dynamically evolves its modulation vectors for different temporal units.
- Semantic conditioned dynamic modulation: The resulting semantic-modulated feature maps activate subsequent temporal convolution to better correlate and compose sentence-related contents over time.SCDM is coupled with each temporal convolutional layer.
- Position prediction: Position prediction uses temporal scale ratios to generate candidate segments, then predicts overlap scores and center and width offsets for refinement.Candidates span multiple temporal feature maps and are ranked and refined with non-maximum suppression in one forward pass.
- Training and inference: Training jointly optimizes overlap prediction and location prediction losses, with Smooth L1 applied to location regression for positive samples.Positive candidates have temporal intersection-over-union above 0.5 with the ground-truth segment.
4 Experiments
Experiments evaluate SCDM across three temporal sentence grounding datasets, showing stronger accuracy, efficient inference, and improved semantic-temporal representations.
- Experimental Setup: SCDM is evaluated on TACoS, Charades-STA, and ActivityNet Captions using R@n, IoU@m, the percentage of queries with a top-n segment exceeding IoU m.Videos use 3D convolutional features representing 1-second clips, with dataset-specific input lengths and padding or truncation for temporal convolution.
- Performance Comparison: Ours-SCDM achieves the highest grounding accuracy overall, with absolute R@1,IoU@0.7 improvements of 10.71% on Charades-STA and 6.26% on ActivityNet Captions.Higher-IoU recall is described as more stable under Charades-STA annotation bias, supporting the model’s precise boundary predictions.
- Ablation Studies: Removing SCDM causes dramatic performance degradation, while directly coupling global sentence representations through multiplication or fully connected fusion does not achieve satisfying results.The ablation analysis attributes this to the need for sentence-guided temporal convolution without disrupting visual correlations and temporal dependencies.
- Ablation Studies: Dynamic modulation vectors improve R@5,IoU@0.7 from 54.57% with Ours-SCM to 58.08% with Ours-SCDM on Charades-STA.The analysis links this gain to adapting sentence-word cues to different temporal feature units and diverse video contents.
- Model Efficiency Comparison: Ours-SCDM has the fastest run-time and smallest model size in the efficiency comparison on TACoS.Its hierarchical convolution processes the video in one pass and covers multi-scale segments without sliding-window candidate matching.
- Qualitative Results: Qualitative visualizations show sentence words receiving higher attention in ground-truth regions, while SCDM groups ground-truth clips more tightly in t-SNE feature space.The visualizations associate content-specific word attention with more coherent sentence-related temporal features.
5 Conclusion
The paper concludes that SCDM uses sentence semantics to dynamically modulate temporal convolutions, improving correlation and composition of sentence-related video contents. Experiments on three widely used datasets demonstrate SCDM's superiority for temporal sentence grounding.
- SCDM modulates temporal convolution operations with sentence semantics to correlate and compose sentence-related video contents over time.
- The modulation dynamically evolves across diverse video contents and temporal granularities within the temporal convolution architecture.
- More accurate temporal boundary predictions result from tightly correlating and composing the video contents described by the sentence.
- Experiments on three widely used datasets demonstrate the superiority of SCDM on temporal sentence grounding.