Source-linked AI summary
Gaussian Temporal Awareness Networks for Action Localization
Fuchen Long, Ting Yao, Zhaofan Qiu, Xinmei Tian, Jiebo Luo, Tao Mei
TL;DR
Temporal action localization must handle actions whose durations and temporal structures vary, while predetermined proposal scales limit existing one-stage approaches. GTAN learns Gaussian kernels to dynamically model proposal intervals and context within a one-stage framework. Experiments on THUMOS14 and ActivityNet v1.3 report performance gains over baselines and state-of-the-art methods.
Problem
Predetermined temporal scales in one-stage action localization limit detection of proposals with varied lengths and complex temporal variations.
Method
GTAN learns Gaussian kernels on feature-map cells, mixes overlapping kernels for varied-length actions, and uses Gaussian pooling for contextual aggregation.
Results
GTAN consistently boosts performance across metrics and datasets, including AR improvements over BSN of 1.1% on THUMOS14 and 0.6% on ActivityNet.
Takeaways & Limitations
Learning temporal structure and aggregating contextual information provides a one-stage approach for localizing action proposals with various lengths.
Abstract
from arXiv · showhide
Temporally localizing actions in a video is a fundamental challenge in video understanding. Most existing approaches have often drawn inspiration from image object detection and extended the advances, e.g., SSD and Faster R-CNN, to produce temporal locations of an action in a 1D sequence. Nevertheless, the results can suffer from robustness problem due to the design of predetermined temporal scales, which overlooks the temporal structure of an action and limits the utility on detecting actions with complex variations. In this paper, we propose to address the problem by introducing Gaussian kernels to dynamically optimize temporal scale of each action proposal. Specifically, we present Gaussian Temporal Awareness Networks (GTAN) --- a new architecture that novelly integrates the exploitation of temporal structure into an one-stage action localization framework. Technically, GTAN models the temporal structure through learning a set of Gaussian kernels, each for a cell in the feature maps. Each Gaussian kernel corresponds to a particular interval of an action proposal and a mixture of Gaussian kernels could further characterize action proposals with various length. Moreover, the values in each Gaussian curve reflect the contextual contributions to the localization of an action proposal. Extensive experiments are conducted on both THUMOS14 and ActivityNet v1.3 datasets, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GTAN achieves 1.9% and 1.1% improvements in mAP on testing set of the two datasets.
1. Introduction
Temporal action localization seeks exact action boundaries and categories in video, but fixed temporal scales can miss actions with varied temporal structure. GTAN addresses this by learning Gaussian kernels within a one-stage architecture.
- Temporal action localization identifies action start and end timestamps while recognizing actions across categories.
- One-stage methods adapt image detectors such as SSD and Faster R-CNN to predict action proposals in a 1D temporal sequence.
- Fixed temporal scales for feature-map cells limit detection of actions whose temporal structure or duration varies.The same ground-truth action can be detected as multiple proposals when its interval does not align with the fixed scale.
- GTAN learns a Gaussian kernel for each feature-map cell to dynamically indicate a particular action-proposal interval.The architecture uses a 3D ConvNet backbone, temporal convolutions, and cascaded anchor layers to generate feature maps at different resolutions.
- GTAN is presented as a one-stage architecture that leverages temporal structure and contextual information for localizing actions with various lengths.
2. Related Work
Prior work separates temporal proposal generation from action classification or uses one-stage detectors with predetermined temporal scales. GTAN remains one-stage while learning temporal structure with Gaussian kernels and leveraging contextual information.
- Temporal action proposal methods localize video segments containing actions, whereas temporal action detection additionally classifies those segments into known categories.
- Content-independent proposal methods sample videos uniformly or with sliding windows, creating substantial computation for later classification.
- Two-stage detection separates proposal generation from classification, which may produce sub-optimal solutions.
- GTAN belongs to one-stage temporal action detection but learns temporal structure instead of predetermining proposal scales.It also studies how contextual information can be better leveraged for action localization.
3. Gaussian Temporal Awareness Networks
GTAN is a one-stage action-localization architecture that replaces fixed temporal scales with learned Gaussian kernels, enabling proposals and contextual features to adapt to varied action durations.
- Gaussian Kernel Learning: Predetermined temporal scales can miss actions whose intervals vary or whose temporal structure is complex.Conventional anchor layers may align with some receptive fields but struggle with proposals having diverse lengths.
- Gaussian Kernel Learning: Each feature-map cell learns a Gaussian kernel whose temporal scale dynamically represents a particular action-proposal interval.The kernel is learned from the cell feature through a 1D convolution and constrained by a sigmoid operation.
- Gaussian Kernel Grouping: Overlapping Gaussian kernels are grouped when their temporal IoU exceeds threshold ε, producing mixed kernels for longer proposals.The grouping procedure iteratively compares adjacent kernels and merges them when the overlap criterion is met.
- Gaussian Pooling: Gaussian pooling computes weighted feature sums from kernel values, so contextual contributions around proposals are adaptively incorporated.The aggregated representation is used for action classification and temporal boundary regression, unlike manual extension with average pooling.
- Prediction and Optimization: GTAN predicts action classes, refined temporal centers and widths, and overlap scores using parallel convolutional layers.The network is trained end-to-end with classification, localization, and overlap losses, including foreground and background IoU thresholds of 0.8 and 0.3.
4. Experiments
Experiments evaluate GTAN for temporal action proposals and localization on THUMOS14 and ActivityNet v1.3, showing consistent gains from learned Gaussian temporal structure and grouping.
- Temporal action proposal: GTAN improves proposal quality across metrics and datasets, achieving AR of 54.3% on THUMOS14 and 74.8% on ActivityNet.These results improve over BSN by 1.1% and 0.6%, respectively.
- Gaussian kernel and grouping: Gaussian grouping further increases mAP by 1.1% on THUMOS14 and 2.7% on ActivityNet v1.3.Its benefit is larger for long action proposals, supporting mixtures of kernels for actions with arbitrary length.
- Comparisons with state-of-the-art: On THUMOS14, GTAN reaches 37.9% mAP@0.5 with C3D, exceeding SSAD and SS-TAD by 13.3% and 8.7%.GTAN also surpasses BSN by 1.0% with C3D and 1.9% with a P3D backbone.
- Comparisons with state-of-the-art: On ActivityNet testing, GTAN surpasses BSN by 1.1% in temporal action detection mAP.The validation-set improvement is 0.6%.
- Visualization: Visualization shows Gaussian kernels capturing proposal temporal structure and predicting default boxes for final regression and classification.The kernels and grouping are learned from the conv_a5 outputs.
5. Conclusions
GTAN explores action temporal structure by learning Gaussian kernels that dynamically predict proposal scales and can represent actions of varying length. Experiments on THUMOS14 and ActivityNet v1.3 validate the approach against advanced one-stage and two-stage techniques.
- GTAN learns Gaussian kernels to dynamically predict the temporal scale of each action proposal.Each anchor-layer cell measures one Gaussian kernel.
- Multiple Gaussian kernels can be mixed to represent action proposals with various lengths.
- Gaussian pooling leverages contextual information to enhance action-proposal features for regression and classification.
- Experiments on THUMOS14 and ActivityNet v1.3 show performance improvements over advanced one-stage and two-stage techniques.