Source-linked AI summary
Temporal Context Network for Activity Localization in Videos
Xiyang Dai, Bharat Singh, Guyue Zhang, Larry S. Davis, Yan Qiu Chen
TL;DR
Temporal activity localization must identify accurate boundaries in untrimmed videos, yet proposal features confined to a segment do not provide enough information for boundary prediction. TCN ranks uniformly placed, multi-scale proposals using pairwise proposal-and-context features processed by a temporal convolution network, then classifies the top proposals. It outperforms state-of-the-art methods on ActivityNet and THUMOS14 and shows that temporal context improves localization.
Problem
Features confined to a proposal lack sufficient information to predict activity boundaries and distinguish good from bad temporal candidates.
Method
TCN places proposals uniformly at multiple temporal scales, samples fixed-length features at proposal and adjacent larger-context scales, ranks them, and classifies the top proposals.
Results
TCN outperforms state-of-the-art methods on proposal metrics for ActivityNet and THUMOS14, while context improves boundary localization.
Takeaways & Limitations
Temporal context is helpful and critical for precise activity localization, particularly when evaluating boundary overlap at higher IoU.
Abstract
from arXiv · showhide
We present a Temporal Context Network (TCN) for precise temporal localization of human activities. Similar to the Faster-RCNN architecture, proposals are placed at equal intervals in a video which span multiple temporal scales. We propose a novel representation for ranking these proposals. Since pooling features only inside a segment is not sufficient to predict activity boundaries, we construct a representation which explicitly captures context around a proposal for ranking it. For each temporal segment inside a proposal, features are uniformly sampled at a pair of scales and are input to a temporal convolutional neural network for classification. After ranking proposals, non-maximum suppression is applied and classification is performed to obtain final detections. TCN outperforms state-of-the-art methods on the ActivityNet dataset and the THUMOS14 dataset.
1. Introduction
Temporal activity localization is needed for untrimmed videos containing multiple activities, but proposal ranking based only on segment-internal features cannot reliably identify activity boundaries. TCN addresses this gap by ranking multi-scale proposals with features sampled from both the proposal and a larger temporal context.
- Motivation: Temporal localization is needed because untrimmed videos can contain multiple activities with distinct beginning and end points.The task supports video retrieval, browsing lengthy videos, and surveillance applications.
- Motivation: Activity localization has received less attention than image object localization because video localization is computationally expensive and large annotated datasets were previously unavailable.
- Motivation: Single-scale proposal features can support recognition but lack the broader temporal information needed to determine whether proposal boundaries align with an activity.Two proposals fully contained within one activity can differ in quality even when their internal features are similar.
- Approach: TCN places proposals uniformly across a video at multiple temporal scales and ranks them using pairwise features sampled at the proposal and a larger context scale.A temporal convolution network learns background and foreground probabilities before top-ranked proposals enter classification.
- Approach: The ranker selects proposals for classification after pairwise sampling captures context boundary information that a typical sliding-window representation can miss.
2. Related Work
Prior work spans hand-crafted trajectory representations, spatio-temporal proposals, temporal grouping, recurrent models, and category-independent proposal pipelines. TCN is positioned among temporal proposal methods while emphasizing that existing temporal localization methods do not employ temporal context.
- Feature-based methods: Dense trajectories encoded with Fisher vectors were widely used for trimmed activity recognition, while PSDF represented segments in action-detection pipelines.
- Proposal methods: Spatio-temporal proposal methods typically target short videos and emphasize spatial rather than temporal localization.They also require training data containing spatio-temporal annotations.
- Temporal segment methods: Temporal segment methods include sparse-dictionary encoding, fragment hierarchies, and multi-stream recurrent networks with tracking.
- Proposal-ranking methods: Category-independent approaches use proposal, classification, and localization networks or recurrent encoders to generate and score segments in long videos.The cited comparison includes Shou et al.'s segment-based 3D ConvNets and DAPs' LSTM encoding.
- Open gap: Existing methods for temporal activity localization do not employ temporal context, although context is used in other computer-vision and video-recognition tasks.The paper identifies temporal context as critical for this localization problem.
3. Approach
TCN generates multi-scale temporal proposals and ranks them with features that explicitly capture context beyond each proposal. Ranked proposals are then classified using a bilinear feature representation and a 201-way Softmax classifier.
- Approach: TCN generates a ranked list of temporal segments, with each segment defined by its beginning and end frames.The ranked segments are produced for a video V consisting of T frames.
- Proposal Generation: At each sliding-window position, TCN generates K proposals at different scales, whose durations increase by powers of two.The fixed-length windows have 50% overlap, and proposals reaching video boundaries are zero-padded.
- Context Feature Representation: TCN samples a fixed number of features from a proposal and its next, larger centered scale instead of using average or max-pooling.This paired-scale sampling explicitly captures temporal context across different proposal durations.
- Context Feature Representation: Context is critical because features inside a proposal alone cannot determine activity endings or distinguish good candidates when proposals lie within activity intervals.The larger scale reveals how much surrounding background the current and next proposals cover.
- Sampling and Temporal Convolution: A temporal convolutional network processes features from consecutive scales separately, concatenates them, and uses fully connected layers to predict proposal-versus-background labels.A two-way Softmax loss maps the resulting representation to proposal labels.
- Classification: For high-scoring proposals, bilinear pooling aggregates features inside the segment, followed by a fully connected layer and a 201-way Softmax classifier.The 201 classes comprise 200 action classes plus background; proposal labels use IoU thresholds above 0.7 and below 0.3.
4. Experiments
Experiments on ActivityNet and THUMOS14 evaluate the proposal ranker, temporal context, context scale, proposal count, classification components, and comparisons with prior methods. Results show that context-based ranking improves precise localization and achieves strong proposal performance with relatively few proposals.
- Experiments evaluate the temporal context network on ActivityNet and THUMOS14.
- ActivityNet experiments: The proposal pyramid provides better coverage than the challenge baseline with 64 proposals instead of 90.
- ActivityNet experiments: 90% recall at IoU 0.5 is reached with 20 proposals, while higher-IoU recall changes little as proposal count increases.
- ActivityNet experiments: Temporal context is critical for precise boundaries: without it, single-proposal recall drops significantly above IoU 0.5, while detection mAP is much higher with context.
- ActivityNet experiments: Context scale performs best up to scale 2; changing scale from 2 to 3 drops performance by 3%, whereas changing it from 1.5 to 1 decreases mAP by 15%.
- THUMOS14 experiments: On THUMOS14, the proposal ranker consistently outperforms SCNN and DAPs, particularly with small proposal counts, and improves significantly with paired context windows.
5. Qualitative Results
Qualitative results show that temporal context produces more accurate activity boundaries than ranking without context. Without context, nearby proposals can fall inside or near ground-truth intervals and become false positives.
- Without temporal context, multiple proposals appear inside or near ground-truth intervals, producing inaccurate boundaries.These nearby detections are marked as false positives when detection metrics are computed.
- With temporal context, proposal boundaries are significantly more accurate than without context.
- Top-5 qualitative visualizations compare ground-truth intervals with predicted proposals.Ground truth is shown in blue, while predictions are shown in green.
6. Conclusion
The paper demonstrates that temporal context helps localize activities in videos and reports state-of-the-art performance on two challenging public datasets. Its qualitative visualization uses top-ranked proposals with ground truth shown in blue and proposals in green.
- Temporal context is helpful for performing activity localization in videos.
- The analysis studies temporal proposals through precision-recall characteristics at multiple overlap thresholds and varying context windows.
- Top-5 ranking visualizations show ground truth in blue and proposals in green.