Source-linked AI summary
TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
Jiyang Gao, Zhenheng Yang, Chen Sun, Kan Chen, Ram Nevatia
TL;DR
TURN addresses efficient, accurate Temporal Action Proposal generation in long untrimmed videos by reusing video-unit features and jointly scoring proposals with temporal boundary regression. It outperforms prior methods on THUMOS-14 and ActivityNet, runs above 880 FPS, and improves downstream localization on THUMOS-14.
Problem
Temporal Action Proposal generation must achieve high precision and recall while remaining computationally efficient for long untrimmed videos.
Method
TURN decomposes videos into reusable short units, builds multi-scale clip features, and jointly predicts proposal confidence with temporal boundary regression offsets.
Results
TURN outperforms previous state-of-the-art methods on THUMOS-14 and ActivityNet, runs at over 880 FPS with C3D features, and raises THUMOS-14 mAP from 19% to 25.6% at tIoU=0.5.
Takeaways & Limitations
TURN provides a fast proposal-generation stage that improves temporal action localization pipelines and generalizes across datasets and ActivityNet subsets without fine-tuning.
Takeaways & Limitations
A reported comparison number should be higher because DAPs used an incorrect frame rate with S-CNN proposals.
Abstract
from arXiv · showhide
Temporal Action Proposal (TAP) generation is an important problem, as fast and accurate extraction of semantically important (e.g. human actions) segments from untrimmed videos is an important step for large-scale video analysis. We propose a novel Temporal Unit Regression Network (TURN) model. There are two salient aspects of TURN: (1) TURN jointly predicts action proposals and refines the temporal boundaries by temporal coordinate regression; (2) Fast computation is enabled by unit feature reuse: a long untrimmed video is decomposed into video units, which are reused as basic building blocks of temporal proposals. TURN outperforms the state-of-the-art methods under average recall (AR) by a large margin on THUMOS-14 and ActivityNet datasets, and runs at over 880 frames per second (FPS) on a TITAN X GPU. We further apply TURN as a proposal generation stage for existing temporal action localization pipelines, it outperforms state-of-the-art performance on THUMOS-14 and ActivityNet.
1. Introduction
TURN addresses efficient temporal action proposal generation in long untrimmed videos by combining temporal boundary regression with reusable video-unit features. It introduces AR-F and reports strong proposal, efficiency, localization-pipeline, and generalization results.
- Motivation: Temporal action proposals target high precision and recall while maintaining computational efficiency in long untrimmed videos.The task is framed analogously to object proposal generation for rapid detection.
- Motivation: Temporal boundary regression addresses the trade-off between localization accuracy and computation that affects sliding-window proposal methods.The paper notes that temporal boundary regression had not previously been attempted for actions.
- TURN: TURN decomposes videos into short units, reuses unit features across multiscale clips, and jointly predicts proposal confidence with start and end boundary offsets.Each clip becomes a proposal candidate, followed by non-maximum suppression to remove redundancy.
- Evaluation: AR-F evaluates average recall as a function of retrieved proposal frequency and is validated through correlation analysis with action localization performance.The metric is intended to compare methods within datasets and the same method across datasets.
- Results: TURN outperforms previous state-of-the-art methods by a large margin under average-recall metrics on THUMOS-14 and ActivityNet.The reported results include more than 880 FPS with C3D features on a TITAN X GPU.
- Results: Replacing proposals with TURN improves THUMOS-14 localization mAP from 19% to 25.6% at tIoU=0.5, while transfer without fine-tuning shows generalization across datasets.The paper also reports generalization across ActivityNet subsets without fine-tuning.
2. Related Work
Related work spans temporal proposal generation, temporal action localization, and spatio-temporal localization, with object-proposal regression motivating temporal regression for actions.
- Temporal proposal methods include Sparse-prop, S-CNN, and DAPs, alongside broader temporal action-localization approaches.
- Spatio-temporal localization methods combine image-region proposals with RGB and optical-flow action detection.
- Object proposal frameworks use either hand-crafted visual features or deep ConvNet features, with some methods adding anchor-based spatial regression.
- TURN adopts temporal regression for action proposal generation, transferring coordinate-regression ideas from object detection to the temporal domain.
3. Methods
TURN decomposes videos into reusable units, builds pooled multi-scale clip representations with context, and jointly predicts action confidence and temporal boundary offsets.
- 3.1. Video Unit Processing: TURN processes non-overlapping video units as reusable building blocks to avoid repeatedly extracting features for overlapping windows.
- 3.1. Video Unit Processing: Each unit receives a visual representation from an investigated C3D, optical-flow CNN, or RGB CNN encoder.
- 3.2. Clip Pyramid Modeling: A clip consists of contiguous units, while separately pooled internal and surrounding context features are concatenated into its final representation.
- 3.2. Clip Pyramid Modeling: TURN scans anchor units with clip pyramids containing windows at multiple temporal resolutions, reusing unit-level features despite clip overlap.
- 3.3. Unit-level Temporal Coordinate Regression: The unit regression model takes clip features and outputs an action confidence score plus temporal coordinate regression offsets.
- 3.3. Unit-level Temporal Coordinate Regression: Regression is performed at unit level with direct start and end offsets rather than frame-level coordinates or coordinate parametrization.
- 3.4. Loss Function: Training assigns positive clips using highest tIoU overlap or tIoU above 0.5, and combines classification with regression losses.
- 3.4. Loss Function: The L1 regression loss is computed only for positive samples, with a background-to-positive mini-batch ratio of 10.
4. Evaluation
The evaluation compares TAP metrics, TURN variants, computational trade-offs, and downstream localization performance on THUMOS-14 and ActivityNet. Results support AR-F and show that temporal context, unit regression, and feature choices affect proposal quality, speed, and generalization.
- Evaluation metrics: AR-F measures average recall as a function of retrieved proposals per second, while AR-N and AR-AN vary retrieval count differently.For a video of length l_i, proposal frequency F retrieves R_i = F l_i proposals.
- Evaluation metrics: The correlation coefficient between mAP and AR-F remains above 0.9 across THUMOS-14 video-length groups.The analysis evaluates correlations between TAP metrics and localization performance across four video-length groups.
- Evaluation metrics: AR-F satisfies both evaluation requirements: comparing methods on one dataset and comparing the same method across datasets.AR-N is affected by video length, whereas AR-AN depends on dataset video-length distributions.
- TURN variants: Dense flow CNN features give TURN-FL the best results, C3D remains competitive, and RGB CNN features perform worse.The comparison indicates that optical flow captures temporal action information effectively.
- TURN variants: The full unit-regression model with temporal context is compared against binary-classification and frame-regression variants using AR-F curves.The four variants isolate the effects of temporal context and coordinate regression.
- Runtime and unit size: Smaller units increase recall and temporal coordinate precision but reduce speed because more unit-level features must be extracted.Sixteen- and thirty-two-frame units represent approximately half a second and one second, respectively.
- Runtime and unit size: TURN-C3D-16 exceeds state-of-the-art AR while accelerating computation by more than 6 times, whereas TURN-FL-16 achieves the highest AR with competitive runtime.TURN runs above 880 FPS with C3D features and 260 FPS with flow CNN features on one TITAN X GPU.
- Downstream localization: TURN proposals improve temporal action localization, with mAP increasing from 19% to 25.6% at tIoU=0.5 when only proposals change.TURN also outperforms other proposal methods with both SVM classifiers and the S-CNN localizer, and exceeds them across tIoU thresholds with S-CNN.
5. Conclusion
The paper concludes that TURN provides fast temporal action proposal generation through unit regression and feature reuse. It introduces AR-F, reports strong proposal performance and generalization, and demonstrates benefits in temporal action localization pipelines.
- TURN is a novel method for fast temporal action proposal generation.The conclusion presents TURN as effective for this task.
- AR-F is proposed as a TAP metric that correlates robustly with temporal action localization and supports comparisons across datasets.The metric evaluates average recall against proposal frequency.
- TURN exceeds 880 FPS while maintaining state-of-the-art average recall performance.The reported speed uses a TITAN X GPU.
- TURN remains robust with both C3D and dense flow CNN visual features and improves proposal-generation stages in localization pipelines.The conclusion reports demonstrations on THUMOS-14 and ActivityNet.