Source-linked AI summary
CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
Zheng Shou, Jonathan Chan, Alireza Zareian, Kazuyuki Miyazawa, Shih-Fu Chang
TL;DR
Temporal action localization needs both action recognition and precise temporal boundaries, but proposal-based methods retain fixed segment boundaries. This paper introduces a CDC network that jointly performs spatial downsampling and temporal upsampling for frame-level predictions, reporting improved labeling and localization precision.
Problem
Temporal action localization must recognize action categories and identify instance boundaries, while proposal-based systems restrict boundaries to fixed segment proposals.
Method
The CDC network places joint convolutional-deconvolutional filters on 3D ConvNets to downsample space, upsample time, and produce dense frame-level scores end-to-end.
Results
The model outperforms state-of-the-art methods in per-frame labeling and temporal action localization, with boundary refinement improving performance especially at high IoU.
Takeaways & Limitations
Frame-level CDC predictions can refine proposal segments into action instances with more precise temporal boundaries.
Takeaways & Limitations
Proposal-based localization remains constrained by fixed segment boundaries, motivating CDC-based boundary refinement beyond segment-level prediction.
Abstract
from arXiv · showhide
Temporal action localization is an important yet challenging problem. Given a long, untrimmed video consisting of multiple action instances and complex background contents, we need not only to recognize their action categories, but also to localize the start time and end time of each instance. Many state-of-the-art systems use segment-level classifiers to select and rank proposal segments of pre-determined boundaries. However, a desirable model should move beyond segment-level and make dense predictions at a fine granularity in time to determine precise temporal boundaries. To this end, we design a novel Convolutional-De-Convolutional (CDC) network that places CDC filters on top of 3D ConvNets, which have been shown to be effective for abstracting action semantics but reduce the temporal length of the input data. The proposed CDC filter performs the required temporal upsampling and spatial downsampling operations simultaneously to predict actions at the frame-level granularity. It is unique in jointly modeling action semantics in space-time and fine-grained temporal dynamics. We train the CDC network in an end-to-end manner efficiently. Our model not only achieves superior performance in detecting actions in every frame, but also significantly boosts the precision of localizing temporal boundaries. Finally, the CDC network demonstrates a very high efficiency with the ability to process 500 frames per second on a single GPU server. We will update the camera-ready version and publish the source codes online soon.
1. Introduction
Temporal action localization must recognize actions and precisely identify their temporal boundaries in untrimmed videos. The paper addresses proposal-boundary limitations with dense frame-level predictions from a CDC network combining semantic abstraction and temporal refinement.
- Temporal action localization requires identifying action categories and the start and end times of each instance.
- Proposal-based localization methods restrict detected boundaries to the pre-determined boundaries of fixed proposal segments.
- Single-frame classifiers and RNNs support fine-grained temporal prediction but do not explicitly model spatio-temporal information in raw videos.
- The CDC filter simultaneously downsamples spatially for semantic abstraction and upsamples temporally for frame-level action prediction.
- The CDC network is trained end-to-end from raw videos to produce dense scores for predicting action instances with precise boundaries.
- The model outperforms state-of-the-art methods in per-frame action labeling and improves temporal localization precision across detection thresholds.
2. Related work
Related work spans frame-level and 3D action recognition, temporal localization, recurrent modeling, and deconvolution-based dense prediction. The paper distinguishes its CDC filter by jointly learning spatial downsampling and temporal upsampling.
- Action recognition and detection: Action-recognition research includes frame-level CNNs trained on ImageNet and 3D convolutional methods for video understanding.
- Action recognition and detection: Spatio-temporal action detection methods localize action regions across frames using supervoxels, tracking, object linking, segmentation, or still images.
- Temporal action localization: Temporal action localization evolved from limited-action settings toward large-scale, complex, fine-grained, and high-level semantic datasets.
- Temporal action localization: RNN-based approaches model temporal state transitions, including proposal generation, decision policies, and multi-label frame annotations.
- De-convolution and semantic segmentation: Deconvolutional methods have been used to reconstruct CNN activations, perform image segmentation, and produce dense video predictions.
- De-convolution and semantic segmentation: Unlike separately stacked convolutional and deconvolutional layers, the CDC filter jointly performs spatial downsampling and temporal upsampling.
3. Convolutional-De-Convolutional networks
CDC networks build on 3D ConvNets to produce frame-level action scores by jointly performing temporal upsampling and spatial downsampling. Their CDC filters preserve high-level semantics while modeling fine-grained temporal dynamics.
- CDC network architecture: 3D ConvNets summarize spatiotemporal patterns but reduce input temporal length, limiting temporal granularity for precise localization.In C3D, layers from conv1a through conv5b reduce temporal length by a factor of 8.
- CDC filter: Transforming FC6 into conv6 lets filters slide across temporally stacked pool5 feature maps and produce outputs at successive time positions.The converted conv6 filter operates on 4x4 pool5 maps and outputs L/8 values in time.
- CDC filter: The separate conv6-plus-deconv6 solution uses 18 parameters, whereas CDC6 integrates the operations through independent convolutional filters on the same input map.CDC6 produces two temporal outputs per 4x4 feature map in the illustrative example.
- CDC filter: CDC filters model temporal dynamics more explicitly because each temporal output uses an independent convolutional kernel rather than sharing one convolutional semantic output.This distinction is presented as making CDC filters more powerful for jointly modeling high-level semantics and temporal dynamics.
- CDC filter: A CDC filter couples convolutional filters over a shared spatial receptive field with one-dimensional temporal deconvolution.This design jointly performs spatial downsampling and temporal upsampling rather than separating them across layers.
- CDC network architecture: In the full architecture, CDC6 upsamples temporal length from L/8 to L/4, while CDC7 and CDC8 each double it back to L before frame-wise softmax scoring.CDC6 also reduces height and width to 1; CDC8 provides K+1 channels for action classes plus background.
4. Experiments
Experiments evaluate CDC for per-frame action labeling and temporal localization on THUMOS’14 and ActivityNet. CDC outperforms comparison methods, with finer temporal predictions improving localization and boundary refinement helping especially at higher IoU.
- 4.1. Per-frame labeling: CDC achieves the best performance among the compared models for per-frame action labeling.The comparison includes single-frame CNN, two-stream CNN, LSTM, and MultiLSTM baselines.
- 4.1. Per-frame labeling: CDC’s joint 3D convolutional and CDC layers outperform alternative upsampling approaches by modeling high-level semantics and temporal action dynamics together.Fixed-weight 3D ConvNets produce a minor performance drop, supporting end-to-end training.
- 4.2. Temporal action localization: CDC outperforms other state-of-the-art methods for temporal action localization across the evaluated IoU thresholds.Temporal localization is evaluated using mAP, with correctness determined by action category and temporal overlap.
- 4.3. Discussions: Increasing temporal granularity from one label every 8 frames to one label per frame progressively increases mAP at IoU 0.5 on THUMOS’14.The evaluated temporal upscaling factors range from x1 to x8.
- 4.3. Discussions: CDC processes approximately 500 frames per second on a 12GB NVIDIA Titan X GPU.The reported speed corresponds to processing a 20-second, 25-FPS video clip within one second.
- 4.3. Discussions: Refining ActivityNet segments with CDC yields significant improvements, particularly when evaluation uses the higher IoU threshold of 0.75.The refinement produces more precise boundaries and larger overlap with ground-truth instances.
5. Conclusion and future works
The paper proposes CDC filters and a CDC network for frame-level action prediction by combining spatial downsampling with temporal upsampling. The model is reported to outperform other methods on per-frame labeling and temporal action localization.
- 5. Conclusion and future works: The CDC filter jointly performs spatial downsampling and temporal upsampling to model semantic and fine-grained temporal information.The CDC network uses this filter to produce dense frame-level scores for precise action boundaries.
- 5. Conclusion and future works: The CDC network significantly outperforms other methods on both per-frame labeling and temporal action localization.The conclusion attributes these tasks to the frame-level action predictions produced by the network.
7. Appendix
The appendix adds implementation details, boundary-refinement analysis, sensitivity experiments, cross-dataset results, visualizations, and deployment caveats for CDC-based localization.
- Boundary refinement: Frame-to-frame gradients of CDC detection scores peak near true action boundaries, supporting fine-grained boundary refinement.The appendix reports high correlation between score differences and true boundaries, and links finer temporal granularity to improved localization.
- Boundary refinement: Boundary refinement extends each proposal, applies CDC frame-level scoring, assigns the highest-average-confidence category, and thresholds scores using Gaussian kernel density estimation.The procedure scans from both ends to retain frames whose class confidence reaches β = µ − σ.
- Implementation details: 32 frames provide a practical window-length trade-off, retaining temporal context while enabling batch size 8 and stable optimization on a 12GB Titan X GPU.The value was selected by grid search over window lengths from 16 to 512 frames.
- Sensitivity analysis: Localization remains stable across reasonable proposal-extension percentages α, balancing wider context against irrelevant frames.Table 4 varies α from 1/8 to 1/4 at IoU 0.5 on THUMOS’14.
- Additional experiments: ActivityNet experiments use proposals with recall 0.681 from 56K proposals at IoU 0.5, while S-CNN is reported as unsuitable for ActivityNet.These proposal and compatibility constraints delimit the appendix’s experimental setup.
- Visualizations: Proposal segments and CDC frame-level scores jointly refine coarse boundaries while helping handle noisy dipped score intervals.The visualizations illustrate refinement for SoccerPenalty and JavelinThrow instances.