PaperlayerView exact revision on arXiv

Source-linked AI summary

Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos

Rui Hou, Chen Chen, Mubarak Shah

arXiv:1703.10664v3cs.CV

TL;DR

Action detection requires spatial and temporal localization, but prior approaches use frame-level proposal linking and separate spatial-temporal networks. T-CNN unifies proposal generation, tube linking, localization, and recognition with 3D convolution features and ToI pooling. Experiments report state-of-the-art performance on trimmed and untrimmed video datasets.

  • Problem

    Action detection must localize every action occurrence in space and time, while prior video methods have limited deep-learning impact and commonly separate proposal association and spatial-temporal processing.

  • Method

    T-CNN divides videos into equal-length clips, generates tube proposals with 3D ConvNet features, links them, and applies ToI pooling for unified localization and recognition.

  • Results

    T-CNN achieves state-of-the-art action-detection performance on trimmed UCF-Sports, J-HMDB, and UCF-101 videos and untrimmed THUMOS’14 videos.

  • Takeaways & Limitations

    The unified 3D framework supports spatio-temporal action localization and recognition, including in untrimmed videos.

  • Takeaways & Limitations

    Videos are processed as fixed-length 8-frame clips to accommodate the variable temporal dimension and reduce GPU-memory cost.

Abstract

from arXiv · show

Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis (e.g. action detection and recognition) has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal detection and association of proposals across frames. Also, these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and for each clip a set of tube proposals are generated next based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals. Extensive experiments on several video datasets demonstrate the superior performance of T-CNN for classifying and localizing actions in both trimmed and untrimmed videos compared to state-of-the-arts.

1. Introduction

Action detection must find every action occurrence and localize it in space and time, yet prior deep-learning approaches separate frame proposals from cross-frame association and spatial from temporal processing. T-CNN unifies proposal generation, linking, localization, and recognition with 3D convolution features, achieving state-of-the-art performance across trimmed and untrimmed datasets.

  • Action detection requires finding every occurrence in long videos and localizing each detection spatially and temporally.
  • Prior approaches detect frame-level proposals, associate them across frames, and separately process spatial and motion information with two-stream networks.
  • T-CNN is an end-to-end action-detection framework using 3D ConvNet features to capture spatio-temporal information in a unified network.
  • Tube proposals are generated for equal-length clips, linked using actionness and proposal overlap, and pooled with ToI pooling for action-label prediction.
  • ToI pooling generalizes RoI pooling to 3D tubes, handles variable spatial and temporal proposal sizes, and greatly improves recognition results.
  • T-CNN achieves state-of-the-art action-detection performance on trimmed UCF-Sports, J-HMDB, and UCF-101 videos and untrimmed THUMOS’14 videos.

2. Related Work

Related work includes frame-based CNN, recurrent, two-stream, 3D CNN, proposal-linking, and R-CNN-derived approaches for video recognition and detection. T-CNN addresses limitations of separate spatial-temporal processing and extends R-CNN concepts to 3D video tubes.

  • Frame-based CNN and recurrent approaches for action recognition neglect temporal information, while two-stream CNNs process image and optical-flow information separately.
  • 3D CNNs capture spatio-temporal information and had been applied to video recognition and human detection, but T-CNN reports the first use of 3D CNN for action detection.
  • Deep action-detection methods commonly generate frame-level proposals and link them across frames, but their separate spatial-temporal processing does not fully explore temporal consistency.
  • R-CNN methods use region proposals and fixed-size pooling for image detection, motivating analogous proposal and pooling operations for video tubes.

3. Generalizing R-CNN from 2D to 3D

The paper extends R-CNN-style processing to 3D video tubes by using fixed-length clips, 3D convolutions, and Tube-of-Interest pooling for variable-sized spatio-temporal proposals.

  • 3. Generalizing R-CNN from 2D to 3D: Fixed-length 8-frame clips make variable-duration videos compatible with a fixed-size ConvNet and reduce GPU-memory cost.Videos vary in temporal dimension, so the method processes them clip by clip.
  • 3. Generalizing R-CNN from 2D to 3D: 3D convolutions and pooling capture motion across time and space while concentrating distinguishable video information.Temporal skip pooling is then used to retain temporal order lost through temporal pooling.
  • 3. Generalizing R-CNN from 2D to 3D: Tube-of-Interest pooling generalizes RoI pooling from 2D regions to 3D tubes and produces fixed-size outputs despite variable tube dimensions.It accepts a feature cube and a sequence of per-frame bounding boxes, then pools spatially and temporally.
  • 3. Generalizing R-CNN from 2D to 3D: ToI pooling performs spatial max pooling into H × W bins before dividing the pooled feature maps temporally into D bins.The separate spatial and temporal operations accommodate boxes with different sizes, aspect ratios, and positions.
  • 3. Generalizing R-CNN from 2D to 3D: During back-propagation, each pooled output sends its gradient to the input activation selected by argmax, while shared inputs accumulate gradients.The formulation uses f(·) to represent the argmax selection.

4. T-CNN Pipeline

T-CNN processes clips with a 3D ConvNet and Tube Proposal Network, links proposals across clips using actionness and overlap, and classifies the resulting variable-sized sequences with ToI pooling.

  • 4. T-CNN Pipeline: The end-to-end T-CNN takes video clips as input, generates tube proposals with a Tube Proposal Network, and uses linked sequences for detection and recognition.The linked proposal sequence represents a spatio-temporal action detection in the video.
  • 4. T-CNN Pipeline: The 3D ConvNet extracts spatio-temporal feature cubes from 8-frame clips using seven convolutional and four max-pooling layers.Convolution and pooling operate in both spatial and temporal dimensions.
  • 4. T-CNN Pipeline: After conv5 reduces the temporal depth to one frame, bounding-box proposals are generated from the conv5 feature cube.The proposal generation follows the faster R-CNN design.
  • 4. T-CNN Pipeline: The Tube Proposal Network learns 12 dataset-adaptive anchor boxes with k-means instead of using hand-picked scales and aspect ratios.Bounding boxes are also filtered using an actionness threshold.
  • 4. T-CNN Pipeline: Temporal skip pooling maps positive conv5 proposals to the eight-slice conv2 feature tube to restore temporal order for frame-level detection.This compensates for temporal order lost when eight frames are concentrated into one conv5 frame.
  • 4.2. Linking Tube Proposals: Network flow links proposals from different clips into video tube proposals, using actionness and adjacent-proposal overlap as linking criteria.The score averages proposal actionness and overlap across consecutive clips, and the highest-scoring sequences are selected.
  • 4. T-CNN Pipeline: ToI pooling converts linked sequences with different sizes into fixed-length feature vectors before fully connected classification into N action classes plus background.The classification head follows ToI pooling with two fully connected layers and dropout.

5. Experiments

T-CNN is evaluated on trimmed and untrimmed action-detection benchmarks using an end-to-end 3D ConvNet pipeline with tube proposals, linking, and non-maximum suppression. It achieves strong performance across datasets, while hard-negative mining further improves detection on THUMOS’14.

  • Datasets: T-CNN is evaluated on UCF-Sports, J-HMDB, UCF-101, and THUMOS’14, covering trimmed and untrimmed videos.UCF-Sports, J-HMDB, and UCF-101 are trimmed datasets; THUMOS’14 contains untrimmed videos.
  • Implementation: The pipeline divides videos into overlapping 8-frame training clips and nonoverlapping 8-frame testing clips, then links 40 highest-confidence tube proposals before NMS.The TPN and recognition network share common-layer weights, and testing uses zero-padding when needed.
  • UCF-Sports: T-CNN clearly outperforms competing methods on UCF-Sports ROC and AUC curves at overlap criterion α = 0.2.Its video mAP reaches 95.2 at α = 0.2 and α = 0.5, compared with Peng et al.’s reported 94.8 and 94.7, respectively.
  • UCF-101: T-CNN yields the best reported performance on UCF-101 and is compared using frame-mAP, video-mAP (α = 0.2), and video-mAP (α = 0.5).The UCF-101 detection subset contains 24 action classes and 3,207 videos with spatiotemporal annotations.
  • THUMOS’14: Negative sample mining improves THUMOS’14 detection beyond both the Sultani et al. baseline and T-CNN without negative mining.Hard negatives are selected from high-scoring boxes in negative clips containing unrelated actions.
  • Qualitative results: Qualitative results show highest-scoring detected action tubes across UCF-Sports, JHMDB, UCF-101, and THUMOS’14.Figure 6 overlays predicted labels, red detections, and green ground-truth boxes.

6. Discussion

The discussion examines ToI pooling and temporal skip connections as components affecting video recognition and localization. Results indicate that whole-video optimization and preserved temporal order improve spatio-temporal modeling.

  • ToI Pooling: ToI pooling replaces C3D’s fifth 3D max-pooling layer and produces one feature vector per video for end-to-end recognition.The comparison uses video-level accuracy on UCF-101.
  • ToI Pooling: Whole-video optimization enables T-CNN to better capture spatio-temporal information than C3D’s clip-based optimization.The ToI pooling layer can also be combined with other deep-learning pipelines such as two-stream CNN.
  • Temporal Skip Connection: Temporal skip pooling preserves temporal order and detailed motion, addressing localization failures caused by using only conv5 features.Without the skip connection, bounding-box regression over eight frames performs substantially worse.
  • Runtime: For a 40-frame video, tube proposal generation takes 1.1 seconds, linking takes 0.03 seconds, and action-label prediction takes 0.9 seconds on one GTX Titan X.

7. Conclusion

The paper concludes that T-CNN unifies action localization and recognition through 3D convolutional features and linked tube proposals. Experiments support its effectiveness for spatio-temporal action localization, including untrimmed videos.

  • Conclusion: T-CNN uses a unified end-to-end framework to extract spatio-temporal features and perform action localization and recognition.Coarse proposal boxes are densely sampled from the 3D convolutional feature cube.
  • Conclusion: Experiments on several benchmark datasets demonstrate T-CNN’s strength for spatio-temporal action localization, even in untrimmed videos.
  • Conclusion: Linked tube proposals support both action recognition and localization within the same detection pipeline.

8. Appendix

The appendix adds temporal skip-pooling analyses and qualitative detections, including videos outside the benchmark datasets. These examples examine performance under detailed motion, crowded backgrounds, camera motion, and noise.

  • Analysis of Temporal Skip Pooling: The appendix compares temporal feature-cube connections while keeping all other parameters fixed.The connected feature cubes are denoted C5 and its combinations with earlier convolutional layers.
  • Analysis of Temporal Skip Pooling: C5 + C2 performs best because conv2 preserves temporal structure while providing more high-level information than conv1.Later layers are more distinguishable but lose temporal information as their temporal dimensions collapse.
  • More Action Detection Results: Qualitative appendix videos use green ground-truth boxes and red detection boxes to show results across multiple datasets.
  • More Action Detection Results: On an unseen YouTube running video with crowded motion and severe camera movement, T-CNN detects both runners.The model was trained on UCF-Sports, and the video is more complex than that dataset.
  • More Action Detection Results: Additional YouTube examples illustrate T-CNN detections for horse riding and diving.The diving example is shown in Figure C.
Loading 1703.10664v3…