Source-linked AI summary
R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
Huijuan Xu, Abir Das, Kate Saenko
TL;DR
Activity detection in continuous, untrimmed videos must recognize activities and precisely localize their temporal boundaries. R-C3D uses a shared, fully convolutional 3D proposal-and-classification network, achieving state-of-the-art results on THUMOS’14 and broader evaluation across ActivityNet and Charades.
Problem
Continuous-video activity detection requires meaningful spatio-temporal representations and precise localization of activity start and end times.
Method
R-C3D encodes videos with fully convolutional 3D features, proposes temporal regions, and classifies them using shared pooled features in an end-to-end model.
Results
R-C3D achieves new state-of-the-art results on THUMOS’14 and Charades, improves ActivityNet results with only C3D features, and is reported 5x faster than current methods.
Takeaways & Limitations
Evaluations on three diverse datasets demonstrate the model’s general applicability to activity detection without relying on particular dataset properties.
Takeaways & Limitations
On Charades, mAP@0.5 is 9.3%, with performance affected by low-illumination indoor scenes and the dataset’s multi-label nature.
Abstract
from arXiv · showhide
We address the problem of activity detection in continuous, untrimmed video streams. This is a difficult task that requires extracting meaningful spatio-temporal features to capture activities, accurately localizing the start and end times of each activity. We introduce a new model, Region Convolutional 3D Network (R-C3D), which encodes the video streams using a three-dimensional fully convolutional network, then generates candidate temporal regions containing activities, and finally classifies selected regions into specific activities. Computation is saved due to the sharing of convolutional features between the proposal and the classification pipelines. The entire model is trained end-to-end with jointly optimized localization and classification losses. R-C3D is faster than existing methods (569 frames per second on a single Titan X Maxwell GPU) and achieves state-of-the-art results on THUMOS'14. We further demonstrate that our model is a general activity detection framework that does not rely on assumptions about particular dataset properties by evaluating our approach on ActivityNet and Charades. Our code is available at http://ai.bu.edu/r-c3d/.
1. Introduction
R-C3D addresses continuous-video activity detection by jointly learning temporal proposals and classification with shared 3D convolutional features. The design targets precise, flexible localization while improving computational efficiency and benchmark performance.
- Continuous-video activity detection requires recognizing activities while precisely localizing their temporal start and end times.
- Existing methods rely on separately learned features, external proposals, or exhaustive sliding windows, limiting end-to-end adaptation and efficiency.
- R-C3D jointly optimizes temporal proposal generation and activity classification using task-dependent, fully convolutional 3D features.
- Shared convolutional feature maps reduce computation, while 3D region pooling supports classification of variable-length temporal proposals.
- R-C3D achieves new state-of-the-art results on THUMOS’14 and improves results on ActivityNet when using only C3D features.
- The model reports detection speeds 5x faster than current methods through shared fully-convolutional C3D features.
2. Related Work
Prior activity-detection research includes stagewise sliding-window pipelines, recurrent models for arbitrary-length segments, weakly supervised localization, and object-detection-inspired frameworks. R-C3D is positioned within efforts to improve efficiency, boundary flexibility, supervision, and cross-dataset applicability.
- Traditional activity-detection methods classify segments from sliding windows, often using stagewise pipelines that are not trained end-to-end.
- Exhaustive sliding-window search is computationally inefficient and constrains detected activity boundaries.
- RNN and LSTM approaches avoid exhaustive windows by modeling temporal evolution and predicting activity labels or segments at each time step.
- Other work addresses weakly supervised localization with video-level labels or incorporates language and activity-length models for detection.
- R-C3D draws on Faster R-CNN’s region proposal and pooling framework while avoiding dataset-specific assumptions through evaluation on varied activity-detection datasets.
3. Approach
R-C3D combines a shared 3D ConvNet, temporal proposal generation, and activity classification with boundary refinement. Its proposal and classification stages support variable-length segments and share convolutional features for efficient end-to-end training.
- 3. Approach: R-C3D consists of a shared 3D ConvNet feature extractor, a temporal proposal stage, and an activity classification and refinement stage.The shared C3D feature maps support efficient computation and end-to-end training.
- 3.1. 3D Convolutional Feature Hierarchies: The 3D ConvNet accepts variable-length video inputs and produces shared spatio-temporal feature maps for both proposal and classification subnets.The number of input frames is limited only by GPU memory, while the shared activations reduce computation.
- 3.2. Temporal Proposal Subnet: The temporal proposal subnet uses multiscale anchor segments to predict variable-length activity proposals, temporal offsets, and activity-versus-background scores.Anchors are distributed across temporal locations and provide reference segments for proposal prediction.
- 3.4. Optimization: R-C3D jointly optimizes classification and regression losses for both the proposal and activity classification subnets.The proposal subnet uses activity-agnostic binary classification and regression, while the classification subnet predicts activity categories and refined boundaries.
- 3.3. Activity Classification Subnet: The classification stage applies NMS, 3D RoI pooling, activity classification, and boundary regression to selected proposals.3D RoI pooling converts arbitrary-length proposal features into fixed-size representations for the downstream fully connected layers.
4. Experiments
R-C3D is evaluated on THUMOS’14, ActivityNet, and Charades using mAP-based detection metrics, with comparisons spanning accuracy, generalization, and inference speed. The experiments report strong benchmark performance, including state-of-the-art results on THUMOS’14 and Charades, improved ActivityNet results, and fast inference.
- Experimental Setup: R-C3D is evaluated on THUMOS’14, ActivityNet, and Charades using mAP at different temporal IoU thresholds.The experiments use three large-scale activity-detection datasets and report mAP@α, where α denotes different IoU thresholds.
- THUMOS’14: 28.9% mAP@0.5 is achieved on THUMOS’14 with the two-way buffer, compared with 23.3% for the reported state-of-the-art method.The one-way buffer reaches 27.0% mAP@0.5, while the two-way setting improves mAP at all evaluated IoU thresholds.
- THUMOS’14: More than 20% absolute AP improvement over the next best method is reported for Basketball Dunk, Cliff Diving, and Javelin Throw on THUMOS’14.R-C3D outperforms other methods in most classes and is narrowly second for some activities, including Billiards and Cricket Shot.
- ActivityNet: 4.3% and 4.5% mAP@0.5 improvements over UPC are obtained on ActivityNet’s validation and test sets, respectively, without UPC’s single-class-per-video assumption.Training on both the training and validation sets yields a further 1.6% improvement.
- Charades: R-C3D outperforms the asynchronous temporal fields model and different baselines on Charades under the reported frame-level evaluation.The model maps predicted activity segments to 25 equidistant frames and uses the evaluation metric from the compared work.
- Inference Speed: 569 fps is reached for joint proposal and classification inference on a single Titan-X Maxwell GPU.The shared features and end-to-end proposal-classification pipeline make R-C3D faster than the compared methods; 1030 fps is reported on Titan-X Pascal.
5. Conclusion
R-C3D is an end-to-end temporal proposal classification network evaluated across diverse datasets. The authors report faster and more accurate detection than existing 3D ConvNet-based models, while noting future feature extensions.
- The model is evaluated on three large-scale datasets with diverse characteristics.
- R-C3D detects activities faster and more accurately than existing models based on 3D ConvNets.
- Additional features can be incorporated to further improve activity detection without sacrificing speed.The authors identify integrating hand-engineered motion features as a possible future direction.