Source-linked AI summary
PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding
Chunhui Liu, Yueyu Hu, Yanghao Li, Sijie Song, Jiaying Liu
TL;DR
Existing 3D action datasets largely emphasize segmented-video recognition, leaving a need for large-scale continuous multimodal benchmarks. The paper introduces PKU-MMD with broad multimodal coverage and a 2D-AP protocol, finding that existing methods remain unsatisfactory for large-scale 3D action detection.
Problem
Existing 3D benchmarks mainly focus on segmented-video recognition, while large-scale continuous multimodal datasets for action detection remain scarce.
Method
The paper constructs PKU-MMD, a continuous multimodal 3D activity dataset, and evaluates detection methods using a 2D-AP protocol that considers overlap and confidence jointly.
Results
Existing methods are not satisfactory for large-scale 3D action detection on PKU-MMD.
Takeaways & Limitations
PKU-MMD provides a benchmark for studying continuous multimodal 3D human activity understanding, especially temporal action detection.
Takeaways & Limitations
Open-domain recognition and detection remain challenging across differing environmental conditions.
Abstract
from arXiv · showhide
Despite the fact that many 3D human activity benchmarks being proposed, most existing action datasets focus on the action recognition tasks for the segmented videos. There is a lack of standard large-scale benchmarks, especially for current popular data-hungry deep learning based methods. In this paper, we introduce a new large scale benchmark (PKU-MMD) for continuous multi-modality 3D human action understanding and cover a wide range of complex human activities with well annotated information. PKU-MMD contains 1076 long video sequences in 51 action categories, performed by 66 subjects in three camera views. It contains almost 20,000 action instances and 5.4 million frames in total. Our dataset also provides multi-modality data sources, including RGB, depth, Infrared Radiation and Skeleton. With different modalities, we conduct extensive experiments on our dataset in terms of two scenarios and evaluate different methods by various metrics, including a new proposed evaluation protocol 2D-AP. We believe this large-scale dataset will benefit future researches on action detection for the community.
1. Introduction
PKU-MMD addresses the shortage of large-scale continuous 3D action-detection benchmarks by combining extensive multimodal data with a unified 2D-AP evaluation protocol.
- Existing 3D benchmarks mainly target recognition on segmented videos, leaving large-scale continuous action detection underrepresented.
- Different modalities capture complementary information: RGB provides appearance, optical flow motion, and skeleton depth-related cues.
- PKU-MMD contains 1076 videos across 51 action categories, 66 subjects, and 3 camera views, totaling 3,000 minutes and 5,400,000 frames.
- The dataset provides RGB, depth, skeleton, and infrared data, with optical flow and motion vectors additionally calculable.
- The proposed 2D-AP protocol jointly considers overlap and detection confidence, producing a single evaluation value instead of multiple overlap-specific average precisions.
2. Related Work
Related work spans recognition, detection, multimodal sensing, and 3D datasets, but prior benchmarks often remain limited in scale, modalities, continuity, or localization scope.
- Activity analysis: Early activity analysis emphasized recognition of segmented videos using handcrafted features, while later deep models learned representations and temporal dynamics.
- Action detection: Action-detection methods commonly use sliding windows or action proposals, with reported trade-offs involving computational efficiency and localization accuracy.
- Open-domain challenges: Open-domain action recognition and detection remains challenging because models perform well under similar training and testing environments but poorly across differing conditions.
- 3D activity understanding: Skeleton-based approaches increasingly model temporal dynamics with recurrent networks, but few methods address action detection on continuous 3D skeleton data.
- 3D activity datasets: Several existing datasets have limited instances, modalities, or video diversity, motivating PKU-MMD as a larger benchmark for data-driven algorithms.
- 3D activity datasets: Prior datasets differ in focus, including gaming, daily activities, multiview capture, multimodality, online detection, and continuous sequences.
3. The Dataset
PKU-MMD is a large-scale dataset for continuous multi-modality 3D action detection, built from long Kinect sequences with synchronized modalities and verified temporal annotations.
- Dataset overview: PKU-MMD contains 1000+ long sequences totaling 5,312,580 frames, 3,000 minutes, and more than 20,000 temporally localized actions.Each sequence lasts approximately 3–4 minutes and contains about 20 action instances.
- Participants and modalities: Data were collected from 66 distinct subjects using Kinect v2 sensors that synchronously capture color, depth, infrared, and skeleton information.
- Dataset composition: The dataset covers 51 action classes, including 41 daily actions and 10 interaction actions.
- Sequence design: Daily actions were arranged into weakly connected sequences to improve continuity across multiple compact actions.Examples combine actions such as taking off a shirt, taking off a cat, drinking water, and sitting down.
- Annotation and validation: Temporal labels were produced through volunteer localization and strengthened using protocol checks and cross-view verification.The validation process checks action overlap and reasonable duration, using multiple camera views to assess consistency.
4. Evaluation Protocols
The benchmark defines dataset partitions and precision-recall protocols for continuous action detection, including a 2D-AP measure that jointly accounts for retrieval and temporal overlap.
- Dataset splits: Evaluation uses cross-view and cross-subject splits to test robustness to viewpoint transformations and variation among actors.Cross-view trains on middle and right cameras and tests on the left; cross-subject trains on 57 subjects and tests on 9.
- Detection criterion: Detection evaluation first determines whether predicted intervals are correct using their overlap with ground-truth intervals, then evaluates precision and recall.
- F1-score: F1-score combines precision and recall for correct detections but does not use the confidence of each interval.
- Average precision: Interpolated average precision uses confidence-ranked retrieval results to form a precision-recall curve and selects the highest precision at or above each recall level.The interpolated average precision is averaged across recall levels, with recall determined by overlap threshold θ.
- mAP: Mean average precision is reported over action categories and over videos through mAPa and mAPv splitting protocols.
- 2D-AP: 2D-AP evaluates retrieval performance across different overlap ratios rather than associating separate AP or mAP values with each overlap threshold.
5. Experiments
The experiments benchmark continuous 3D action detection using multiple representations, temporal localization methods, sliding-window settings, and modality combinations. Results expose trade-offs between context, computation, localization strictness, and modality choice.
- Experimental setup: The benchmark evaluates detection methods through video representation followed by temporal localizing and category classifying.Baseline methods are divided into these two phases because few 3D action-detection implementations exist.
- Video representations: Raw, convolutional skeleton, Deep RGB, and Deep Optical Flow features provide alternative representations for multi-modality input.Convolution Skeleton adds temporal difference and normalization, while Deep RGB and Deep Optical Flow use deep convolutional features.
- Temporal detection methods: Sliding-window detection is tested with BLSTM, SVM, STA-LSTM, and joint classification-regression RNN approaches.The methods cover stacked bidirectional recurrent classifiers, SVM classification, spatial-temporal attention, and frame-level joint prediction.
- Results: Deep Optical Flow outperforms traditional features in motion description, while Joint Classification Regression RNN achieves remarkable results under stricter localization requirements.The paper attributes the latter result to frame-level predictions, which better match precise activity-boundary localization.
- Sliding-window analysis: Smaller strides improve performance through denser sampling but increase time cost, whereas window size trades context, noise, and computation complexity.The experiments set window size and stride to 30 as a performance-speed trade-off.
- Multi-modality scenarios: Multi-modality experiments compare independent and combined RGB, optical-flow, skeleton, and differential-skeleton representations using cross-view and cross-subject evaluation.Table 3 reports mAP across overlapping ratios from 0.1 to 0.7 and ensembles modalities by averaging predicted confidences.
6. Conclusion
PKU-MMD is proposed as a large-scale multi-modality 3D benchmark for action detection, combining diverse long videos with a new evaluation protocol. The reported results show that existing methods remain inadequate, leaving large-scale 3D action detection unresolved.
- Dataset and scope: PKU-MMD contains 1076 long sequences from 66 actors, with 20 action instances per sequence across 51 action classes.The dataset includes 3,000 minutes and 5.4 million frames, with variation across three camera views.
- Dataset and scope: The dataset provides RGB, depth, skeleton, and infrared modalities for human activity understanding and temporal action detection.Its scale and modality diversity support experiments with deep networks such as LSTM or CNN.
- Evaluation: The proposed 2D-AP protocol jointly considers overlap and detection confidence in a single action-detection evaluation value.The protocol replaces a list of mean average precisions at corresponding overlap ratios with one value.
- Conclusion: Existing methods are reported as unsatisfactory, indicating that large-scale 3D action detection remains far from solved.The conclusion calls for further research on action-detection methodologies.