Source-linked AI summary
RGB-D-based Action Recognition Datasets: A Survey
Jing Zhang, Wanqing Li, Philip O. Ogunbona, Pichao Wang, Chang Tang
TL;DR
RGB-D action-recognition research has many datasets, making dataset and evaluation-protocol selection difficult and potentially unfair. This paper reviews the available datasets and protocols, identifies dataset saturation and protocol limitations, and provides recommendations for future dataset design and evaluation.
Problem
Researchers face difficulty selecting appropriate RGB-D datasets and establishing fair evaluation protocols because available datasets capture different factors.
Method
The paper comprehensively reviews RGB-D action datasets and analyzes the evaluation protocols used with them.
Results
The review identifies near-perfect accuracy on some datasets, unclear protocol comparisons, and greater realism in cross-view and cross-environment evaluation.
Takeaways & Limitations
The paper recommends broader environmental factors and clearer dataset and evaluation-protocol choices for future action-recognition research.
Takeaways & Limitations
Current datasets can exhibit saturation, with reported algorithms achieving near-perfect performance on some datasets.
Abstract
from arXiv · showhide
Human action recognition from RGB-D (Red, Green, Blue and Depth) data has attracted increasing attention since the first work reported in 2010. Over this period, many benchmark datasets have been created to facilitate the development and evaluation of new algorithms. This raises the question of which dataset to select and how to use it in providing a fair and objective comparative evaluation against state-of-the-art methods. To address this issue, this paper provides a comprehensive review of the most commonly used action recognition related RGB-D video datasets, including 27 single-view datasets, 10 multi-view datasets, and 7 multi-person datasets. The detailed information and analysis of these datasets is a useful resource in guiding insightful selection of datasets for future research. In addition, the issues with current algorithm evaluation vis-á-vis limitations of the available datasets and evaluation protocols are also highlighted; resulting in a number of recommendations for collection of new datasets and use of evaluation protocols.
1. Introduction
RGB-D action recognition has produced many datasets, but researchers lack a comprehensive basis for selecting datasets and evaluation protocols fairly. This survey addresses that gap by reviewing datasets, comparing protocols, and identifying limitations and recommendations.
- Many RGB-D benchmark datasets have been created since 2010 to support action-recognition algorithm development and evaluation.
- Existing reviews mainly focus on algorithms, leaving RGB-D datasets insufficiently summarized and analyzed.
- Selecting an appropriate dataset is difficult because datasets capture different factors that affect algorithm evaluation.
- The survey provides comprehensive summaries and analysis of RGB-D action datasets and their associated evaluation protocols.
- The survey covers action and activity datasets while excluding gesture datasets because gesture recognition is treated as an independent topic.
- It compares applications, complexity, results, and protocols, then offers recommendations for dataset use, protocol selection, and future dataset creation.
2. RGB-D Action/Activity Datasets
The survey organizes publicly available RGB-D datasets by capture viewpoint and number of actors. It records core dataset characteristics to support comparison and selection.
- The survey summarizes dataset creation dates, institutions, actions, subjects, repetitions, classes, samples, capture settings, backgrounds, and environments.
- Datasets are categorized as single-view, multi-view, or human-human interaction and multi-person activity datasets.
- Single-view datasets capture each action from one viewpoint, whereas multi-view datasets capture each action from two or more viewpoints.
- Single-view and multi-view datasets generally involve one actor at a time, while multi-person datasets involve interactions or activities by multiple people.
2.1. Single-view action/activity datasets
The survey catalogs single-view RGB-D action and activity datasets spanning controlled gestures and actions, daily activities, human-object interactions, and online recognition. Their settings and annotations vary substantially.
- MSR-Action3D: MSR-Action3D provides 20 fixed-view actions performed by 10 subjects three times, with segmented samples and action, subject, and repetition information.
- Dataset conditions: Several datasets emphasize practical variation, including daily activities, human-object interaction, noisy skeleton tracking, actor-dependent views, and occlusions.
- Online evaluation: G3D includes peak-frame action points that can be used to evaluate online action-recognition algorithms.
2.1.15. UCFKinect
The surveyed datasets address recognition under latency, initial-pose ambiguity, sequential activity, environmental, and multi-action challenges. They provide varied modalities, annotations, and interaction structures.
- UCFKinect: UCFKinect measures how quickly recognition systems overcome ambiguity in initial poses during 16 actions performed by 16 subjects.
- UCFKinect: UCFKinect contains 1,280 action samples represented only by 15-joint skeleton coordinates, orientations, and binary confidence values.
- Osaka University Kinect Action Dataset: The Osaka University dataset contains 10 actions performed once by 8 subjects, with RGB, depth, and skeleton data captured under unchanged conditions.
- Human Morning Routine: The Human Morning Routine dataset supports recognition and monitoring of kitchen activities for robot response, using two Kinect devices.
- Sequential and compositional datasets: Sequential and compositional datasets model continuous recognition and complex activities by combining actions into annotated activity classes.
- MAD Dataset: MAD contains 35 sequential actions performed by 20 subjects and provides labels suitable for both detection and classification.
2.2. Multi-view action/activity datasets
Multi-view RGB-D datasets capture actions from multiple viewpoints, either with several cameras or repeated performances from different views. The survey reviews their specifications, applications, and acquisition settings.
- Two approaches generate multi-view datasets: multiple cameras at different positions and angles, or repeated actions from different viewpoints.
- The reviewed multi-view datasets are mostly captured by multiple cameras and are summarized chronologically with basic specifications in Table 2.
- ATC42 dataset: ATC42 supports view variation for practical applications such as smart homes and e-healthcare, using four Kinect sensors at different heights and angles.
- Falling Detection dataset: The Falling Detection dataset contains laboratory-based fall and fall-like actions, publishes depth sequences, and annotates start and end frames for each fall process.
- Multi-view coverage ranges from two Kinect viewpoints in ReadingAct and Multi-View TJU to three simultaneous cameras in several datasets and four side views in UWA3D.
2.3. Interaction/Multi-person activity datasets
Interaction and multi-person datasets represent human-human, human-object, and mixed interactions across controlled, office, gaming, and daily-life settings. Their annotations and modalities vary substantially.
- Human-human interaction datasets typically involve two people, whereas multi-person activity datasets do not require a fixed number of participants.
- SBU dataset: SBU provides segmented two-person interactions with action labels and active/inactive actor identification as ground truth.
- K3HI: K3HI also contains two-person interactions, but RGB and depth data were omitted to preserve the spatial information of both skeletons.
- LIRIS Human Activities Dataset: LIRIS captures human-human, human-object, and human-human-object activities in complex scenarios using a Kinect mounted on a mobile robot, with frame-level bounding boxes.
- G3Di: G3Di records multiplayer gaming through a gamesourcing approach, with six participant pairs performing sports-related actions, often containing multiple action classes per sequence.
- Office Activity dataset: Office Activity combines single-subject and two-subject office activities across multiple views and contains 1180 RGB-D activity sequences.
3. Analysis
The analysis evaluates RGB-D datasets by application, acquisition and presentation, environmental complexity, protocols, and reported algorithm results. It links dataset structure to which evaluation tasks can be supported.
- The survey frames dataset analysis around application scenarios, acquisition and presentation, evaluation dependence, environmental complexity, protocols, and state-of-the-art results.
- Applications: The paper distinguishes HCI applications, such as games and device control, from daily-activity applications including surveillance, healthcare, robotics, and smart rooms.
- Acquisition and presentation modes: Four acquisition and presentation modes, plus two processed variants, are identified to organize how actions and activities are stored and labeled.
- Algorithm evaluation: Action detection requires continuous capture with accurate action-segment points, and only Modes 2, 3, and 4 support it.
- Algorithm evaluation: Online action recognition requires continuously presented unlabeled sequences with randomly ordered actions, making Mode 4 the only suitable mode.
- Environmental complexity: Most datasets include execution-rate and anthropomorphic variation, but viewpoint variation is limited to multi-view datasets, while occlusion, clutter, and camera motion are uncommon.
- Environmental complexity: The Kinect depth range restricts capture environments largely indoors, limiting the usefulness of datasets for testing outdoor algorithms.
Background clutter and occlusion
The survey classifies background and occlusion complexity into low, medium, and high levels according to background variation, clutter, and subject occlusion.
- Low complexity means a fixed, clean background with no subject occlusion.
- Medium complexity means a fixed but cluttered background where some subject occlusion may occur.
- High complexity means backgrounds vary among samples or are cluttered, with occlusions that may affect the actions.
Kinematic complexity
The survey classifies kinematic complexity by movement simplicity and duration, ranging from low to high.
- Low-complexity movements are relatively simple and short.
- Medium-complexity movements last longer and have moderate complexity.
- High-complexity movements are complex and long-duration.
Variability amongst actions
The survey measures variability amongst actions by the variation in complexity levels within a dataset.
- Low variability means complexity levels vary little among actions within a dataset.
- Medium variability indicates a moderate difference in complexity levels among actions.
- High variability indicates substantial differences in complexity levels among actions.
Execution rate
The survey evaluates RGB-D datasets, algorithms, and protocols across acquisition characteristics, environmental complexity, modalities, and reported results. It emphasizes matching datasets and evaluation procedures to algorithm goals because inconsistent protocols and modalities complicate fair comparison.
- Dataset characteristics: Dataset complexity includes execution-style variation, composable actions, and human-object interaction.The reviewed datasets are also classified by environmental factors and acquisition characteristics.
- Dataset selection: Dataset selection should reflect environmental factors relevant to the algorithm’s intended application.The survey specifically links dataset choice to the environmental factors underpinning an algorithm’s purpose.
- Evaluation protocols: Cross-subject, cross-view, cross-environment, and leave-one-out protocols define distinct training-testing splits.Cross-view applies to multi-view datasets, while cross-environment applies when actions are captured in different environments.
- Evaluation limitations: Comparisons are not unequivocal when algorithms use different evaluation protocols, datasets, or data modalities.The survey states that cross-dataset comparisons can be unfair and that modality differences also require consideration.
- State-of-the-art results: Tables 6–8 summarize state-of-the-art methods, modalities, protocols, and reported results across single-view, multi-view, and multi-person datasets.The tables organize methods alongside the settings used for training and evaluation.
- Recommendations: The survey recommends unified standard evaluation protocols for new datasets and clear justification for any new protocols.These recommendations are intended to support future algorithm design and performance evaluation.
4. Discussion
The discussion identifies limitations in current RGB-D datasets and evaluation protocols, then recommends more varied datasets, richer ground truth, application-matched protocols, and cross-dataset evaluation.
- Limitations of current datasets: Current RGB-D datasets are limited by small size, restricted action types, laboratory collection, subtle performance variation, and poor ground-truth labels.Typical datasets contain 10 to 20 actions, while actions are often performed under fixed instructions and settings.
- Limitations of current datasets: RGB-D datasets are smaller than comparable 2D video datasets because their videos must be captured manually under time, financial, and labor constraints.The cited 2D examples contain 203 action classes in 849 hours and 487 classes across 1 million videos.
- Limitations of evaluation protocols: Common evaluation protocols can produce unfair algorithm comparisons because fixed cameras, backgrounds, locations, and instructions allow training and test conditions to remain too similar.The authors note that algorithms may inadvertently rely on background information or actor position.
- Limitations of current datasets: Near-perfect accuracy on some dataset modalities indicates saturation, which may obscure whether algorithms are suitable for real-world deployment.The paper links this concern to datasets that do not sufficiently reflect environmental complexity and realistic action performance.
- Recommendations: Future datasets should increase sample and action variety, support realistic free-form performances, provide action segments and actor or object locations, and include standard protocols.Protocols should match application expectations; cross-subject testing is suggested for surveillance, while leave-one-sequence-out suits monitoring a specific subject.
- Recommendations: Cross-dataset evaluation is advocated because it varies actors, viewpoints, environments, and performance styles while allowing related datasets to be combined.The paper describes this setup as closer to applications where a system transfers across similar scenarios without complete retraining.
5. Conclusion
The review consolidates RGB-D action-recognition datasets and evaluation practices, providing analysis and recommendations for selecting datasets, developing algorithms, and conducting fair comparisons. It also identifies dataset simplicity and evaluation limitations that motivate more realistic, statistically significant, and cross-dataset testing.
- 5. Conclusion: The review provides comprehensive descriptions and analyses of commonly used RGB-D action-recognition datasets, including their characteristics and potential applications.This information supports dataset selection for algorithm development and evaluation and helps identify gaps for new dataset collection.
- 5. Conclusion: Simple datasets have produced a “saturated” state in which algorithmic improvement has stalled.The conclusion links this limitation to the need for datasets representing a broader selection of challenging environmental factors.
- 5. Conclusion: The paper advocates cross-dataset evaluation setups to provide a more realistic testing scenario.This recommendation addresses the limitations of evaluating algorithms within overly simple or narrow dataset settings.
- 5. Conclusion: The review recommends comprehensive evaluation protocols incorporating statistical significance tests for fair algorithm comparisons.It also summarizes commonly used evaluation and validation setups and addresses their drawbacks.
- 5. Conclusion: The authors report plans for an open-source software suite enabling easier action-recognition evaluation, especially with cross-dataset schemes.The proposed suite is intended to support practical adoption of the recommended evaluation approach.