Source-linked AI summary
Unstructured Human Activity Detection from RGBD Images
Jaeyong Sung, Colin Ponce, Bart Selman, Ashutosh Saxena
TL;DR
The paper addresses activity detection and recognition in uncontrolled, cluttered environments, where differences in people’s habits and mannerisms make recognition difficult. It uses Kinect-based pose and visual features with a hierarchical two-layer MEMM, achieving strong performance for both familiar and previously unseen people.
Problem
Activity recognition in unstructured homes and offices is difficult because cluttered environments and person-specific habits and mannerisms complicate detection.
Method
The approach combines Kinect RGBD features describing human pose and motion with a two-layer hierarchical MEMM that models sub-activities and selects graph structure through dynamic programming.
Results
84.7%/83.2% precision/recall was achieved for “have seen” people, versus 67.9%/55.5% for “new person” settings.
Takeaways & Limitations
The full two-layer MEMM performed best among the evaluated models, while combined skeletal and skeletal HOG features improved robustness to people and background variation.
Takeaways & Limitations
The method does not model occlusions and may require contextual information about objects involved in activities.
Abstract
from arXiv · showhide
Being able to detect and recognize human activities is essential for several applications, including personal assistive robotics. In this paper, we perform detection and recognition of unstructured human activity in unstructured environments. We use a RGBD sensor (Microsoft Kinect) as the input sensor, and compute a set of features based on human pose and motion, as well as based on image and pointcloud information. Our algorithm is based on a hierarchical maximum entropy Markov model (MEMM), which considers a person's activity as composed of a set of sub-activities. We infer the two-layered graph structure using a dynamic programming approach. We test our algorithm on detecting and recognizing twelve different activities performed by four people in different environments, such as a kitchen, a living room, an office, etc., and achieve good performance even when the person was not seen before in the training set.
I. INTRODUCTION
The paper addresses activity detection and recognition in cluttered, uncontrolled environments, where variations in people’s habits and mannerisms make recognition difficult. It proposes Kinect-based sensing and a hierarchical MEMM approach, reporting strong performance across familiar and unfamiliar people.
- Unstructured homes and offices create clutter, visual confusion, and person-specific variations that complicate daily-activity detection and recognition.The paper motivates assistive-robotics applications such as reminding elderly people to drink water.
- Prior approaches commonly used 2D video or RFID, but 2D video achieved relatively low accuracy and RFID required intrusive tag placement.The cited 2D-video example reports 78.5% accuracy even without clutter.
- The proposed system uses an inexpensive Microsoft Kinect RGBD sensor, an articulated skeleton, raw images, and depth data to estimate activities.The skeleton is generated from Kinect RGBD data and combined with image and depth information.
- Activities are modeled hierarchically with a two-layer MEMM, while graph-structure selection adapts to differences in task speed and style.The method also combines PrimeSense skeleton tracking with HOG computer-vision features.
- 84.7%/83.2% precision/recall was achieved for seen people, versus 67.9%/55.5% for new people, across twelve activities and five environments.The four participants performed activities in kitchens, offices, bathrooms, living rooms, and bedrooms.
II. RELATED WORK
Related work spans space-time video features, filtering and patch sampling, biologically inspired hierarchies, HMMs, and discriminative temporal models. The paper distinguishes its approach from prior methods through RGBD sensing and hierarchical activity modeling.
- Space-time video features and related extensions model points of interest, but this approach classifies activities rather than detecting them.
- Hierarchical activity-recognition methods have often emphasized neurologically inspired visual-cortex models with motion-direction-sensitive filters.
- HMM-based methods have modeled two-handed actions, 3D occupancy grids, motion templates, and other activity-recognition settings.
- MEMMs and CRFs were presented as discriminative alternatives to HMMs that avoid fixed-observation dependencies and permit longer interactions among observations.
- Robotic activity-recognition research has addressed aggression, human–robot nonverbal communication, and imitation of human activities.
III. OUR APPROACH
The approach uses supervised learning on labeled Kinect RGBD data and trains a two-layer MEMM to represent hierarchical activities and temporal transitions between sub-activities.
- The system collects ground-truth labeled RGBD data from a Kinect sensor and extracts features for supervised learning.
- A two-layer maximum-entropy Markov model captures the hierarchical structure of activities and transitions between sub-activities over time.
A. Features
Features combine Kinect-derived skeletal pose and motion information with HOG descriptors computed from image and depth data. The feature design represents pose relative to the body and captures movement over time.
- Features: A PrimeSense tracker represents the person with a fifteen-joint articulated skeleton, including joint coordinates and orientations relative to the sensor.
- Body pose features: Joint rotations are transformed relative to the torso and represented with half-space quaternions to obtain sensor-location-invariant pose features.The representation is applied to ten joints with orientation information.
- Hand Position: Hand features encode hand positions relative to the torso and head to capture activity-relevant configurations.
- Motion Information: Skeletal features comprise body-pose, hand, and motion descriptors, including 396 motion features from 9 × 11 × 4 quaternion values.
- Image and point-cloud features: HOG descriptors are computed from RGB or grayscale images and depth maps, counting gradient orientations within specified image bounding boxes.The described HOG descriptor provides 32 features.
B. Model Formulation
The model represents complex human activities hierarchically, linking high-level activities to sequences of hidden sub-activities and adapting those connections during inference.
- Activities are modeled as sequences of sub-activities, such as squeezing toothpaste, raising a toothbrush, and brushing.
- Because subjects perform sub-activities at different rates, the model cannot fix connectivity between activity and sub-activity layers in advance.
- A two-layered hierarchical MEMM represents sub-activities in a hidden middle layer beneath high-level activities.
- The feature input x_t is extracted from the articulated skeleton model at each time frame.
- Each high-level activity z_i spans a contiguous interval and connects to the corresponding sub-activities during that interval.
C. MEMM with Hierarchical Structure
The hierarchical MEMM combines visual evidence, sub-activity transitions, and activity transitions to score activity structures, while using learned or manually specified components for different terms.
- Each high-level activity connects to consecutive middle-layer sub-activities, capturing activities as sequences of component actions.
- A Gaussian mixture model clusters training data into sub-activities, using five clusters per activity plus location-specific and negative-example clusters.
- The model scores sub-activities from features, sub-activity transitions conditioned on the activity, and transitions between successive high-level activities.
- For neutral activity, the sub-activity transition model relies on sequences observed in non-neutral activities because all unknown non-activity sequences cannot be collected.
- A naive Bayes conditional-independence assumption treats the previous sub-activity and high-level activity as independent of current features given the current sub-activity.
- The factorized joint probability can be optimized efficiently by optimizing terms containing only two variables.
D. Graph Structure Selection
Graph structure selection determines which sub-activities belong to each high-level activity and uses dynamic programming to find the optimal segmentation without enumerating all possible structures.
- Each sub-activity node can either remain attached to the current activity or begin a new high-level activity, creating 2^n possible structures for n nodes.
- Candidate structures combine an optimal prefix graph with one final substructure spanning the remaining time interval under a single activity.
- The dataset examples span twelve activities, including household, office, and personal-care actions such as drinking water and brushing teeth.
- The algorithm evaluates candidate graph structures and activities to maximize likelihood given all observations so far.
- Dynamic programming stores probabilities for each possible most recent high-level activity and reuses them across earlier time points.
- The resulting graph-selection algorithm is shown optimal by induction on time, using an optimal earlier structure plus the final substructure.
- Space complexity is O(n · t), where n is the number of activities and t is the time, with a maximum substructure size used to control computation.
IV. EXPERIMENTS
The experiments evaluate RGBD sensing, feature choices, training settings, and model variants for activity detection across multiple environments and subjects. The comparisons include naive SVM, one-level MEMM, and hierarchical MEMM models, with testing on both familiar and unseen people.
- Experimental setup: The dataset covers five environments and twelve activities, using RGBD data collected from four people.The environments are office, kitchen, bedroom, bathroom, and living room; the subjects included two males and two females.
- Experimental setup: The test data includes random movements in addition to the target activities, supporting evaluation of activity detection rather than classification alone.Random movements range from standing still to walking around and stretching or waving.
- Experimental setup: Testing uses separate “new person” and “have seen” settings to measure performance with and without subject-specific training data.The new-person setting uses leave-one-out cross-validation, while the have-seen setting includes half of the test subject’s data during training.
- Model comparisons: The full model is compared with a multiclass SVM baseline and a one-level MEMM, both trained with the full feature set.The SVM maps features directly to high-level activities, while the one-level MEMM combines SVM-based activity probabilities with activity-transition probabilities.
- Feature comparisons: Feature experiments compare RGB-only HOG features, RGB-and-depth HOG features, and skeletal features combined with RGB-and-depth skeletal HOG features.These variants are used to assess the improvement associated with richer input features in the hierarchical MEMM.
A. Results and Discussion
The two-layered MEMM achieved strong activity detection and recognition across seen and unseen people, with robustness supported by hierarchical modeling and combined skeletal, image, and depth features. Errors often involved similar activities or neutral predictions, while occlusion and missing object context remained limitations.
- 84.7%/83.2% precision/recall in the “have seen” setting and 67.9%/55.5% in the “new person” setting were achieved by the full model.
- The full two-layer MEMM performed best because it captures hierarchical activity structure beyond the temporal modeling of a one-level MEMM.The naive classifier and one-level MEMM performed worse when activity characteristics were subtle.
- The confusion matrices show that mistakes commonly involved similar activities, such as cooking-chopping versus cooking-stirring and rinsing mouth versus brushing teeth.
- Random activity data was usually classified as neutral, helping distinguish target activities from unseen activities and reducing likely misfires.Training on regular and mirrored data also supported performance with left- and right-handed people.
- The method was evaluated without modeling occlusions, and some activities may require contextual information about objects beyond human pose.
V. CONCLUSION
The paper addresses activity detection and recognition in unstructured homes and offices using an inexpensive RGBD sensor and a hierarchical two-layered MEMM. Tested across twelve activities, four people, and five environments, the approach achieved good detection performance for both seen and unseen people.
- The study targets human activities performed in unstructured environments such as homes and offices.
- An inexpensive Microsoft Kinect supplies RGBD input for the approach.
- The two-layered MEMM models activity hierarchy, sub-activity transitions over time, and relations between sub-activities and feature types.
- Inference exploits hierarchical activity structure to determine the best MEMM graph structure.
- Evaluation covered twelve activities, four people, and five environments, with activities often interleaved with random non-target actions.
- The algorithm achieved good detection performance when the person was either seen or not seen during training.