Source-linked AI summary
Multimodal Multipart Learning for Action Recognition in Depth Videos
Amir Shahroudy, Gang Wang, Tian-Tsong Ng, Qingxiong Yang
TL;DR
Action recognition in depth videos is complicated by the articulated movement of body parts and the limitations of relying on skeleton features alone. The paper jointly learns classifiers and selects parts using structured sparsity over multimodal features and a hierarchical mixed norm. It reports outperforming other methods on all three tested datasets.
Problem
Skeleton-only features are limited in representing body-object interactions and can be unreliable when depth-based joint data is noisy or occluded.
Source: §1 · p. 1Method
The method jointly learns classifiers and selects sparse body parts using structured sparsity and a hierarchical mixed norm over multimodal part features.
Source: §1 · p. 1Results
The proposed method outperforms other available methods on three depth-based action-recognition datasets.
Source: §1 · p. 1Takeaways & Limitations
A sparse combination of multimodal part-based features can effectively and discriminatively represent the available action classes.
Source: §5 · p. 6A general L2 norm can suppress information in some dimensions, with greater concern when depth-action training samples are limited.
Source: §3.2 · p. 4
Abstract
from arXiv · showhide
The articulated and complex nature of human actions makes the task of action recognition difficult. One approach to handle this complexity is dividing it to the kinetics of body parts and analyzing the actions based on these partial descriptors. We propose a joint sparse regression based learning method which utilizes the structured sparsity to model each action as a combination of multimodal features from a sparse set of body parts. To represent dynamics and appearance of parts, we employ a heterogeneous set of depth and skeleton based features. The proper structure of multimodal multipart features are formulated into the learning framework via the proposed hierarchical mixed norm, to regularize the structured features of each part and to apply sparsity between them, in favor of a group feature selection. Our experimental results expose the effectiveness of the proposed learning method in which it outperforms other methods in all three tested datasets while saturating one of them by achieving perfect accuracy.
1 INTRODUCTION
The paper treats actions as concurrent behaviors of body joints and combines skeleton and depth appearance features to represent them. It proposes jointly learning classifiers and selecting parts through structured sparsity, and reports superiority over other methods on three datasets.
1 INTRODUCTION
- Actions can be treated as concurrent behaviors of body joints, called “parts” in this paper.The paper defines each part as a body joint, using the terminology of.Source: §1 · p. 1
- Skeleton-only features are limited by noisy or occluded joints and their inability to represent body-object interactions, motivating depth-based appearance features.The paper identifies LOP and HON4D as complementary appearance and motion descriptors around joints.Source: §1 · p. 1
- The prior actionlet approach mines discriminative joint sets, but its heuristic selection is inconsistent with subsequent learning and it concatenates feature types.The paper identifies these as drawbacks because optimal task-specific feature combinations are not guaranteed.Source: §1 · p. 1
- The method jointly learns classifiers and selects a sparse set of informative body parts for each action class.It bundles each part’s features and applies group sparsity; the hierarchical mixed norm regularizes learning weights at multiple levels.Source: §1 · p. 1
- The proposed method outperforms other available methods on three depth-based action-recognition datasets.The evaluated datasets are MSR-DailyActivity, MSR-Action3D, and 3D-ActionPairs.Source: §1 · p. 1
2 RELATED WORK
Prior work spans skeleton and depth-map features, actionlet-based joint selection, and mixed-norm methods for structured sparsity and multitask learning. This paper combines skeleton, LOP, and HON4D features and extends group sparsity to jointly select parts and handle heterogeneous features.
2 RELATED WORK
- Skeleton features are compact and discriminative, but noise, occlusion, and limited representation of object interactions constrain their reliability.Depth-map methods include occupancy patterns, spatio-temporal descriptors, and four-dimensional surface normals.Source: §2 · p. 2
- The authors combine skeleton, LOP, and HON4D features, which they treat as complementary depth-based descriptors.Prior work combined joint features with LOP or HON4D, while this paper uses all three feature types.Source: §2 · p. 2
- The actionlet approach mines informative joint sets and uses them as kernels; this paper instead applies group sparsity for joint feature selection.Features from each part are grouped, with an L1 norm between groups to obtain sparse active parts.Source: §2 · p. 2
- The paper applies an L2/L4 mixed norm to regularize each part’s multimodal features and generalizes it to an L1/L2/L4 hierarchical norm for sparsity between parts.Related mixed-norm work includes grouped and hierarchical variable selection and structured feature fusion.Source: §2 · p. 2
- For shared latent factors across binary action classifiers, the method uses L2 regularization across tasks followed by L1 selection across features.This follows a multitask joint-feature-selection pattern described in prior work.Source: §2 · p. 2
3 MULTIMODAL MULTIPART LEARNING
The method organizes action features by body part and modality, then learns class-specific classifiers with structured sparsity. A hierarchical mixed norm regularizes modality-level features and selects parts, while a two-step procedure addresses the large number of weights relative to limited training data.
Notations
- The notation defines nested mixed norms by taking norms within partitioned sets and then across those sets.The partitions can be further nested to extend the structural mixed norm to higher orders.Source: Notations · pp. 2–3
3.1 Multipart Learning by Structured Sparsity
- The feature vector is partitioned first by body part and then by modality, with depth- and skeleton-based features describing movement and local depth appearance.The learning task uses one-versus-all squared-error regression for the action classes.Source: §3.1 · p. 3
- The multipart structured norm groups each part’s learning weights and sparsifies across parts, selecting active parts for each class within regression-based learning.This integrates part selection into learning rather than relying on the discrete, separate heuristic selection described for actionlets.Source: §3.1 · pp. 3–4
3.2 Multimodal Multipart Learning via Hierarchical Mixed Norm
- A general L2 penalty across heterogeneous features ignores modality structure, which may suppress information and weaken generalization when training samples are limited.The proposed hierarchy instead regularizes coefficients within each modality group.Source: §3.2 · p. 4
- The hierarchical mixed norm applies L4 regularization within each modality group, L2 across modalities within each part, and L1 sparsity between parts.The outer sparsity is applied for each action class; the paper calls this method MMMP.Source: Figure 1 · p. 3, §3.2 · p. 4
3.3 Two Step Learning Approach
- A two-step procedure first learns partially optimal weights separately for each modality, then fine-tunes them jointly around those points.The authors motivate this procedure by the large number of weights relative to the limited training samples in depth-action benchmarks.Source: §3.3 · p. 4
4 EXPERIMENTS
The experiments evaluate the method on three depth-based action-recognition benchmarks using Kinect depth and skeleton features. The full method outperforms reported alternatives on each benchmark and reaches perfect performance on 3D Action Pairs.
4.1 Experimental Setup
- Experiments use Kinect depth and skeleton data, ignore RGB, and derive all features from 20 estimated body joints.Skeleton features are normalized and encoded with a Fourier temporal pyramid; local LOP and HON4D features provide depth-based appearance and dynamics.Source: §4.1 · p. 5
4.2 MSR-DailyActivity3D Dataset
- 84.03% accuracy in subject-wise cross-validation follows adding multimodality grouping to multipart grouping, compared with 80.61% for plain L2 and 81.55% for multipart structured norm.The comparison uses all 252 possible train/test splits of five out of ten subjects and omits the multitask term.Source: §4.2 · p. 5
- 91.25% accuracy on MSR-DailyActivity is the result of the proposed hierarchical mixed norm on the standard split.Higher inner-norm orders reached the same accuracy with slightly higher processing time.Source: Table 2 · p. 5, §4.2 · p. 6
- The multimodal multipart method reaches 91.25% on MSR-DailyActivity, exceeding the cited 88.75% temporal-warping result with skeleton and HON4D features.Using all three modalities yields 91.25%; the reported skeleton-plus-LOP result is 88.13%.Source: Table 3 · p. 5, §4.2 · p. 6
4.3 MSR-Action3D Dataset
- On MSR-Action3D, the method has the highest reported accuracy, though its margin is smaller than on the other datasets.The paper attributes the smaller margin to simple actions that are often distinguishable from skeleton features alone; multipart learning still outperforms other methods.Source: §4.3 · p. 6, Table 4 · p. 6, Table 5 · p. 6
4.4 3D Action Pairs Dataset
- Perfect performance on 3D Action Pairs surpasses the other reported methods when the full method uses all three feature modalities.The dataset is described as the easiest benchmark, with fewer classes and no within-class variation.Source: §4.4 · p. 6
5 CONCLUSION
The paper combines skeleton trajectories, depth occupancy patterns, and depth-surface histograms in a multimodal multipart action-recognition approach. Its hierarchical mixed norm structures feature learning and supports concurrent group selection; evaluations on three datasets report advantages over other methods.
5 CONCLUSION
- The method combines 3D joint trajectories, depth occupancy patterns, and histograms of depth-surface normals as features for each part.Source: §5 · p. 6
- A hierarchical mixed norm structures multimodal multipart features through part-modality diversity, within-part modality linking, and sparsity-based soft part selection.Source: §5 · pp. 6–7
- Evaluations on three challenging depth-action datasets support the framework’s strength compared with other methods.Table 6 is identified as a performance comparison for the 3D Action Pairs dataset, but its values are not supplied here.Source: §5 · p. 7, Table 6 · p. 7