Source-linked AI summary
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos
Amir Shahroudy, Tian-Tsong Ng, Yihong Gong, Gang Wang
TL;DR
RGB and depth offer complementary strengths, motivating analysis that separates shared information from modality-specific cues for action recognition. The paper develops a deep autoencoder-based hierarchical factorization network with structured sparsity classification, reporting state-of-the-art performance across five RGB+D benchmarks.
Problem
RGB and depth sequences have different strengths and limitations, so their complementary shared and modality-specific information requires dedicated multimodal analysis for action recognition.
Method
A deep autoencoder-based network hierarchically factorizes RGB+D features into shared and modality-specific components, while mixed-norm structured sparsity selects components and layers for classification.
Results
State-of-the-art performances were achieved on all reported benchmarks across five RGB+D action-recognition datasets.
Takeaways & Limitations
The experiments support the strength of deep shared-specific component analysis combined with structured sparsity learning for RGB+D action recognition.
Takeaways & Limitations
On NTU RGB+D, RGB features performed poorly for cross-view evaluation, so the proposed framework was evaluated only cross-subject.
Abstract
from arXiv · showhide
Single modality action recognition on RGB or depth sequences has been extensively explored recently. It is generally accepted that each of these two modalities has different strengths and limitations for the task of action recognition. Therefore, analysis of the RGB+D videos can help us to better study the complementary properties of these two types of modalities and achieve higher levels of performance. In this paper, we propose a new deep autoencoder based shared-specific feature factorization network to separate input multimodal signals into a hierarchy of components. Further, based on the structure of the features, a structured sparsity learning machine is proposed which utilizes mixed norms to apply regularization within components and group selection between them for better classification performance. Our experimental results show the effectiveness of our cross-modality feature analysis framework by achieving state-of-the-art accuracy for action classification on five challenging benchmark datasets.
1. Introduction
RGB and depth sequences provide complementary information for action recognition: RGB captures appearance and motion, while depth supplies 3D scene structure. The paper proposes hierarchical shared-specific factorization and structured sparsity to analyze and classify multimodal features.
- Motivation: RGB action sequences capture appearance and motion, but lack explicit 3D scene structure that depth sequences provide.The paper frames depth as complementary to RGB textural and appearance information.
- Motivation: CCA and KCCA seek common multimodal representations, but are limited respectively by linearity, kernel-space capacity, and poor scalability.The paper motivates a learned nonlinear alternative for high-level cross-modal complexity.
- Approach: The proposed deep autoencoder-based network discovers shared and informative components of RGB+D inputs through nonlinear common component analysis.Its goal is to overcome limitations of analytic CCA and KCCA solutions.
- Approach: Each network layer factorizes multimodal features into shared and modality-specific components, and stacking layers decodes increasingly complex nonlinear representations.Shared components capture cross-modal information while specific components retain modality-dependent discriminative features.
- Classification: Structured sparsity learning groups features within components and selects groups between them to improve classification of hierarchical factorized representations.The framework uses mixed norms for component grouping and group selection.
2. Related work
Prior RGB+D action-recognition methods fuse descriptors or analyze modalities in shallow or conventional frameworks. This paper instead performs hierarchical nonlinear shared-specific analysis across network layers and uses structured sparsity for concurrent component and layer selection.
- Deep multimodal learning: Some deep multimodal methods learn a common representation, but prior approaches perform multimodal analysis in only one deep-network layer.The proposed method performs common component analysis at every layer.
- Proposed distinction: Its structured sparsity machine extends sparse selection to concurrent component and layer selection for hierarchical factorized outputs.This is presented as a distinction from structured-sparsity methods developed for related tasks.
- Depth-based recognition: Depth-only action-recognition methods analyze either depth maps directly or 3D skeleton joint positions.The related work distinguishes depth-map descriptors from skeleton-based representations.
- RGB+D multimodal analysis: Earlier multimodal methods fused RGB and depth descriptors using concatenation, genetic programming, hidden Markov models, or structured sparsity.These approaches include STIP-based descriptor fusion and other modality-combination strategies.
- Proposed distinction: The proposed framework factorizes RGB+D features into shared and modality-specific components across a hierarchy of nonlinear layers.The authors state that the framework can analyze cross-modality components for any type of multimodal features.
3. Deep shared-specific component analysis
The paper factorizes RGB+D features into shared and modality-specific components, then stacks nonlinear layers to progressively disentangle higher-order cross-modal structure. Local and holistic analyses are combined before classification.
- Motivation: RGB and depth features encode complementary information, motivating fusion in a factorized common and modality-specific feature space.RGB captures 2D motion, appearance, and texture, whereas depth captures 3D shape and motion.
- Single-layer analysis: Each input modality is mapped into shared components Y and modality-specific components Z through learned projections and nonlinear activations.The shared components correspond to the highest correlation between modalities, while Z retains individual features.
- Single-layer analysis: The factorization objective combines shared-component similarity, reconstruction costs, weight regularization, and sparsity penalties.Reconstruction prevents degeneration, while KL-divergence sparsity is applied to Y and Z outputs.
- Deep analysis: The deep network feeds each layer’s shared Y components into the next layer to progressively factorize more complex cross-modal correlations.Layers are trained greedily and layer-wise, with each layer optimized after the previous one converges.
- Convolutional analysis: The convolutional DSSCA analyzes local video sub-cubes, concatenates their factorized components with holistic features, and processes the combined representation through a holistic network.This design preserves local cross-modal information that holistic analysis alone may lose.
4. Structured sparsity learning machine
The structured sparsity learning machine classifies hierarchically factorized features while regularizing correlated features within components and selecting useful components and layers.
- Classifier formulation: The classifier learns a linear projection from hierarchically factorized features A to a class-assignment matrix F.The objective makes A^T B approximate binary class membership labels.
- Component selection: Component-wise mixed-norm regularization applies ℓ2 penalties within components and ℓ1 selection between components.This encourages correlated features within a component to activate or deactivate together while selecting discriminative components.
- Layer selection: A second mixed norm performs layer-wise group selection because discriminative features may emerge at different hierarchy levels.The layer norm groups parameters belonging to each network layer and applies ℓ1 sparsity between layers.
- Optimization: The optimization also includes general weight decay to reduce classifier overfitting and is solved using L-BFGS.After learning the optimal projection matrix, testing samples are classified with exemplar features.
5. CCA-RICA factorization as a baseline method
As a baseline, the paper combines linear CCA with reconstruction independent component analysis to separate correlated and modality-specific RGB+D features.
- CCA stage: The CCA-RICA baseline first extracts correlated components from RGB and depth features using linear projections.CCA solves the projections analytically as an eigenproblem that maximizes cross-modal correlation.
- RICA stage: After fixing the CCA projections, RICA separately extracts modality-specific components for RGB and depth.This second stage addresses the insufficiency of shared components alone for recognition.
- Output representation: Each trajectory is represented by the quadruple {Zr(k), Yr(k), Yd(k), Zd(k)} after both optimization stages converge.The representation contains RGB-specific, RGB-shared, depth-shared, and depth-specific components.
6. Experiments
The experiments evaluate the proposed methods on RGB+D action-recognition datasets using multimodal handcrafted features, baseline fusion methods, and alternative DSSCA structures.
- Datasets: The proposed methods are evaluated on five RGB+D action-recognition datasets collected indoors with Microsoft Kinect sensors.The sensor captures RGB videos, depth maps, and 3D positions of 20 body joints.
- Features: RGB representations use dense trajectories with HOG, HOF, MBHX, and MBHY descriptors, vector-quantized into 2K-codeword temporal-pyramid representations.The final sample representation concatenates max-pooled codes across descriptors and temporal levels.
- Experimental setting: Because RGB and depth sequences are usually unsynchronized, convolutional cubes are chosen to cover corresponding portions of both modalities.All evaluated datasets except RGBD-HuDaAct have this alignment issue.
- Features: Depth and RGB inputs combine multiple motion, appearance, joint, occupancy, normal, and dynamic-pattern descriptors.Depth uses joint Fourier coefficients, LOP, HON4D, DS, and DDP; RGB uses DCP and dense trajectories.
- Comparisons: The study compares descriptor-level fusion, kernel-level combination, and DSSCA Kernel and DSSCA SSLM configurations.It also compares holistic, local, and stacked local+holistic DSSCA structures; the authors report complementary local and holistic information and improved accuracy from stacking them.
6.2. Online RGBD action dataset
On the Online RGBD Action dataset, the experiments evaluate DSSCA, SSLM, alternative network structures, modality-specific components, and comparisons with state-of-the-art methods across three scenarios.
- Dataset and evaluation: The dataset contains 336 RGB+D videos from seven daily action classes, with two cross-subject scenarios and one cross-environment scenario.The first two scenarios swap training and testing actors; the third tests actors recorded at another location.
- Baseline comparison: DSSCA and SSLM improve classification over baseline methods, with DSSCA producing stronger representations and SSLM selecting discriminative components and layers.The table compares descriptor concatenation, kernel combination, correlation-independence analysis, and structured sparsity learning.
- Network structures: Stacked local+holistic networks improve accuracy over separate holistic or local structures because the authors find local and holistic features complementary.The evaluated structures include a 3-layer holistic network, a 2-layer local network, and their stacked combination.
- Component analysis: Including modality-specific components improves accuracy over a shared-components-only correlation network, while multiple layers yield larger improvements.The shared-only reference network acts similarly to traditional CCA, whereas the proposed structure decomposes features layer by layer.
- State-of-the-art comparison: The proposed method outperforms the limited state-of-the-art comparisons on this dataset by a large margin.The authors connect the result with the importance of RGB+D fusion and the effectiveness of their method for action recognition.
6.3. MSR-DailyActivity3D dataset
The experiments extend evaluation to MSR-DailyActivity3D and 3D Action Pairs, covering challenging and less challenging RGB+D benchmarks with baseline and state-of-the-art comparisons.
- MSR-DailyActivity3D: MSR-DailyActivity3D contains 320 RGB+D samples from 16 daily-activity classes and uses a cross-subject split with five training subjects.The remaining subjects are used for testing, and results are reported in Tables 5 and 6.
- MSR-DailyActivity3D: The proposed method reduces the error rate by more than 40% compared to the best previously reported result on MSR-DailyActivity3D.The comparison appears in Table 7 against state-of-the-art methods.
- 3D Action Pairs: 3D Action Pairs contains six action-class pairs whose classes share body motions but differ in temporal order, with 360 RGB+D videos overall.The first five subjects are used for testing and the others for training.
- 3D Action Pairs: The proposed framework ties two recent methods by achieving 100% accuracy on the 3D Action Pairs benchmark.The comparison is reported in Table 10 against MMMP and BHIM.
6.5. NTU RGB+D dataset
On NTU RGB+D, the proposed framework is evaluated using RGB+D features, but only under cross-subject evaluation because RGB features perform poorly for cross-view evaluation. The stacked DSSCA analysis achieves 99.0% accuracy on RGBD-HuDaAct, outperforming atomic local analysis and other reported methods.
- NTU RGB+D contains 56880 RGB+D samples spanning 60 action classes and supports cross-subject and cross-view evaluation.
- The experiments use Fourier temporal pyramids over skeletons, HON4D, and LOP for depth inputs, while RGB inputs use the feature set used on other datasets.
- The proposed framework is evaluated only under cross-subject evaluation because the RGB features contribute insufficiently to cross-view multimodal analysis.
- RGBD-HuDaAct provides synchronized and aligned RGB and depth videos, enabling atomic-level analysis of local features within video samples.
- Atomic local level analysis: 97.9% correct classification is achieved by SSCA on atomic local features, improving the error rate by more than 40% over CCA-RICA.
- Global level analysis: 99.0% accuracy is achieved by stacked DSSCA analysis, outperforming current methods and atomic local analysis on RGBD-HuDaAct.
6.7. Comparison with single modality
The proposed method is compared with a single-modality baseline using kernel combinations over holistic and multiple local features. It outperforms that baseline by integrating RGB and depth information.
- The proposed method outperforms the single-modality baseline under kernel-combination classification.The comparison uses holistic and multiple local features from each modality and applies kernel combination to both methods.
6.8. Analysis of component contributions in the classifier
SSLM assigns substantial weights to both shared and modality-specific factorized components. The weighting pattern supports complementary contributions from shared and modality-specific features.
- SSLM assigns relatively high weights to the Y3 shared component, supporting its robustness and discriminative properties.
- The Z components of both modalities receive weights across all three layers, indicating informative and complementary modality-specific features.
7. Conclusion
The paper introduces DSSCA, a hierarchical shared-specific factorization framework for RGB+D action features, together with SSLM for structured component and layer selection. Experiments on five datasets report state-of-the-art performance across the reported benchmarks.
- DSSCA uses autoencoder-based factorization units to decompose multimodal inputs into common and modality-specific components at each layer.
- The deep factorization framework is extended to a convolutional setting.
- SSLM uses mixed norms to select factorized components and layers for fusing decomposed feature representations.
- Experiments on five RGB+D action recognition datasets achieve state-of-the-art performance on all reported benchmarks.