Source-linked AI summary
Deep Learning on Lie Groups for Skeleton-based Action Recognition
Zhiwu Huang, Chengde Wan, Thomas Probst, Luc Van Gool
TL;DR
Skeleton action recognition needs representations that handle temporal misalignment and high-dimensional Lie group trajectories more effectively than shallow, two-step methods. The paper introduces LieNet, a deep architecture with manifold-preserving mappings, pooling, and logarithm mapping, and reports superiority over shallow Lie group methods and most conventional deep models on standard datasets.
Problem
Existing Lie group action-recognition methods face temporal misalignment, high-dimensional representations, and costly computation, often using dynamic time warping in a two-step pipeline.
Method
LieNet uses rotation mapping and rotation pooling layers on Lie group data, followed by logarithm mapping for regular output layers and manifold-aware weight optimization.
Results
LieNet outperforms existing shallow Lie group feature-learning methods and most conventional deep learning methods on standard 3D human action-recognition datasets.
Takeaways & Limitations
Deep learning on Lie groups provides an effective way to learn spatial representations for skeleton-based action recognition.
Takeaways & Limitations
The paper leaves multiple extensions open, including deeper end-to-end learning from raw joints, temporal modeling, and additional Lie group layers or pooling operations.
Abstract
from arXiv · showhide
In recent years, skeleton-based action recognition has become a popular 3D classification problem. State-of-the-art methods typically first represent each motion sequence as a high-dimensional trajectory on a Lie group with an additional dynamic time warping, and then shallowly learn favorable Lie group features. In this paper we incorporate the Lie group structure into a deep network architecture to learn more appropriate Lie group features for 3D action recognition. Within the network structure, we design rotation mapping layers to transform the input Lie group features into desirable ones, which are aligned better in the temporal domain. To reduce the high feature dimensionality, the architecture is equipped with rotation pooling layers for the elements on the Lie group. Furthermore, we propose a logarithm mapping layer to map the resulting manifold data into a tangent space that facilitates the application of regular output layers for the final classification. Evaluations of the proposed network for standard 3D human action recognition datasets clearly demonstrate its superiority over existing shallow Lie group feature learning methods as well as most conventional deep learning methods.
1. Introduction
Skeleton action-recognition methods use Lie group representations but face temporal misalignment, high dimensionality, and costly nonlinear computation. The paper addresses these issues with a deep Lie group network trained through manifold-aware optimization.
- Motivation: Lie group methods often require dynamic time warping to handle speed variations and temporal misalignment.This adds computation and produces a two-step system that typically performs worse than end-to-end learning.
- Motivation: Lie group representations can become extremely high-dimensional because features are extracted per skeletal segment and stacked.Computations on the resulting nonlinear trajectories are therefore expensive and complicated.
- Approach: The proposed LieNet uses rotation mapping layers to transform input Lie group features into better-aligned representations.The design is motivated by deep networks and classical manifold learning theory.
- Approach: Rotation pooling layers reduce the dimensionality of Lie group features within the deep architecture.
- Contribution: The paper introduces a neural architecture that deeply learns Lie group representations for skeleton-based action recognition.
- Contribution: LieNet incorporates Lie group structure into deep learning and generalizes traditional neural networks to non-Euclidean Lie groups.
- Training: A variant of stochastic gradient descent is used to train the network within the backpropagation framework.
2. Relevant Work
Prior work applied Lie group theory and non-Euclidean ideas to neural networks, but this paper identifies a distinct gap in deep architectures operating directly on Lie groups for skeleton action recognition.
- Existing approaches: Non-Euclidean deep learning research included graph convolutions and recurrent models for spatio-temporal graphs.
- Existing approaches: Earlier studies applied Lie group theories to regular neural networks and generalized common networks to non-Euclidean domains.
- Research gap: The paper presents its architecture as the first work studying deep learning on Lie groups for skeleton-based action recognition.
3. Lie Group Representation for Skeletal Data
The representation encodes pairwise skeletal-part geometry with rotation matrices, yielding motion trajectories on products of special orthogonal groups. The architecture is illustrated with Lie group data and weight spaces in its RotMap and RotPooling layers.
- Skeleton representation: Each skeleton comprises body joints and oriented rigid-body bones, whose pairwise relative geometry is represented in local coordinate systems.The local system is obtained by minimally rotating one body part so its starting joint is the origin and it aligns with the x-axis.
- Skeleton representation: The representation uses both directional rotation matrices between each pair of body parts to encode their relative geometry.
- Lie group structure: A motion sequence is represented as a curve on a product of SO3 Lie groups.The special orthogonal group is a differentiable Riemannian manifold with tangent spaces at each point.
- Lie group structure: Logarithm and exponential maps at a rotation provide manifold-to-tangent-space and inverse mappings expressed through matrix logarithm and exponential.
- LieNet architecture: Figure 1 assigns Lie group data spaces to RotMap and RotPooling layers and Lie group weight spaces to RotMap layers.
4. Lie Group Network for Skeleton-based Action Recognition
LieNet incorporates Lie group structure into a deep architecture for skeleton-based action recognition, using RotMap, RotPooling, and LogMap layers to learn aligned, compact, classifiable representations.
- RotMap Layer: LieNet transforms input rotation matrices into new valid Lie group representations intended to align skeletal features more accurately for matching.RotMap layers use transformation matrices constrained as rotations, preserving the product-manifold structure.
- RotPooling Layer: RotPooling reduces Lie group feature dimensionality through spatial and temporal pooling, but the architecture retains only the first spatial pooling step to avoid misalignment.Spatial pooling operates on paired basic bones; temporal pooling produces more compact motion-sequence representations and reduces model complexity.
- RotPooling Layer: The spatial max-pooling rule selects rotation representations using an ordering based on the angle representation of each rotation matrix.The paper adopts angle form rather than quaternion or axis-angle ordering because angle comparisons are simpler to define.
- LogMap Layer: The LogMap layer flattens product Lie-group data into its Lie algebra, whose skew-symmetric matrices lie in Euclidean space for subsequent regular layers.The logarithm map is computed through an axis-angle relationship, enabling traditional element-wise matrix-gradient calculation.
- Output Layers: After LogMap, outputs can be vectorized and concatenated frame by frame before applying regular ReLU and fully connected layers.This provides the interface between manifold-valued representations and conventional output layers.
5. Training Procedure
LieNets are trained by backpropagation-based stochastic gradient descent, with manifold-aware updates for rotation-mapping weights on SO3. Gradients for ordinary layers follow the chain rule, while RotMap weights require Riemannian gradients and retraction-based updates.
- Optimization setup: LieNets are represented as compositions of layer functions and trained with stochastic gradient descent using a layer-specific loss derived from the final output loss.The model parameters comprise the weights of all layers, and each intermediate loss is defined through the remaining network composition.
- Backpropagation: Backpropagation computes gradients for each layer's weights and lower-layer data through the chain rule.The weight gradient updates Wk, while the data gradient propagates to update Rk−1.
- Backpropagation: RotPooling and LogMap data gradients are propagated through the usual backpropagation framework.RotPooling uses an approach analogous to max pooling, while LogMap gradients are computed element-wise on the involved rotation matrices.
- Manifold-aware updates: RotMap weights require Riemannian rather than Euclidean gradients because they are constrained to the SO3 manifold.Using Euclidean gradients alone would not produce valid rotation weights.
- Manifold-aware updates: The Riemannian gradient is obtained by parallel transporting Euclidean gradients to the relevant tangent space before updating manifold-constrained weights.The resulting update searches along a tangent direction and maps the updated weight back onto SO3 through retraction.
- Manifold-aware updates: The SO3 weight update uses a retraction operation and a learning rate to return tangent-space updates to the rotation manifold.The notation identifies Γ as the retraction operation and λ as the learning rate.
6. Experiments
Experiments on three standard 3D action datasets evaluate LieNet against shallow manifold methods and deep networks, while ablations examine its layers, pooling schemes, representations, and convergence.
- Datasets and protocols: LieNet is evaluated on G3D-Gaming, HDM05, and NTU RGB+D using dataset-specific sequence lengths and established testing protocols.Sequences are normalized to lengths 100, 16, and 64 for G3D-Gaming, HDM05, and NTU RGB+D, respectively.
- G3D-Gaming results: On G3D-Gaming, the three-block LieNet achieves the best performance among the compared shallow and reported methods, without Fourier Temporal Pyramid post-processing.The comparison includes RBM-HMM, SE, and SO; FTP raises the SE and SO accuracies to 91.09% and 90.94%, respectively, but those values are excluded from the main comparison.
- G3D-Gaming results: The G3D-Gaming ablations verify the usefulness of RotMap, RotPooling, and LogMap layers, while the LieNet with and without DTW performs approximately equally at 88.89% and 89.10%.The authors interpret RotMap as taking the role of DTW for speed variations; adding a ReLU-like layer after LogMap instead reduces performance to 87.58%.
- Architecture ablations: A regular eight-layer LeNet-like network reaches 85.49% on G3D-Gaming, supporting the claim that improvements arise from RotMap and RotPooling rather than depth alone.The comparison network applies fully connected and max-pooling operations to concatenated Euclidean outputs of LogMap.
7. Summary and Future Work
The paper presents LieNet as a successful deep architecture for skeleton-based action recognition, addressing speed variation and high-dimensional Lie group features through specialized layers. Evaluations on three standard 3D action datasets demonstrate effectiveness, while several extensions remain open.
- Summary: LieNet uses mapping, pooling, and logarithm layers to process Lie group features for skeleton-based action recognition.The logarithm mapping produces Euclidean vectors for regular output layers.
- Summary: The architecture is designed to address speed variation and high dimensionality in Lie group features.
- Summary: Evaluations on three standard 3D action datasets demonstrate the proposed network’s effectiveness and compare different configurations.
- Summary: The authors also visualize the network to disclose aspects of its intrinsic mechanism.
- Future Work: Future work includes multiple rotation mappings, Lie-group nonlinear layers, deeper raw-joint-to-Lie learning, and temporal modeling.These directions are presented as open issues or potentially more effective extensions.