Source-linked AI summary
Lie-X: Depth Image Based Articulated Object Pose Estimation, Tracking, and Action Recognition on Lie Groups
Chi Xu, Lakshmi Narasimhan Govindarajan, Yu Zhang, Li Cheng
TL;DR
Depth-image pose estimation, tracking, and action recognition are often treated separately, and existing approaches commonly use distinct principles or focus on a single object type. Lie-X uses a unified Lie group-based representation for these tasks and reports competitive performance across fish, mouse, and human-hand applications.
Problem
Pose estimation, tracking, and action recognition from depth images are closely related but are commonly addressed with diverse principles and methods focused on particular articulated-object types.
Source: §1 · p. 1Method
Lie-X uses a unified Lie group paradigm that represents poses as manifold points, tracks as curves, and actions as curve segments, with learned pose scoring and Brownian-motion particle sampling for tracking.
Source: §1 · p. 1, §4.1 · p. 9Results
Lie-X reports competitive performance across fish, mouse, and human-hand applications; on the NYU hand benchmark, its average joint error is 14.51 mm, described as best to the authors’ knowledge.
Source: §1 · p. 2, §5.2 · p. 18, §6 · p. 21Takeaways & Limitations
The unified approach is evaluated across lab animals and human hand, and its CPU pose-estimation implementation runs at 83–267 FPS on a desktop computer.
Source: §5.2 · p. 20, §6 · p. 21Pose-estimation errors remain, with Lie-Hand deviations from ground truth most significant and Lie-Fish results best aligned.
Source: §5.2 · p. 20
Abstract
from arXiv · showhide
Pose estimation, tracking, and action recognition of articulated objects from depth images are important and challenging problems, which are normally considered separately. In this paper, a unified paradigm based on Lie group theory is proposed, which enables us to collectively address these related problems. Our approach is also applicable to a wide range of articulated objects. Empirically it is evaluated on lab animals including mouse and fish, as well as on human hand. On these applications, it is shown to deliver competitive results compared to the state-of-the-arts, and non-trivial baselines including convolutional neural networks and regression forest methods.
1 Introduction
The paper addresses fragmented approaches to depth-image pose estimation, tracking, and action recognition by proposing a unified Lie group-based paradigm for diverse articulated objects.
1 Introduction
- Lie-X represents pose as a manifold point, a motion track as a curve, and an action as a curve segment, framing three depth-image tasks within one Lie group-based paradigm.Pose estimation infers a point, action recognition classifies a curve segment, and tracking uses Brownian motion on Lie groups to generate pose candidates.Source: §1 · p. 1
- An iterative learned pipeline uses multiple initial poses to handle location, orientation, and size variation, then scores candidate poses with a metric trained to mimic practical evaluation.The learned approach replaces traditional Jacobian matrices for the inverse kinematics problem.Source: §1 · p. 1
- The approach demonstrates competitive performance on fish, mouse, and human hand across imaging modalities, while pose estimation runs at around 83-267 FPS on a desktop computer without GPUs.Source: §1 · p. 2
- The work creates annotated depth-image and video datasets of articulated objects and states that the datasets and code will be made publicly available.The evaluated imaging devices include structured-illumination and light-field technologies.Source: §1 · p. 2
2 Related Work
Prior work addresses articulated-object pose estimation, tracking, and action recognition through varied methods and often task- or object-specific approaches; Lie-X positions its contribution as a unified 3D treatment across object categories.
2 Related Work
- Hand-pose studies include CNN methods, deep-learning feedback, geometry-aware modelling, and a theoretically motivated random forest method.Other hand-pose work uses hierarchical sampling, gradient-based optimization, matrix completion, and personalization.Source: §2 · p. 2
- Tracking approaches include particle swarm optimization, PSO combined with ICP, learned salient points, and volumetric tracking-by-detection.Action-recognition work includes recurrent-net variants, CNN and Fourier temporal-pyramid features, and joint pose/action modelling.Source: §2 · pp. 2–3
- Fish and mouse research is described as relatively sparse and mostly 2D, with emphasis on pose estimation; cross-category work includes zebrafish, lab mouse, and human face.A study of group-level mouse behavior uses a simplified straight-line skeleton.Source: §2 · p. 3
- Related Lie-based work covers rigid-object detection and tracking in 2D, full-body pose estimation, and action features, whereas Lie-X targets three related problems in 3D across objects.Other group-action approaches address 2D pose estimation across object types or 3D hand pose.Source: §2 · p. 3
3 Notations and Mathematical Background
The mathematical setup represents articulated skeletons through rigid transformations and kinematic chains on Lie groups. It introduces Lie algebras, exponential maps, chain composition, and manifold-based motion sampling as tools used later in the paper.
3 Notations and Mathematical Background
- Fish and mouse are modelled as single kinematic chains, while the hand is a multi-chain tree; poses are joint locations or the SE(3) transformations applied to a home position.The skeletal models use joints connected by fixed-length bones, and the hand contains five chains sharing a base joint.Source: §3 · pp. 3–4, Figure 2 · p. 5
- SE(3) represents rigid rotations and translations, and products of SE(3) transformations along a kinematic chain remain a Lie group.The paper uses this group structure to represent articulated skeletal models.Source: §3 · p. 4
- A Lie algebra is the tangent space at the identity; in se(3), a twist’s angular and translational components describe motion around a joint, and its exponential realizes a rigid motion.The exponential map is introduced as the matrix exponential from se(3) to SE(3).Source: §3 · pp. 4–5
- Forward kinematics for a partial chain is expressed as a product of exponentials, while the adjoint representation relates a joint’s body- and spatial-coordinate twists.The partial chain becomes the full chain when all joints are included.Source: §3 · pp. 5–6
- Brownian motion on manifolds generalizes Gaussian sampling; the paper approximates left-invariant motion on SE(3) with Gaussian increments and interpolates sampled points by geodesics.The increments are independent and Gaussian-distributed, with a covariance matrix specifying their distribution.Source: §3 · p. 6
4 Our Approach
This section presents Lie-X’s Lie-group framework for articulated-object pose estimation, tracking, and action recognition, including iterative refinement of skeletal estimates and selection among candidate poses. It describes the pose-indexed depth features and random-forest split tests used by the regressors, and how the three modules produce their outputs.
4 Our Approach
- The skeletal models represent fish and mouse as single kinematic chains and the human hand as multiple chains, with joint degrees of freedom tailored to each anatomy.The models contain 21 fish joints, 5 mouse joints, and 23 hand joints; initialization centers the base joint on the foreground point cloud and perturbs orientation and bone lengths.Source: §4 · pp. 6–7
4.1 Pose Estimation
- Local regressors iteratively correct each joint through Lie-group actions, replacing Jacobian-based inverse-kinematics updates with learned local rules.Multiple initial poses are refined into candidates, then a learned scoring function that mimics average joint error selects the final estimate.Source: §4.1 · pp. 7–9
4.2 Tracking
- Tracking applies Brownian-motion particle sampling only to the base joint, while estimating the remaining joints with the pose-estimation inference procedure.The base joint is treated as crucial because errors in its prediction can seriously damage the remaining joint estimates; particle weights use the learned metric.Source: §4.2 · pp. 9–10
4.3 Action Recognition
- Action recognition treats pose sequences as curves on a manifold and learns a predictor to classify their action categories.Sequences are normalized to 32 frames; temporal-pyramid features summarize Lie-algebra pose representations, with a compact 252-dimensional representation for fish actions.Source: §4.3 · pp. 11–12
4.4 Random Forests, Pose-indexed Depth features, and Binary Tests
- Action recognition uses a unique feature set, while pose-indexed depth features serve the first two regressors; random forests are used in practice, though other reasonable learners are possible.Source: §4.4 · p. 12
- Pose-indexed depth features extend depth features with pose-indexed features, projecting offsets transformed by the current pose into image-space samples.The projections provide depth-sample locations for feature construction.Source: §4.4 · p. 12
- A binary test pairs a feature with a threshold, sending a pixel left when its feature value exceeds the threshold and right otherwise.Source: §4.4 · p. 12
- At each split, candidate features and thresholds are tested for gain; trees stop at L levels or when a node has fewer than ln training examples.The gain is defined using the partition of training examples and deviations for the joints.Source: §4.4 · pp. 12–13
- The first two regressors use mean-shift mode search in Hough voting space, while action recognition selects the category with the largest averaged-histogram count using classical random forests.Source: §4.4 · p. 13
5 Empirical Evaluations
The evaluations test pose estimation on fish, mouse, and hand, then assess mouse tracking and fish action recognition. Lie-X compares favorably with pose-estimation baselines, while temporal tracking and tangent-vector action features also improve reported results.
5.1 Datasets
- On NYU hand, evaluation follows the 14-joint protocol out of 23 joints; the experiments also span fish light-field depth and structured-illumination depth for mouse and hand.The hand subset includes fingertips and the hand base.Source: §5.1 · pp. 14, 17
5.2 Pose Estimation of Fish, Mouse, and Human Hand
- Lie-X outperforms RF and CNN on fish and mouse, and reaches 14.51 mm average joint error on NYU hand versus 16.50 mm for.The fish/mouse comparison is reported in Table 1; the hand result is reported in Table 2.Source: Table 1 · p. 17, §5.2 · p. 18, Table 2 · p. 18
- Multiple initial poses improve pose estimation, while removing the learned metric degrades performance; the authors report stable behavior across broad parameter ranges.The learned metric scores candidate poses, whereas its ablation averages candidates joint-by-joint; selected parameters balance performance and efficiency.Source: §5.2 · pp. 18–19
- The CPU implementation runs at 83 FPS for fish, 267 FPS for mouse, and 123 FPS for hand; the hand speed is comparable to most GPU-based methods.The runtime measurements used an Intel i7-960 desktop PC.Source: Table 3 · p. 19, §5.2 · p. 20
- Common pose errors include orientation flips, z-direction displacement, and sub-optimal shape fitting, with the most noticeable visual discrepancies on hand.The authors relate the pattern to task complexity, from simpler kinematic chains to more complex kinematic trees.Source: §5.2 · p. 20
5.3 Tracking of Mouse
- Mouse tracking reduces average joint error to 7.19 mm from 8.42 mm for single-frame pose estimation, with generally less noisy predictions.Some tracking frames are slightly worse, which the authors attribute to averaging operations in the tracker.Source: §5.3 · p. 20
5.4 Action Recognition of fish
- On the nine-class fish action dataset, tangent-vector features achieve 91.33% average classification accuracy, versus 79.19% for joint-position features.The authors report improved separation among similar scoot, J-turn, C-turn, and R-turn actions.Source: §5.4 · pp. 20–21
6 Conclusion and Future Work
The paper proposes a unified Lie group approach to pose estimation, tracking, and action recognition for diverse articulated objects from depth images, and reports very competitive performance on human hand, fish, and mouse datasets. Future work includes more diverse objects and interactions with other articulated or background objects.
6 Conclusion and Future Work
- A unified Lie group approach addresses pose estimation, tracking, and action recognition for diverse articulated objects from depth images.Source: §6 · p. 21
- The approach is evaluated on human hand, fish, and mouse datasets with very competitive performance.Source: §6 · p. 21
- Future work targets more diverse articulated objects and their interactions with other articulated objects and background objects.Source: §6 · p. 21