Source-linked AI summary
AvatarPoser: Articulated Full-Body Pose Tracking from Sparse Motion Sensing
Jiaxi Jiang, Paul Streli, Huajian Qiu, Andreas Fender, Larissa Laich, Patrick Snape, Christian Holz
TL;DR
Sparse head-and-hand tracking supports Mixed Reality interaction but leaves full-body avatar motion underdetermined and often requires extra body sensors or pelvis tracking. AvatarPoser predicts full-body poses from only headset and hand inputs using Transformer features, global-motion decoupling, and inverse-kinematics refinement. It achieves state-of-the-art performance on AMASS and related evaluations while offering real-time-capable inference, with iterative IK adding a measurable runtime cost.
Problem
Sparse Mixed Reality tracking provides head and hand signals but insufficient information for full-body avatar pose estimation, while added pelvis or body sensors increase setup complexity.
Method
AvatarPoser uses a Transformer encoder with global-motion decoupling and combines learned pose prediction with inverse kinematics to match tracked hand positions.
Results
AvatarPoser achieves state-of-the-art accuracy on AMASS and outperforms other methods across the reported metrics, with network inference reaching up to 662 fps.
Takeaways & Limitations
The method provides a practical learning-based inverse-kinematics solution for full-body avatar control from current MR headset tracking inputs.
Takeaways & Limitations
Inverse-kinematics refinement costs approximately 6 ms per iteration, and the method uses five iterations to balance inference speed with final hand-position accuracy.
Abstract
from arXiv · showhide
Today's Mixed Reality head-mounted displays track the user's head pose in world space as well as the user's hands for interaction in both Augmented Reality and Virtual Reality scenarios. While this is adequate to support user input, it unfortunately limits users' virtual representations to just their upper bodies. Current systems thus resort to floating avatars, whose limitation is particularly evident in collaborative settings. To estimate full-body poses from the sparse input sources, prior work has incorporated additional trackers and sensors at the pelvis or lower body, which increases setup complexity and limits practical application in mobile settings. In this paper, we present AvatarPoser, the first learning-based method that predicts full-body poses in world coordinates using only motion input from the user's head and hands. Our method builds on a Transformer encoder to extract deep features from the input signals and decouples global motion from the learned local joint orientations to guide pose estimation. To obtain accurate full-body motions that resemble motion capture animations, we refine the arm joints' positions using an optimization routine with inverse kinematics to match the original tracking input. In our evaluation, AvatarPoser achieved new state-of-the-art results in evaluations on large motion capture datasets (AMASS). At the same time, our method's inference speed supports real-time operation, providing a practical interface to support holistic avatar control and representation for Metaverse applications.
1 Introduction
Mixed Reality systems provide sparse head-and-hand tracking that supports interaction but leaves full-body avatar motion underdetermined and can reduce immersion. AvatarPoser addresses this gap using only those inputs, combining learned pose estimation with inverse-kinematics refinement for accurate, practical full-body control.
- Motivation: Sparse head-and-hand tracking supports immersive interaction but does not provide complete body tracking for users or other participants.This limitation is especially apparent when users see their own bodies or when collaborative applications represent participants only by hovering upper bodies.
- Motivation: Full-body pose estimation from sparse tracking is vastly underdetermined, while existing approaches use constrained motion diversity, additional body trackers, or an implicit pelvis input.These requirements increase setup complexity or limit the breadth of recoverable motion.
- Approach: AvatarPoser predicts full human poses from only headset and hand or controller poses using a Transformer-based method that decouples global motion from learned pose features.The method is designed to operate without pelvis location or inertial trackers and to recover motion across a wide variety of motion classes.
- Approach: Inverse kinematics refines predicted arm articulation to match tracked hand positions, balancing full-body style realism with accurate hand control.The refinement addresses errors that accumulate through the avatar’s kinematic chain.
- Results: 662 fps is the reported maximum network inference rate, while AvatarPoser achieves state-of-the-art accuracy on AMASS and generalizes to recorded HTC VIVE inputs.The authors present the method as suitable for practical AR and VR applications using current headset tracking information.
2 Related Work
Prior sparse-input full-body pose methods have traded motion diversity or practical simplicity against tracking requirements and generality. AvatarPoser’s related work context combines Transformer modeling with learning-based and traditional inverse-kinematics approaches to address these constraints.
- Sparse-input pose estimation: Many prior full-body pose systems use up to six body-worn inertial sensors, making motion capture inflexible and unwieldy.The sensors are distributed across the head, arms, pelvis, and legs.
- Sparse-input pose estimation: LoBSTr predicts lower-body pose from sequences of tracking signals, extending sparse-input estimation with a recurrent GRU-based model.The cited passage situates LoBSTr among methods addressing sparse full-body pose estimation.
- Sparse-input pose estimation: CoolMoves uses only headset and hand-held controller inputs but interpolates from a smaller dataset with specific activities, leaving scalability to diverse data unclear.Its KNN-based design is contrasted with methods intended for larger and more varied datasets.
- Transformer models: Transformers have improved performance across vision tasks and have also been applied to human pose estimation.This motivates Transformer-based modeling as part of the paper’s approach.
- Inverse kinematics: AvatarPoser combines a deep neural network with inverse-kinematics optimization to refine arm articulation toward tracked hand positions.The combination follows prior efforts to make inverse-kinematics prediction more robust and flexible.
3 Method
AvatarPoser reconstructs full-body pose in world coordinates from sparse head-and-hand motion using a time-series Transformer, global-motion stabilization, kinematics, and inverse-kinematics refinement.
- Problem formulation: AvatarPoser maps sparse headset and hand positions and orientations to full-body joint positions in world coordinates.The formulation uses the SMPL skeleton, representing the first 22 joints and ignoring finger poses.
- Input and output representation: The input concatenates position, linear velocity, rotation, and angular velocity for each sparse tracker, yielding 54 features per timestep for three trackers.Rotations and angular velocities use 6D representations.
- Input and output representation: The network predicts global pelvis orientation and local joint rotations, producing 132 output dimensions per timestep for 22 joints.The pelvis rotation represents global orientation, while other rotations are defined relative to parent joints.
- Overall framework: A Transformer Encoder extracts temporal pose features, while the Stabilizer decouples global orientation and estimates global translation through the body kinematic chain.The Stabilizer is a 2-layer MLP operating on 256-dimensional Transformer features.
- Overall framework: Forward kinematics converts predicted rotations into joint positions, and inverse kinematics adjusts shoulder and elbow rotations to reduce hand-position errors.The IK optimization uses the network output, fixes the shoulder position, and stops after a fixed number of iterations for faster inference.
- Training objective: Training combines L1 local-rotation, global-orientation, and positional losses, while the IK module is excluded from training for faster optimization.The loss weights λori, λrot, and λfk are 0.05, 1, and 1, respectively.
4 Experiments
AvatarPoser is evaluated on AMASS subsets using standard pose, position, and velocity metrics, comparisons against sparse-input baselines, cross-dataset testing, ablations, runtime measurements, and live VR recordings.
- Data preparation: The experiments use CMU, BMLrub, and HDM05 subsets of AMASS, with 90% training and 10% testing splits.Inputs are unified to 60 Hz for VR-device use.
- Evaluation protocol: AvatarPoser is compared with Final IK, CoolMoves, LoBSTr, and VAE-HMD using MPJRE, MPJPE, and MPJVE.Methods are trained and tested on the same data, with three-input and four-input settings considered.
- Evaluation results: AvatarPoser achieves the best results on all three reported metrics for both four-input and three-input evaluations.Final IK performs worst on MPJPE and MPJRE, while the results support data-driven motion learning from mocap datasets.
- Generalization: Cross-dataset evaluation gives AvatarPoser the best results over almost all metrics across CMU, BMLrub, and HDM05.Final IK is slightly better on MPJVE for CMU, but has much larger MPJRE and MPJPE errors.
- Ablation studies: Removing the Stabilizer increases MPJPE, while directly regressing pelvis translation performs worse than computing it through the kinematic chain.The Stabilizer learns global orientation from richer network features rather than relying directly on head rotation.
- Ablation studies: Removing the IK module increases hand position error by almost 41%, while removing the FK module increases MPJPE and decreases MPJRE.The IK module specifically improves hand-position accuracy, whereas the FK module accounts for joint positions after forward kinematics.
- Runtime and live evaluation: Live recordings from an HTC VIVE headset and two controllers are used to qualitatively assess robustness on real six-degree-of-freedom inputs.The recorded-input evaluation compares user poses with AvatarPoser predictions.
5 Conclusions
AvatarPoser estimates realistic full-body human poses from only Mixed Reality headset and hand motion signals. Its combination of global-motion guidance and model-based optimization supports robust pose estimation and accurate hand control for VR/AR applications.
- AvatarPoser estimates realistic human poses using only Mixed Reality headset and hand or controller motion signals.
- The method decouples global motion from learned pose features to remain robust without pelvis signals.
- Combining learning-based prediction with model-based optimization balances full-body style realism and accurate hand control.
- Experiments on AMASS showed that AvatarPoser surpasses state-of-the-art methods for full-body pose estimation.
- The authors position AvatarPoser as a useful learning-based inverse-kinematics solution for practical VR/AR applications.