Source-linked AI summary
QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatars
Alexander Winkler, Jungdam Won, Yuting Ye
TL;DR
Sparse AR/VR sensors provide little information about the lower body, making real-time full-body tracking difficult. QuestSim uses reinforcement learning to drive a physics-simulated avatar from headset and controller signals, trained with dense motion supervision. The system produces physically plausible poses and reconstructs lower-body motion from sparse upper-body input, including the HMD alone.
Problem
AR/VR wearable devices provide sparse body signals with no lower-body information, making physically plausible real-time full-body tracking difficult.
Method
QuestSim trains a reinforcement-learning policy with imitation objectives to output torques for a physics simulator from sparse headset and controller observations.
Results
The framework predicts lower-body pose even from the HMD alone and tracks diverse motions, body sizes, and environments with a single policy.
Takeaways & Limitations
Physics simulation compensates for missing sensor information by synthesizing physically plausible full-body poses from sparse observations.
Takeaways & Limitations
The method can fall on dynamic out-of-distribution motions, may synthesize a lower-body pose that differs from the user's, and currently has 160 ms latency.
Abstract
from arXiv · showhide
Real-time tracking of human body motion is crucial for interactive and immersive experiences in AR/VR. However, very limited sensor data about the body is available from standalone wearable devices such as HMDs (Head Mounted Devices) or AR glasses. In this work, we present a reinforcement learning framework that takes in sparse signals from an HMD and two controllers, and simulates plausible and physically valid full body motions. Using high quality full body motion as dense supervision during training, a simple policy network can learn to output appropriate torques for the character to balance, walk, and jog, while closely following the input signals. Our results demonstrate surprisingly similar leg motions to ground truth without any observations of the lower body, even when the input is only the 6D transformations of the HMD. We also show that a single policy can be robust to diverse locomotion styles, different body sizes, and novel environments.
1 INTRODUCTION
AR/VR avatars require real-time full-body motion tracking, but wearable devices provide sparse signals and no lower-body information. QuestSim combines sparse sensing with physics to synthesize plausible lower-body motion and reduce artifacts.
- AR/VR avatars need accurate real-time full-body motion to support richer self-expression and social experiences.
- Wearable sensing works across rooms and outdoors, but AR/VR signals omit the lower body and leave pose reconstruction under-constrained.
- Sparse inputs make purely kinematic reconstruction difficult because many possible poses can produce artifacts such as jitter, foot skating, and unstable contacts.
- QuestSim uses a physics simulator to constrain reconstructed poses to physically valid solutions.
- A single deep-reinforcement-learning policy tracks users of different heights, predicts lower-body pose even from the HMD alone, and produces fewer artifacts than kinematic approaches.
2 RELATED WORK
Prior motion-tracking methods use vision, body-worn sensors, or HMDs and controllers, while increasingly sparse inputs require more pose synthesis. QuestSim emphasizes physics as a hard constraint for reconstructing plausible full-body motion without lower-body sensors.
- Motion-tracking approaches are categorized by sensor type, with sparser inputs requiring more synthesized pose information.
- Vision-based methods face depth ambiguity and temporal jitter, while IMU methods can depend on entire sensor trajectories or paired motion-capture data.
- HMD-and-controller methods use generative models such as VAEs and flow-based models to infer full-body poses from sparse observations.
- Physics is especially useful for under-constrained sparse sensing because physical laws narrow the many kinematically valid pose solutions.
- Unlike approaches that treat physics as a soft constraint, QuestSim places a physics simulator after the controller so physical laws are enforced as hard constraints.
- QuestSim uses a single end-to-end MLP with only 6 DoF headset-and-controller state and reports believable motion without non-physical forces.
3 METHOD
QuestSim trains a policy with reinforcement learning to maximize discounted return in a physics-simulator environment. PPO learns the neural-network weights from trajectories generated by policy-environment interaction.
- Reinforcement learning trains a policy to maximize expected discounted return over trajectories collected in the environment.
- The policy parameters are neural-network weights, while trajectories contain states and actions generated during interaction with the simulator.
- The trajectory probability combines the initial-state distribution, simulator dynamics, and policy action probabilities.
- Proximal Policy Optimization is used to learn the policy weights.
3.2 Overview
The method reconstructs full-body pose from sparse user observations by having a policy produce torques for a physics simulator. Training uses ground-truth motion, simulated state feedback, user sensor observations, and body-scale information.
- 3.2 Overview: A policy maps sparse user observations to joint torques, and the physics simulator converts those torques into the reconstructed pose.
- 3.2 Overview: Training rewards simulated poses that closely match corresponding ground-truth poses from a motion database.
- 3.2 Overview: Feedback from the simulated character state reduces ambiguity about plausible next poses when user observations are sparse.
- 3.2 Overview: Policy inputs include simulated-character observations, headset and controller observations, and user scale.
- 3.2 Overview: The simulated avatar exposes joint angles, velocities, link positions, and link orientations to the policy.
- 3.2 Overview: Training data pairs synthetically offset headset-and-wrist sensor signals with ground-truth motion, using an in-house dataset of 8 hours from 172 subjects.
3.5 User Pose Observations
User pose observations combine headset and controller positions and orientations, expressed relative to a shared frame. The policy can also use future observations and user scale to improve anticipation and generalize across body sizes.
- User pose observations: The sensor input contains positions and orientations for the headset and both controllers, represented relative to frame S.Controller orientations use the first two columns of their rotation matrices in R6.
- User pose observations: Six future user observations let the avatar anticipate upcoming motion, but introduce 160 ms of latency.The complete user-pose observation has dimensionality R162.
- User scale observations: A single policy is designed to track users across different body scales rather than requiring one policy per user scale.The policy receives user scale so it can adjust torques for differences in mass and inertia.
- User scale observations: Training extracts each subject’s height from the initial A-pose, while inference initializes the avatar from the headset-estimated height.The training clips come from 172 individual subjects.
3.7 Reward
The reward trains the simulated character to imitate ground-truth motion using joint-level and Cartesian similarities. An additional foot-contact term discourages unnatural high-frequency stepping by shaping how the avatar unloads a leg before lifting it.
- Reward components: The imitation reward compares the simulated character with ground truth using joint angles, joint-angle velocities, Cartesian positions, and Cartesian velocities.Each similarity term is expressed with a Gaussian kernel and weighted in the total reward.
- Reward components: Dense full-body supervision is available during training because sparse observations are synthetically generated from known motion-capture poses.Inference still requires only sparse sensor data.
- Foot-contact reward: Without lower-body observations, the basic imitation terms produce short, high-frequency steps that look unnatural.This motivates adding a foot-contact reward term.
- Foot-contact reward: The foot-contact term penalizes abrupt decreases in vertical contact force, encouraging natural leg unloading before foot lift while allowing forceful step-downs.The term considers the left and right feet separately.
3.8 Training
The controller directly outputs torques for a physics simulation, which simplifies control relative to target-angle formulations. The policy and simulation run at 36 fps, while headset data is downsampled from 72 fps during inference.
- Controller and simulation: The policy directly outputs torque values instead of PD target angles, providing stable tracking with less controller code complexity.The comparison includes Stable PD and other more involved controller formulations.
- Controller and simulation: The policy and physics simulation run at 36 frames per second, the largest stable timestep reported by the authors.Headset sensor data arrives at 72 fps and is downsampled during inference.
4 RESULTS
The results evaluate pose reconstruction from synthesized and real Meta Quest headset data, with either headset-plus-controller input or headset-only input. Despite lacking lower-body sensors, the method is reported to match state-of-the-art metrics from a system using leg-mounted IMUs.
- Results: The reconstructed motion distinguishes locomotion modes, turning, transitions, and activities with weaker upper- and lower-body correlation.The reported qualitative behavior includes writing on a whiteboard and boxing, with feet often contacting at the correct time and position.
- Results: Despite having no lower-body sensors, the reported metrics match state-of-the-art PIP results that use IMUs attached to the legs.This comparison is stated in the Table 1 caption.
- Evaluation setup: The method reconstructs full-body motion from Meta Quest headset and controller poses, and can also operate using only the 6 DoF headset pose.Evaluation covers both synthetic Lafan inputs and real headset recordings.
- Evaluation setup: The evaluation reports MPJRE, MPJPE, global root error, SIP error, and jitter across reconstructed poses.MPJRE and SIP are measured in degrees, while MPJPE and root error are measured in centimeters; lower jitter indicates smoother motion.
5 DISCUSSION
The framework extends sparse-sensor tracking across users, avatar scales, and virtual environments, while exposing limitations for out-of-distribution and ambiguous motions.
- Tracking users of different scale: The same policy tracks users of different heights without retraining and can also control avatars of different scales.Inference scale is initialized from the HMD height; larger avatars may adopt more crouched poses while preserving physical validity.
- Sparse sensor input: The sparse input includes headset and controller positions, while orientation information is also used but not visualized.The input visualization identifies headset position as saturated and controller positions as less saturated, with Y as the vertical dimension.
- Environment interaction and adaptation: Physics simulation lets tracked avatars interact with objects and adapt their motion to virtual environments such as rough terrain.Collisions can influence avatar pose, while the avatar can also affect external objects; terrain adaptation changes foot placement to match slopes.
- Limitations: The method can fail entirely on dynamic out-of-distribution motions whose torque controls were not learned, including break-dance and jumping.Because simulated characters cannot teleport and require precise joint-torque sequences, they may drift, fall, or struggle to catch up.
- Limitations: With identical upper-body sensor data, the policy may generate a natural physically valid lower-body pose that does not match the user’s actual pose.This ambiguity arises when upper- and lower-body motions are weakly correlated.
6 CONCLUSION AND FUTURE WORK
The method tracks users from sparse sensor data by using physics simulation to synthesize physically plausible missing poses. Future work targets motion quality, latency, body-shape modeling, and motion diversity.
- Physics simulation compensates for missing sensor information by synthesizing physically plausible poses from sparse sensor data.
- The simulated avatar can still look stiff and unnatural, motivating improved reward strategies or GAN-based approaches.
- The current system has 160 ms latency because it relies on future observations, reducing tracking responsiveness.
- The scale model coarsely approximates users by linearly scaling skeleton elements, despite differing proportions among users of the same height.
- Future work aims to increase the diversity of motions that avatars can imitate.
A ALGORITHM PARAMETERS
The algorithm uses a high-frequency torque policy and value function implemented as multilayer perceptrons, with bounded outputs and Gaussian exploration during training.
- The policy outputs torques at a frequency of 1/36s.
- The policy and value function are multilayer perceptrons with three hidden layers of [400, 300, 200] nodes and tanh activations.
- Gaussian exploration noise during training is 0.03, and policy outputs in [−1, 1] are scaled to [−200, 200]Nm.