Source-linked AI summary
Physical Inertial Poser (PIP): Physics-aware Real-time Human Motion Tracking from Sparse Inertial Sensors
Xinyu Yi, Yuxiao Zhou, Marc Habermann, Soshi Shimada, Vladislav Golyanik, Christian Theobalt, Feng Xu
TL;DR
Sparse inertial motion capture is ambiguous and can produce unstable, non-physical motion. PIP combines a neural kinematics estimator with a physics-aware optimizer using six IMUs, and experiments report improved accuracy, temporal stability, and physical correctness.
Problem
Sparse inertial motion capture is inherently ambiguous, struggles with long-period motions, and commonly produces artifacts because physical constraints are not modeled.
Method
PIP first regresses motion status from six IMUs with a neural kinematics estimator, then refines it using a physics-aware motion optimizer.
Results
PIP significantly outperforms previous sparse IMU-based methods in tracking accuracy, physical plausibility, and challenging-pose disambiguation.
Takeaways & Limitations
Combining learned kinematics with physics optimization yields higher-accuracy, more realistic sparse-IMU motion capture and supports live applications.
Takeaways & Limitations
The method assumes flat ground and a known body shape, so it cannot capture humans walking upstairs without model adaptation.
Abstract
from arXiv · showhide
Motion capture from sparse inertial sensors has shown great potential compared to image-based approaches since occlusions do not lead to a reduced tracking quality and the recording space is not restricted to be within the viewing frustum of the camera. However, capturing the motion and global position only from a sparse set of inertial sensors is inherently ambiguous and challenging. In consequence, recent state-of-the-art methods can barely handle very long period motions, and unrealistic artifacts are common due to the unawareness of physical constraints. To this end, we present the first method which combines a neural kinematics estimator and a physics-aware motion optimizer to track body motions with only 6 inertial sensors. The kinematics module first regresses the motion status as a reference, and then the physics module refines the motion to satisfy the physical constraints. Experiments demonstrate a clear improvement over the state of the art in terms of capture accuracy, temporal stability, and physical correctness.
1. Introduction
PIP addresses the ambiguity and physical artifacts of sparse inertial motion capture with a six-IMU, two-stage method combining learned kinematics and physics-based optimization.
- PIP runs at 60 frames per second with 16ms latency for real-time applications.
- Sparse inertial capture avoids image occlusions and restricted camera space, but remains ambiguous and physically challenging.
- PIP estimates human motion, joint torques, and ground reaction forces using only six IMUs.
- The method first regresses motion status with learned kinematics, then optimizes motion using physical constraints.
- A learning-based RNN initialization strategy helps disambiguate motion from sparse inertial measurements.
- A dual PD controller combines local and global pose control to improve tracking accuracy and physical plausibility.
2. Related Work
Prior inertial and hybrid approaches reduce some optical-capture limitations, but sparse inertial capture remains ambiguous and prone to delay and non-physical artifacts.
- Inertial mocap avoids occlusions and restricted moving space, while commercial systems commonly require inconvenient suits with 17 IMUs.
- Sparse inertial sensing is preferable for flexibility but makes motion capture highly ambiguous and challenging.
- Previous sparse methods require future information, cannot stably capture ambiguous poses, and produce artifacts such as jitter and foot-sliding.
- Prior physics-aware approaches impose constraints or reconstruct forces, but naive constraints are difficult to apply to articulated human motion.
3. Method
The proposed real-time system tracks human motion from six IMUs and augments kinematic estimates with physics-based refinement and physical-property outputs.
- The input is sequential acceleration and orientation measurements from six IMUs mounted on the forearms, lower legs, head, and pelvis.
- The output includes joint angles, global translation, ground reaction forces, and joint torques.
- A neural kinematics estimator infers motion, and a physics-aware optimizer refines it while outputting physical properties.
3.1. Neural Kinematics Estimator
The neural kinematics estimator uses recurrent networks to infer full-body motion status from sparse IMUs, while retaining historical information to resolve pose ambiguity.
- The estimator uses calibrated six-IMU inputs and predicts joint positions, velocities, and rotations in the root-relative coordinate frame.
- An RNN-based multi-stage network predicts leaf positions, full joint positions, rotations, all-joint velocities, and foot-ground contact probabilities.
- Sparse IMUs cannot distinguish standing still from sitting still because both produce identical orientations and zero accelerations.
- Unlike fixed-window biRNNs, RNNs retain complete historical information to capture state changes outside a limited temporal window.
- The learning-based initialization strategy avoids constant zero hidden states by using the known initial pose during training and inference.
3.2. Physics-aware Motion Optimizer
The physics-aware motion optimizer refines kinematic estimates into motion, torques, and ground reaction forces that satisfy physical constraints. Its dual PD controller jointly controls local rotations and global positions while the optimizer enforces dynamics, contact, friction, and regularization constraints.
- Dynamics module: The dynamics module takes estimated motion status as reference and optimizes physically realizable motion, joint torques, and ground reaction forces.It uses a torque-controlled floating-base simulated character, then updates the character state to produce the final motion.
- Physics model: The physics model represents pose and translation as generalized coordinates, with accelerations and forces governed by the character’s equation of motion.The model permits a small residual force at the root to compensate for mismatch between the simulated model and real humans.
- Dual PD controller: The dual PD controller combines a local joint-rotation controller with a joint-position controller that constrains the global pose while preserving a quadratic optimization problem.The rotation controller derives desired angular acceleration from reference rotations, while the position controller predicts reference positions from current positions and estimated velocities.
- Motion tracking optimizer: The motion-tracking optimizer solves a quadratic program for generalized acceleration, joint torques, and ground reaction forces subject to equations of motion, friction-cone, and no-sliding constraints.The contact constraints use the contact Jacobian and characterize physically admissible ground interactions.
- Objective and regularization: The objective combines PD tracking with regularization that penalizes contact-force energy, root residual force, and joint-torque magnitudes.The regularization weights for contact forces, root residuals, and joint torques are 10, 0.1, and 0.01, respectively.
- Contact constraints: The friction constraint requires upward vertical ground force and bounds horizontal forces by friction, using an empirical friction coefficient µ = 0.6.These friction-cone and sliding constraints are applied only at contact points.
4. Experiments
Experiments evaluate PIP against sparse-IMU baselines on DIP-IMU and TotalCapture, including accuracy, stability, physical plausibility, latency, qualitative reconstruction, and ablations. PIP achieves the strongest overall results, while its components reduce translation drift, jitter, and physical artifacts.
- Quantitative comparison: PIP reduces pose error by 15%, jitter by 87%, motion imbalance by 89%, and latency by 83% versus TransPose.Table 1 compares state-of-the-art methods on DIP-IMU and TotalCapture.
- Quantitative comparison: PIP achieves the lowest cumulative translation error on TotalCapture.The result is attributed to learning-based RNN initialization and the dual PD controller.
- Qualitative comparison: On TotalCapture, PIP visually provides the most accurate mesh reconstructions across the compared methods.The qualitative evaluation reports mesh error distributions, AUC values, and four representative examples.
- Physical properties: Estimated ground-reaction forces approximately equal gravity during walking, while arm torques increase when waving starts and stops.The torque increase is associated with acceleration during motion onset and cessation.
- Ablation studies: Removing learned RNN initialization makes the system less accurate and stable, especially on highly ambiguous motions such as long sitting.The ablation evaluates upper-leg orientation error over a long-sitting sequence.
- Ablation studies: Removing physics optimization or the dual PD controller significantly worsens translation accuracy and produces floating, sinking, and foot-sliding artifacts.The full method also substantially reduces motion jitter, although SIP error on TotalCapture increases slightly by 0.1° with physics optimization.
- Applications: Reducing latency from 94ms to 16ms supports real-time applications such as gaming, animation, and motion retargeting.The paper identifies the latency reduction as critical for gaming applications.
5. Conclusion and Limitations
PIP combines kinematics and physics modules to estimate human motion, joint torques, and ground reaction forces from six IMUs, while assuming a flat ground that prevents upstairs walking capture.
- PIP is a real-time physics-aware approach estimating human motion, joint torques, and ground reaction forces from six IMUs.
- The method combines kinematics and physics modules to improve motion accuracy and realism.
- Assuming flat ground makes PIP incapable of capturing humans walking upstairs.
A. Implementation Details
The implementation uses recurrent networks with learned state initialization, calibrated inertial inputs, finite-difference velocity targets, and a dual-controller physics design with specified gains.
- Network Structure: The kinematics networks use linear-ReLU layers, two 256-width LSTMs, linear outputs, and 40% dropout.The RNNs PL, PA, RA, VA, and CF share this structure.
- Network Structure: Three-layer fully-connected networks initialize the hidden and cell states of the RNN LSTM layers.The FC widths are 256, 512, and 1024, with ReLU activation.
- Rotation Representation: The inertia input contains calibrated accelerations and rotation matrices, while non-root rotations use a 6D representation.
- Ground-truth joint velocities are synthesized from successive global positions, root orientation, and the frame interval.The velocity target is formed by finite differencing positions in the root coordinate frame.
- With Δt = 1/60, the controller gains are 3600 proportional and 60 derivative, while experiments use 2400 for joint rotation proportional gain.The lower rotational proportional gain produces smoother angular accelerations.
- Table 3 compares PIP with prior online and offline methods across metrics, reporting lower latency for online comparisons and higher physical correctness than offline methods.
B. Comparisons on More Metrics
Additional evaluations show that PIP outperforms previous online methods across metrics with reduced latency, while matching offline capture accuracy and improving physical correctness.
- The evaluation reports angular error, positional error, and relative jitter alongside the main-paper metrics.Positional error is measured in centimeters, while relative jitter is measured in km/s3.
- TotalCapture acceleration measurements contain an obvious constant bias relative to accelerations computed from Vicon subject motions.
- PIP outperforms previous online methods on all reported metrics with largely reduced latency.
- Compared with offline methods, PIP achieves comparable motion accuracy but higher physical plausibility on Absolute Jitter and ZMP Distance.
C. Discussions and Future Works
The paper discusses indirect physics evaluation and long-term inertial drift, reporting stable sitting-pose tracking but acknowledging unavoidable translation drift and limited force ground truth.
- Quantitative Evaluations of Physics: No public dataset provides both IMU measurements and ground-truth joint torques or ground reaction forces.The paper therefore uses qualitative force visualizations and indirect metrics such as jitter and ZMP distance.
- Hard contact constraints make contacting-part sliding and ground-penetration errors zero.
- Drifts in Long-term Tracking: A perfectly still sitting-pose test produces 4.2 degrees of total orientation drift over 1 million frames, or 4.6 hours.
- Drifts in Long-term Tracking: PIP inevitably experiences long-term translation drift, measured at about 4.6% and dependent on how far the subject moves.
- Drifts in Long-term Tracking: The system tracks long-period sitting stably for half an hour in a live experiment without worsening over time.