Source-linked AI summary
Deep Inertial Poser: Learning to Reconstruct Human Pose from Sparse Inertial Measurements in Real Time
Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll
TL;DR
Full-body pose from six IMUs is under-constrained, and collecting paired inertial and ground-truth data is costly. DIP addresses this with synthetic IMU training, recurrent temporal modeling, and a sliding-window bi-directional RNN. It achieves 15.85° angular error on TotalCapture and 85ms latency, while operating in real time and comparing favorably with SIP.
Problem
Estimating full-body pose from six IMUs is under-constrained, while collecting paired IMU and ground-truth pose data is expensive and difficult in many target settings.
Method
DIP synthesizes IMU orientations and accelerations from Mocap data and uses recurrent neural networks, including a sliding-window bi-directional architecture, to predict SMPL pose parameters.
Results
15.85° angular error on TotalCapture is lower than SIP, while DIP operates in real time with 85ms latency.
Takeaways & Limitations
DIP reconstructs full-body pose from six IMUs in real time without calibrated cameras or explicit smoothness and joint-angle limits.
Takeaways & Limitations
Robustness to more poses, datasets, and settings remains future work, with acceleration modeling identified as a main limitation.
Abstract
from arXiv · showhide
We demonstrate a novel deep neural network capable of reconstructing human full body pose in real-time from 6 Inertial Measurement Units (IMUs) worn on the user's body. In doing so, we address several difficult challenges. First, the problem is severely under-constrained as multiple pose parameters produce the same IMU orientations. Second, capturing IMU data in conjunction with ground-truth poses is expensive and difficult to do in many target application scenarios (e.g., outdoors). Third, modeling temporal dependencies through non-linear optimization has proven effective in prior work but makes real-time prediction infeasible. To address this important limitation, we learn the temporal pose priors using deep learning. To learn from sufficient data, we synthesize IMU data from motion capture datasets. A bi-directional RNN architecture leverages past and future information that is available at training time. At test time, we deploy the network in a sliding window fashion, retaining real time capabilities. To evaluate our method, we recorded DIP-IMU, a dataset consisting of $10$ subjects wearing 17 IMUs for validation in $64$ sequences with $330\,000$ time instants; this constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes.
1 INTRODUCTION
DIP targets real-time, minimally intrusive full-body pose reconstruction from six IMUs in everyday settings. It combines synthetic training data, recurrent modeling, and limited future context to achieve competitive real-time accuracy.
- Pose reconstruction must be real-time, usable in everyday settings, and minimally invasive in user instrumentation.
- Six IMUs make full-body pose estimation severely under-constrained, while long-range temporal information and expensive training data complicate learning.
- DIP synthesizes IMU orientations and accelerations from Mocap sequences with virtual sensors on SMPL meshes, then trains an RNN to predict SMPL parameters.
- Five future frames improve reconstruction quality in a sliding-window bi-directional RNN while adding only 85ms of latency.
- Approximately 90 minutes of real IMU data are used to fine-tune the model and reduce the synthetic-to-real distribution gap.
- 15.85° angular error on TotalCapture is lower than SIP while DIP runs in real time without requiring the full motion sequence.
2 RELATED WORK
Prior pose-reconstruction systems trade off sensor burden, environmental infrastructure, computational cost, or motion generality. DIP instead uses six sparse IMUs and deep learning for real-time full-body reconstruction without calibrated cameras.
- Camera-based motion capture: Camera-based systems often require markers, calibrated multi-camera environments, or offline processing, limiting mobile and outdoor use.
- Camera-based motion capture: DIP avoids calibrated environmental cameras by using sparse orientation and acceleration measurements, enabling infrastructure-free operation where traditional Mocap is impractical.
- Inertial trackers: Commercial inertial trackers use 17 or more IMUs to constrain pose, but dense placement is intrusive, time-consuming, and error-prone.
- Optimization from sparse IMUs: Optimization from six IMUs can recover accurate pose but requires non-convex offline optimization over the entire sequence, potentially taking hours per recording.
- Learning based methods: Earlier learning methods rely on global joint positions, specialized technology, user-specific movements, or pre-defined cyclic motion patterns.
- Learning based methods: DIP is presented as the first deep-learning method to reconstruct full-body motion from sparse sensors in real time.
3 METHOD OVERVIEW
DIP reconstructs articulated human pose from six sparse IMUs by combining a statistical body model with a recurrent neural network trained largely on synthesized measurements. Bidirectional temporal modeling, real-data fine-tuning, and auxiliary acceleration supervision address under-constrained inference while retaining real-time operation.
- Problem and approach: DIP regresses SMPL pose parameters from sparse IMU orientations and accelerations using a deep recurrent neural network.The task is severely under-constrained because multiple poses can correspond to the same IMU inputs.
- Data generation: Synthetic training pairs are generated by placing virtual sensors on SMPL meshes, retrieving orientations through forward kinematics, and computing accelerations via finite differences.The approach uses AMASS and related motion-capture data to overcome the scarcity of paired real IMU and ground-truth pose data.
- Temporal modeling: Bidirectional RNNs use past and future temporal information, improving qualitative and quantitative results over unidirectional RNNs.At runtime, a sliding window supplies short past and future subsequences, enabling real-time prediction with minimal latency.
- Training objective: The auxiliary acceleration loss quantitatively improves pose predictions.This loss is introduced because knee and arm bending can otherwise be problematic and helps preserve information through the network stack.
- Domain adaptation: Fine-tuning on real DIP-IMU data reduces the distribution gap between synthetic and real IMU measurements, especially for underrepresented motion types and usage scenarios.The dataset contains real IMU recordings paired with reference SMPL poses for evaluation and adaptation.
5 EXPERIMENTS
Experiments evaluate DIP against offline baselines and across offline, online, qualitative, and live settings. The results show strong accuracy, benefits from acceleration loss and bidirectional context, improved DIP-IMU performance after fine-tuning, and real-time operation.
- 5.1 Quantitative evaluation: Offline evaluation on TotalCapture shows the best BiRNN configuration outperforms SIP by more than one degree, with an error-distribution peak near 8°.The comparison uses complete input sequences for all models.
- 5.1 Quantitative evaluation: Acceleration reconstruction loss and input dropout improve both RNN and BiRNN models, while BiRNNs produce smoother predictions than unidirectional RNNs.The BiRNN advantage is attributed to access to future steps.
- 5.1.2 Fine-tuning on real data: Fine-tuning on DIP-IMU produces a clear performance increase, bringing results close to TotalCapture while minimally reducing TotalCapture performance.The authors report no catastrophic forgetting after fine-tuning.
- 5.1.3 Online evaluation: In online evaluation, a BiRNN with acceleration loss reaches 15.88° ± 13.57° versus 16.26° ± 13.54° without it using 20 past and 5 future frames on TotalCapture.With 50 past and 5 future frames, the corresponding errors are 15.77° ± 13.41° and 16.10° ± 13.42°.
- 5.1.4 Influence of future window length: Using 20 past and 5 future frames is selected as the best compromise between prediction accuracy and latency for the live system.The final implementation uses a sliding window at runtime despite training on entire sequences.
- 5.2 Qualitative evaluation: Qualitatively, DIP is comparable to SIP outdoors, improves challenging TotalCapture motions, and more faithfully reconstructs DIP-IMU motion with fewer inter-penetrations and smoother outputs.The live implementation streams raw IMU readings and runs at approximately 29 fps while producing faithful results.
6 DISCUSSION AND LIMITATIONS
DIP generalizes across held-out datasets and varied root orientations, but remains vulnerable to acceleration-related ambiguities, especially in extreme poses such as leg raises. Robustness to broader poses, datasets, and settings remains future work.
- Generalization: DIP generalizes to unseen real recordings, Playground motions, and live-demo settings despite differences in motion, sensors, and preprocessing.
- Limitations: Robustness to more poses, datasets, and settings remains future work, with acceleration modeling identified as a main limitation.
- Synthetic versus real data: 5.2°: training a BiRNN on the smaller real DIP-IMU dataset instead of synthetic AMASS data increases TotalCapture error to 21.03°±16.35°.
- Synthetic versus real data: 2.71° and 8.84°: re-synthesis improves performance on TotalCapture and DIP-IMU, respectively, indicating differences between synthetic and real measurements.
- Failure cases: 43.68°±8.53°: the worst 5% of DIP-IMU test poses have high mean joint angle error, with leg raises among the most difficult motions.
- Failure cases: Extreme poses with the body parallel to the floor challenge the model, while both DIP and baselines fail to reconstruct leg raises.
7 CONCLUSION AND FUTURE WORK
DIP estimates pose from six IMUs in real time while avoiding camera line-of-sight requirements, using synthetic-data training and temporal bidirectional inference. It reaches 15.85° angular error on TotalCapture with 85 ms latency, while lacking global translation and facing difficult interaction poses.
- Conclusion: DIP requires only 6 IMUs, runs in real time, and avoids the direct line-of-sight requirements of camera-based systems.
- Conclusion: 15.85° angular error: DIP generalizes from synthetic SMPL-based IMU data to real TotalCapture recordings.
- Conclusion: 85 ms latency: the bidirectional RNN uses the last 20 frames and 5 future frames at test time while producing real-time pose estimates.
- Conclusion: DIP performs comparably to or better than the competing offline SIP approach and produces smooth estimates generally without inter-penetrations.
- Future work: Future work includes multi-person and object interactions, visual input for difficult contact poses, latency reduction, and global translation estimation.
A ADDITIONAL ARCHITECTURES
Additional experiments compare WaveNet and a feed-forward network with the proposed BiRNN on synthetic AMASS training data. The BiRNN offers better performance or flexibility than these alternatives.
- Architectures: The FFN uses 20 past and 5 future frames at each time step, while Table 4 reports mean joint angle error on TotalCapture and DIP-IMU.
- Architectures: 4.4°: the FFN performs worse on TotalCapture than the best BiRNN using the same 20-past, 5-future-frame window.
- Architectures: WaveNet improves joint angle error and visual quality relative to the FFN but retains visible trembling artifacts.
- Architectures: The BiRNN window length can change on the fly without retraining or increasing trainable parameters, unlike the FFN.
B ADDITIONAL FIGURES
The appendix provides architecture details for the BiRNN and identifies the three highest-error poses from the DIP-IMU test set.
- Additional figures: Figure 12 gives the BiRNN architecture details, while Figure 13 presents the three poses with the highest mean joint angle error in DIP-IMU.
C NORMALIZATION
The method normalizes sensor measurements relative to the root and rescales inputs and outputs using training-set statistics. Alternative sequence-level and heading-only schemes were evaluated but not adopted.
- C NORMALIZATION: Root-relative normalization is applied to sensor orientations and accelerations before model processing.The paper introduces root-based normalization and evaluates additional schemes that were not beneficial.
- C NORMALIZATION: The architecture feeds normalized accelerations and orientations through a dense layer into two bidirectional recurrent layers.The final recurrent output predicts mean and standard-deviation vectors for SMPL parameters and accelerations.
- C NORMALIZATION: 40.18° mean angular error resulted from per-frame heading removal, 24.41° worse than the adopted per-frame normalization on TotalCapture.This scheme replaces the full root orientation with a yaw-based rotation before normalization.
- C NORMALIZATION: The normalized inputs are transformed as (x_t − µ)/σ using statistics computed over the entire training dataset, and outputs receive the same treatment.The statistics µ and σ are 60-dimensional vectors.
D INFLUENCE OF ACCELERATION
Acceleration inputs improve pose estimates: removing them increases error on both evaluated datasets, including a reported TotalCapture error of 19.45° after fine-tuning.
- D INFLUENCE OF ACCELERATION: 19.45° TotalCapture error was observed after fine-tuning a BiRNN without acceleration inputs.The no-acceleration configuration also had higher mean and standard-deviation errors on both TotalCapture and DIP-IMU.
- D INFLUENCE OF ACCELERATION: The three highest-error test poses commonly show ambiguity or poor reconstruction in the upper arms and lower extremities.The lower-arm orientation can remain reasonable while the upper-arm orientation is incorrectly disambiguated.
E HARDWARE SPECIFICATIONS
The live system measures end-to-end model inference across local and remote visualization settings, while the sensor update rate imposes a 60 Hz runtime cap.
- E HARDWARE SPECIFICATIONS: Table 7 reports FPS for grabbing IMU measurements, network transfer, and retrieving model predictions for each time step.Local and remote-laptop settings distinguish whether visualization runs on the training machine or a commodity laptop.
- E HARDWARE SPECIFICATIONS: DIP-IMU data collection uses a protocol designed to include activities underrepresented in existing motion-capture datasets.The supplied table caption identifies Table 6 as the dataset capture protocol.
- E HARDWARE SPECIFICATIONS: 60 Hz is the technical runtime cap imposed by the Xsens sensor update rate.The RNN inference runtime is therefore evaluated within the sensor streaming rate.
F DATA COLLECTION
DIP-IMU was collected using a protocol emphasizing activities underrepresented in existing motion-capture datasets. The live-demo measurements and predictions were also benchmarked through an FPS protocol.
- F DATA COLLECTION: The DIP-IMU collection protocol paid particular attention to activities underrepresented in existing motion-capture datasets.Table 6 summarizes the protocol used to collect the new real dataset.
- F DATA COLLECTION: FPS measurements include capturing current IMU measurements, network transmission, and retrieving predictions for the current time step.The table distinguishes local visualization from visualization on a commodity laptop.