PaperlayerView exact revision on arXiv

Source-linked AI summary

Transformer Inertial Poser: Real-time Human Motion Reconstruction from Sparse IMUs with Simultaneous Terrain Generation

Yifeng Jiang, Yuting Ye, Deepak Gopinath, Jungdam Won, Alexander W. Winkler, C. Karen Liu

arXiv:2203.15720v3cs.CVcs.GR

TL;DR

Sparse-IMU reconstruction must contend with temporal inconsistency and motion drift while estimating full-body motion from six sensors. TIP combines a history-aware Transformer with learned stationary body points and terrain generation to estimate and regularize motion in real time. It reports improvement over recent state-of-the-art methods on flat terrains.

  • Problem

    Real-time motion reconstruction from sparse IMUs faces temporal inconsistency and global translational drift, while portable, less costly capture is sought for use beyond studios.

    Source: §1 · pp. 1–2

  • Method

    TIP uses a history-conditioned Transformer to predict motion and stationary body points, with analytical drift correction and simultaneous terrain generation that regularizes global motion.

    Source: §1 · p. 2

  • Results

    TIP shows clear improvement over recent state-of-the-art methods on flat terrains and includes qualitative demonstrations across varied motion types.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The combined approach presents a self-contained, economic six-IMU setup for motion reconstruction with simultaneous plausible terrain generation.

    Source: §5 · p. 7

    Terrain can show penetration during generation because newly active SBPs are ignored briefly, though the final terrain should show little penetration during playback.

    Source: §4.2 · p. 7

Abstract

from arXiv · show

Real-time human motion reconstruction from a sparse set of (e.g. six) wearable IMUs provides a non-intrusive and economic approach to motion capture. Without the ability to acquire position information directly from IMUs, recent works took data-driven approaches that utilize large human motion datasets to tackle this under-determined problem. Still, challenges remain such as temporal consistency, drifting of global and joint motions, and diverse coverage of motion types on various terrains. We propose a novel method to simultaneously estimate full-body motion and generate plausible visited terrain from only six IMU sensors in real-time. Our method incorporates 1. a conditional Transformer decoder model giving consistent predictions by explicitly reasoning prediction history, 2. a simple yet general learning target named "stationary body points" (SBPs) which can be stably predicted by the Transformer model and utilized by analytical routines to correct joint and global drifting, and 3. an algorithm to generate regularized terrain height maps from noisy SBP predictions which can in turn correct noisy global motion estimation. We evaluate our framework extensively on synthesized and real IMU data, and with real-time live demos, and show superior performance over strong baseline methods.

1 INTRODUCTION

The paper targets real-time, portable motion capture from six IMUs, addressing prediction inconsistency and drift while generating terrain alongside motion. It introduces a history-aware Transformer, stationary body points, and terrain regularization, and reports improvements over recent methods on flat terrain.

Motivation

  • Six-IMU motion capture offers a self-contained, untethered option for indoor and outdoor use, including settings beyond camera field of view.The introduction motivates portable, less costly, minimally intrusive real-time capture for use beyond research labs and professional studios.Source: §1 · pp. 1–2

Challenges

  • The method addresses temporal inconsistency and global translational drift in real-time reconstruction from six IMUs.Source: §1 · p. 2

Transformer reconstruction

  • A conditioned Transformer decoder explicitly uses prediction history and yields more accurate joint-angle estimates than RNN-based methods, especially for sitting and standing.Source: §1 · p. 2

Drift correction

  • Stationary Body Points (SBPs) are predicted alongside motion and used by analytical routines to counteract gradual drift, including under noisy sensors or unseen motions.Source: §1 · p. 2

Terrain generation

  • Predicted SBPs support plausible terrain height maps that are estimated alongside motion and used to regularize global-motion reconstruction in real time.Source: §1 · p. 2

Evaluation

  • The system improves over recent state-of-the-art methods on flat terrains and is qualitatively demonstrated on varied motion types, including motions unseen in previous works.Source: §1 · p. 2

2 RELATED WORK

Prior work spans multimodal motion estimation, sparse-IMU reconstruction, headset-conditioned motion synthesis, and Transformer-based motion generation. TIP is positioned as six-IMU reconstruction that also addresses global motion and noisy real-IMU input.

Multimodal motion estimation

  • Vision and IMU systems combine sensors through offline or online optimization and learned networks, but retain vision limits such as narrow capture regions, occlusion, and light sensitivity.Source: §2 · p. 2

Sparse-IMU tracking

  • Sparse-IMU work includes six-sensor offline SIP and learned DIP, while commercial systems requiring at least 17 IMUs hinder everyday accessibility.Source: §2 · p. 2

Global motion reconstruction

  • TransPose estimates global root motion with supporting-foot heuristics and a learned network, while concurrent PIP further optimizes motion to reduce physics violations.Source: §2 · p. 3

Headset-conditioned synthesis

  • Headset-conditioned systems synthesize full-body motion with sparse upper-body sensors; LoBSTR uses four sensors, while this work uses six including lower-body information for reconstruction.Source: §2 · p. 3

Transformer motion models

  • Transformers have been used for motion generation and reconstruction, including a model found more effective than recurrent networks, while this work additionally handles real-IMU noise.Source: §2 · p. 3

3 TRANSFORMER INERTIAL POSER (TIP)

TIP combines a conditional Transformer motion estimator with analytical SBP-based drift correction and a terrain module that generates height maps while correcting motion. Its components use prediction history, near-stationary body points, and terrain assumptions to address temporal consistency and motion drift.

3 TRANSFORMER INERTIAL POSER (TIP)

  • TIP estimates full-body joint angles and root velocity from six IMUs, then uses predicted SBPs for analytical drift correction and real-time terrain generation.The terrain height map is an additional output and also helps refine motion reconstruction.Source: §3 · p. 3, Figure 2 · p. 3

3.1 Transformer Motion Estimator

  • A conditional Transformer generates motion using both IMU sequences and its prior predictions, addressing ambiguity between motions with similar sensor readings.The model predicts sequences in parallel during training and uses autoregressively updated history at test time.Source: Figure 3 · p. 3, §3.1 · p. 4

3.2 Stabilizing Drift with SBPs

  • Learned near-zero-velocity SBPs provide analytical constraints against global and joint drift, with soft-IK feeding corrected joint angles into prediction history.Root correction is limited to the horizontal plane; terrain estimation handles height correction. Heel-to-toe contact can have a stationary patch despite nonzero foot-link velocity.Source: §3.2 · pp. 4–5, Figure 4 · p. 4

3.3 Plausible Terrain Generation

  • The terrain module uses pelvis and foot SBPs plus estimated vertical root velocity to produce a height map and root correction.Nearby SBPs within 1 m and 0.1 m in height are clustered under a no-gradual-slope assumption; unvisited cells take the height of their nearest SBP-visited neighbor.Source: §3.3 · p. 5

4 EVALUATIONS

TIP improves accuracy across almost all reported evaluations and demonstrates terrain reconstruction and drift correction through simulation, ablations, and live demonstrations.

4.1 Quantitative Evaluation on Flat Terrain

  • Evaluation covers real held-out DIP and TotalCapture data and held-out synthetic DanceDB for cross-motion-type generalization.The datasets contain ground-truth full-body motions; TotalCapture's ground truth and synthesized readings also appear in AMASS training, following prior practice.Source: §4.1 · pp. 5–6

4.2 Quality of Simultaneously Generated Terrains

  • Simulation and live demos show plausible terrain reconstructions consistent with human motion, including staircases, without real non-flat-terrain training data or ground-truth terrain supervision.Body parts may briefly penetrate terrain during generation; the final terrain should show little penetration during playback.Source: §4 · p. 5, §4.2 · pp. 6–7, Figure 5 · p. 7

4.1 Quantitative Evaluation on Flat Terrain

  • The system achieves better accuracy in almost all evaluations against TransPose and DIP.The largest reported improvements are in root translation on TotalCapture and DanceDB.Source: Table 1 · p. 6, §4.1 · p. 6

4.3 Ablation Studies on Root Drifting

  • No-SBP and unused-SBP variants perform worst on TotalCapture, while history-free SBP predictions become intermittent and less useful at runtime.Without terrain-based root correction, biased sensor error can cause large vertical drift that also harms generated terrain.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 Qualitative Comparisons of Run-time Joint IK Correction

  • SBP-based IK correction stabilizes a two-minute sitting sequence by keeping joint-prediction history consistent with predicted SBPs.This is shown with the authors' sensors and public real-IMU DIP sequences.Source: §4.4 · p. 7

4.5 Live Demo

  • Live demos with six Xsens sensors cover locomotion, manipulation, jumping, dancing, and other motions, with slight latency from preprocessing and degraded performance on one female subject.The demos tested one male and one female subject.Source: §4.5 · p. 7

5 CONCLUSION

The conclusion presents TIP as a six-IMU method for reconstructing motion while generating plausible terrain, and identifies broader real-IMU terrain data and personalization as future improvements.

5 CONCLUSION

  • TIP combines a conditional Transformer, learned stationary body information, and terrain generation to reconstruct motion from six wearable IMUs.The terrain module also regularizes terrain and corrects noisy global motion estimates.Source: §5 · p. 7
  • The authors identify collecting real IMU motion data on varied terrains as a major direction for improving the method.Source: §5 · p. 7
  • Personalized finetuning or calibration may improve reconstruction for individual users.Source: §5 · p. 7

A ACCELERATION READINGS AND SIM-TO-REAL

The appendix addresses synthetic-to-real acceleration mismatch with shared filtering and adds integrated acceleration history to improve stair recognition on real hardware.

Moving Average Filtering

  • Synthetic acceleration noise from double differentiation differs from noisy real acceleration, motivating preprocessing to narrow the mismatch.The authors note that real orientation data are usually less noisy because of in-sensor Kalman filtering.Source: A ACCELERATION READINGS AND SIM-TO-REAL · p. 10
  • An 11-step moving average brings synthetic and real acceleration sufficiently close for one-stage training on their combined data.This replaces two-stage training and avoids the risk of catastrophic forgetting during finetuning.Source: A ACCELERATION READINGS AND SIM-TO-REAL · p. 10, Figure 6 · p. 10

Summing Up (Integrate) Past Accelerations

  • Adding recent acceleration sums as features improves stair recognition on real hardware.The added history covers a small recent window such as 0.5 seconds and increases acceleration features from R18 to R36.Source: A ACCELERATION READINGS AND SIM-TO-REAL · p. 10

B MODEL DETAILS

The model is trained on synthesized AMASS data and DIP data, with standard losses for its outputs. It predicts a trajectory window during training, uses exponential moving average post-processing, and runs in real time once trained.

B MODEL DETAILS

  • Training data combine synthetic motions from AMASS with data from 8 of 10 DIP subjects; DIP pseudo-ground-truth SBPs are labeled using a pretrained model.The DIP real IMU data lack root motion; pyBullet supports forward kinematics, SBP labeling, root correction, and visualization.Source: B MODEL DETAILS · p. 10
  • The model uses mean-squared error for joint rotations, root velocity, and Cartesian SBP outputs, and binary cross-entropy for binary SBP outputs.Joint rotations use the first two columns of each rotation matrix as 6D representations.Source: B MODEL DETAILS · p. 10
  • Exponential moving average post-processing replaces jerk loss, which produced no visible improvement, at the cost of slightly higher joint accuracy errors.The authors suggest the jerk loss may not help because test-time prediction uses only the last output at each step.Source: B MODEL DETAILS · p. 10
  • The model runs at 60 fps on a 2080Ti, has 3,677,315 parameters, and requires an initial full-body pose after sensor calibration.The stated bottleneck is the pyBullet Python wrapper; the initial pose is available in practice after T-pose calibration.Source: B MODEL DETAILS · p. 10

C TERRAIN GENERATION DETAILS

Terrain generation initializes a height map using a lower-bound assumption, then limits how much each newly active SBP can influence the map. SBP timing and pelvis-height checks help avoid premature or misplaced terrain updates.

C TERRAIN GENERATION DETAILS

  • The initial height map is zeroed relative to the initial root height and a lower bound w, which assumes no cluster means fall below that bound.A tighter w, such as the starting ground plane, can avoid dents below that plane; demos use a tight bound.Source: C TERRAIN GENERATION DETAILS · p. 11
  • Each new SBP is limited to a 1m × 1m influence area because two different-height SBPs alone could imply an infinitely wide stair.The region could exceed a real stair's width, but later SBPs can crop the terrain as more information arrives.Source: C TERRAIN GENERATION DETAILS · p. 11
  • Newly active SBPs are ignored for 50 frames or until inactive, and pelvis SBPs contribute only when more than 0.2m from the feet.These rules allow SBP height to settle and avoid generating terrain at pelvis height while standing.Source: C TERRAIN GENERATION DETAILS · p. 11

D SENSOR CALIBRATION

Calibration maps sensor readings into a user-specified global frame and estimates the transforms needed to express orientations and accelerations in that frame. It uses stationary sensor alignment and a worn T-pose, with the sensor assumed to remain fixed relative to its bone during use.

D SENSOR CALIBRATION

  • The system expects bone orientation and acceleration in the specified global frame, with constant acceleration bias—usually gravity—accounted for.Calibration obtains the transform between raw sensor coordinates and the global frame.Source: D SENSOR CALIBRATION · p. 11
  • The first calibration step aligns all sensors to the specified global frame and averages readings while they remain still on the ground for three seconds.The user specifies global x as front, y as left, and z as up; this axis convention differs from DIP and TransPose.Source: D SENSOR CALIBRATION · p. 11
  • For the second step, the user wears the sensors in a T-pose, faces the specified front, and holds the pose for three seconds while readings are averaged.The procedure assumes each sensor stays static relative to its bone throughout the system run.Source: D SENSOR CALIBRATION · p. 11

E ADDITIONAL ANALYSIS

Additional analyses examine performance on longer motion segments, acceleration preprocessing, and representative reconstructions. The reported results show little degradation on longer sequences, benefits from preprocessing on real-IMU data, and qualitative differences in sitting and terrain handling.

E ADDITIONAL ANALYSIS

  • Performance degradation was minimal on random 3000-frame (50s) windows, and the statistics trends between TIP and TransPose remained unchanged.The comparison uses random 50-second windows rather than 600-frame (10-second) windows; shorter test motions may unevenly bias the statistics.Source: E ADDITIONAL ANALYSIS · pp. 11–12, Table 3 · p. 11
  • Acceleration preprocessing visibly improved results on real-IMU DIPEval and TotalCapture data, while it was unimportant for synthesized DanceDB data.The ablation removes average filtering and summation during both training and testing.Source: E ADDITIONAL ANALYSIS · p. 12, Table 4 · p. 12
  • TIP produced a more stable sitting posture than TransPose and handled terrain without assuming flat ground, as illustrated for chair sitting and step climbing.Figure 7 shows real-IMU chair sitting and synthesized-IMU step climbing, with predicted SBPs and reconstructed terrains.Source: E ADDITIONAL ANALYSIS · p. 12, Figure 7 · p. 12

F DISCUSSIONS

The discussion identifies limits in terrain reconstruction, motion generalization, and post-processing. It also notes that terrain ambiguity and body-type bias constrain performance in particular settings.

F DISCUSSIONS

  • TIP tends to underestimate terrain height; the authors suggest more annotated terrain data and uneven-terrain upsampling.Source: F DISCUSSIONS · p. 12
  • Terrain-height estimates depend solely on motion prediction and remain susceptible to sensor noise; slightly bumpy and flat ground are near-ambiguous at IMU noise levels.Source: F DISCUSSIONS · p. 12
  • Real-hardware reconstruction degrades on motion types rare in training, affecting the quality of generated terrain.Source: F DISCUSSIONS · p. 12
  • The authors do not claim improved motion jitter, and say the nonideal smoothing filter hurts motion accuracy by increasing latency.Source: F DISCUSSIONS · p. 12
  • The model generalizes better to taller than shorter users; synthesized characters ranged from 1.6m to 1.8m, and per-user fine-tuning may be needed.Source: F DISCUSSIONS · p. 12
  • IMU-only terrain estimates cannot distinguish a foot resting on terrain from one stationary in air; environment-distribution priors could guide more likely terrain.Source: F DISCUSSIONS · p. 12

Related papers

Loading 2203.15720v3…