Source-linked AI summary
IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds
Vimal Mollyn, Riku Arakawa, Mayank Goel, Chris Harrison, Karan Ahuja
TL;DR
Consumers generally lack practical ways to track full-body pose without special suits or sensor arrays, despite potential uses in fitness, rehabilitation, and other activities. IMUPoser estimates pose from whatever subset of consumer-device IMUs is available, and reports natural, temporally coherent estimates even with one device. Its output remains approximate, and the current implementation does not predict global root translation.
Problem
The paper asks whether body pose can be estimated from consumer devices people already own, despite sparse, changing sensor locations and noisier commodity IMUs.
Source: §1 · p. 2Method
IMUPoser uses a recurrent model to estimate full-body pose from available consumer-device IMU inputs, masking absent locations and allowing as little as one device.
Source: §1 · p. 2, §4.1 · p. 4Results
Estimates were natural and temporally coherent with as little as one device; across both datasets, one-device MPJVE was 16.27 cm (SD=9.93 cm).
Source: §6.1 · p. 8, §11 · p. 10Takeaways & Limitations
The authors conclude that consumer-device IMUs can support whole-body applications without additional user instrumentation.
Source: §11 · p. 10The current implementation does not predict global root translation.
Source: §10 · p. 10
Abstract
from arXiv · showhide
Tracking body pose on-the-go could have powerful uses in fitness, mobile gaming, context-aware virtual assistants, and rehabilitation. However, users are unlikely to buy and wear special suits or sensor arrays to achieve this end. Instead, in this work, we explore the feasibility of estimating body pose using IMUs already in devices that many users own -- namely smartphones, smartwatches, and earbuds. This approach has several challenges, including noisy data from low-cost commodity IMUs, and the fact that the number of instrumentation points on a users body is both sparse and in flux. Our pipeline receives whatever subset of IMU data is available, potentially from just a single device, and produces a best-guess pose. To evaluate our model, we created the IMUPoser Dataset, collected from 10 participants wearing or holding off-the-shelf consumer devices and across a variety of activity contexts. We provide a comprehensive evaluation of our system, benchmarking it on both our own and existing IMU datasets.
1 INTRODUCTION
IMUPoser explores full-body pose tracking from consumer devices people already carry, addressing changing, sparse sensor locations and noisier commodity IMUs. It pairs device IMUs with Vicon tracking for evaluation, while targeting approximate pose for longitudinal health and wellness uses rather than fidelity-critical applications.
1 INTRODUCTION
- Pose tracking could support gaming, fitness, rehabilitation, life logging, and context-aware interfaces, but consumers generally lack tools and are unlikely to wear special-purpose suits or add home sensors.The paper proposes using devices users already own as a way to generate useful pose information.Source: §1 · p. 2
- The system must handle changing device locations and numbers, incomplete inputs down to one device, and noisier consumer IMUs than professional-grade suits.A phone may move from a pocket to a hand, while earbuds add another sensed point.Source: §1 · p. 2
- The evaluation dataset pairs professional-grade Vicon optical tracking with commodity IMU data from common worn and held locations.Source: §1 · p. 2
- With at most three sensed positions estimating hundreds of degrees of freedom, output is approximate and ill-suited to high-fidelity effects or VR avatars, but suited to passive longitudinal health and wellness sensing.The paper names calorie counting, physical-therapy progress, and exercise form and rep-count monitoring as potential uses.Source: §1 · p. 2
2 RELATED WORK
Related work spans external capture, non-IMU body-worn sensing, and full-body estimation from worn IMUs. The paper positions IMUPoser against systems whose hardware, mobility, sensor density, calibration, or sensor quality differ from consumer devices.
2 RELATED WORK
- The review covers external and worn capture and IMU-based pose sensing, and points readers to surveys for a broader pose-estimation review.Source: §2 · p. 2
2.1 Body Capture using External Sensors
- External systems range from accurate but costly Vicon or OptiTrack setups to depth and RGB cameras, VR tracking hardware, and other sensing approaches.Depth cameras can balance cost, availability, and accuracy but generally require immobile setups; specialized external systems also carry setup requirements.Source: §2.1 · pp. 2–3
2.2 Body Capture with non-IMU Worn Sensors
- Non-IMU worn-sensing approaches include exoskeletons, ultrasonic beacons, pressure sensors, RFID, and body-worn cameras, often requiring specialized hardware users do not own.Pose-On-The-Go uses smartphone sensors, but much of its full-body pose is guessed with inverse kinematics.Source: §2.2 · p. 3
2.3 Body Pose Estimation using Worn IMUs
- Prior full-body worn-IMU systems commonly use homogeneous, high-quality sensors at known calibrated positions, unlike consumer devices whose IMUs vary in quality, noise, and framerate.Prior methods include approaches using four IMUs or between six and seventeen body-worn IMUs.Source: §2.3 · p. 3
3 POSSIBLE DEVICE COMBINATIONS
The paper enumerates possible phone, watch, and earbud locations, then reduces device-position arrangements to body-location combinations relevant to the system. Its counts rely on simplifying assumptions, including continuous earbud-case IMU streaming as a future possibility.
3 POSSIBLE DEVICE COMBINATIONS
- Phones have six location or absence states, watches three, and earbuds four; the paper assumes future firmware could allow continuous IMU streaming from an earbud charging case.Source: §3 · pp. 3–4
- Enumerating device states, excluding three atypical phone-to-head and earbud combinations and the no-device case, leaves 68 arrangements: 14 with one active device, 36 with two, and 18 with three.Source: §3 · p. 4
- The system groups the 68 arrangements into 24 body-location combinations, treating highly correlated signals from nearby devices as one body point and making further hand and head location simplifications.A smartphone held to either ear counts as a head location because the AirPods implementation fuses their IMUs into one head 6DOF estimate.Source: §3 · p. 4, Figure 3 · p. 4
4 IMPLEMENTATION
The implementation streams consumer-device IMUs to a device for inference, uses a masked-input recurrent model, synthesizes training examples from motion-capture data, and refines predictions using instrumented-joint orientations.
4.1 Model
- A two-layer bidirectional LSTM accepts available IMU orientations and accelerations with absent locations masked, and outputs 144 SMPL pose parameters for 24 joints.A forward-kinematics module computes joint positions during training; the model has 10.7M trainable parameters.Source: §4.1 · p. 4
4 IMPLEMENTATION
- The real-time architecture uses smartphones, smartwatches, and earbuds with IMUs, wireless transmission, and local compute; less-capable devices stream data to the most capable carried device.Source: Figure 4 · p. 4, §4 · p. 4
4.2 IMU Dataset Synthesis
- Training data is synthesized from AMASS motion capture by attaching virtual IMUs to five SMPL locations, generating acceleration and orientation, resampling to 25 FPS, and masking absent devices.The resulting combined dataset contains 134.8M synthetic IMU instances paired with ground-truth SMPL poses across 24 device-location combinations.Source: §4.2 · pp. 4–5
4.3 Training
- The model is trained end-to-end for 80 epochs on five-second, 125-sample windows, using the sum of pose and joint-position mean squared error losses.Training uses batch size 256 and Adam with a learning rate of 3e-4.Source: §4.3 · p. 5
4.4 Joint Rotation Refinement
- A final inverse-kinematic refinement optimizes selected bone orientations around instrumented wrist, head, and hip joints using absolute IMU orientations.The optimization uses MSE and Adam for 10 iterations per frame, which the authors found did not impede real-time performance.Source: §4.4 · p. 5
5 EVALUATION
The evaluation combines DIP-IMU and a newly collected consumer-device dataset with real-time-style testing across device configurations. The protocol covers varied activities, calibrates device orientations, and measures pose accuracy and motion smoothness.
5.1 DIP-IMU Dataset
- DIP-IMU provides approximately 90 minutes of 60 Hz data from 10 participants across five activity classes and enables comparison with prior work.The classes include upper-body, lower-body, interaction, freestyle, and locomotion activities.Source: §5.1 · p. 5
5.2.1 Data Collection Apparatus.
- The apparatus used two pocketed iPhones, two Apple Watches, and AirPods at 25 Hz, with Vicon capture used to construct ground-truth SMPL poses.Vicon data were captured at 120 FPS, synchronized to the IMU streams, and fitted with Mosh++.Source: Figure 5 · p. 5, §5.2.1 · p. 6
5.2 IMUPoser Dataset
- IMUPoser complements DIP-IMU with consumer-device IMU data paired with Vicon-derived ground truth, collected from 10 participants in naturally chosen device placements.Participants averaged roughly seven minutes of continuous activity each; clothing, pocket styles, and smartwatch placement were not controlled.Source: §5.2 · p. 6, §5.2.3 · pp. 6–7
5.2.2 Device Calibration.
- Devices were aligned using a three-second orientation window, then a three-second T-pose established offsets between each device and its attached bone joint.The T-pose calibration accommodates different device orientations, such as a phone held in the hand versus a watch worn on the wrist.Source: §5.2.2 · p. 6
5.2.3 Data Collection Procedure.
- The obstacle-course procedure covered upper- and lower-body motions, locomotion, freestyle, head motions, interaction, and miscellaneous activities, with continuous capture through transitions.The listed activities included scrolling on a phone, walking, jogging, and limb exercises.Source: §5.2.3 · pp. 6–7
5.3 Evaluation Protocol
- Evaluation fine-tuned the AMASS-trained model on the first eight DIP-IMU participants and tested on the last two using a rolling 125-sample, five-second window with one-sample overlap.The protocol used smoothing and assessed MPJRE, MPJPE, MPJVE, and Jitter; MPJVE was the primary metric for most tasks.Source: §5.3 · p. 7
- Jitter measures predicted-motion jerk, with lower values indicating smoother, more natural motion.Source: §5.3 · p. 7
6 RESULTS
Across the evaluated settings, pose error generally decreases with more devices, while accuracy varies by body region and motion. On DIP-IMU, IMUPoser remains less accurate than six-sensor systems but is close to some prior methods using more sensors.
6.1 Accuracy Across Device-Location Combinations
- Walking works well across device combinations, while uncorrelated limb motions fail unless those limbs are instrumented.The sample SMPL meshes encode per-vertex error in red, ranging from 0 to 1 m.Source: Figure 7 · p. 7, §6.1 · p. 8
- 16.27 cm (SD=9.93 cm) with one device fell to 13.9 cm (SD=8.36 cm) with two and 11.1 cm (SD=6.51 cm) with three (MPJVE).These averages span both datasets and all possible combinations for each device count.Source: §6.1 · pp. 7–8
6.2 Accuracy Across Body Regions
- With a right-hand IMU, right-arm MPJVE is 14.65 cm versus 21.6 cm for the left arm; right- and left-hand MPJVE are 17.2 cm and 26.9 cm, respectively.These figures average across both datasets; error is highest when a body region has no instrumented limb.Source: §6.2 · p. 8
- With one arm IMU, both-arm MPJVE is 18 cm; with both arm IMUs, it is 14.5 cm.The corresponding right- and left-hand errors are 22.17 cm and 20.4 cm with one IMU, versus 17.35 cm and 16.9 cm with both.Source: §6.2 · p. 8
- Left-leg MPJVE is 10.3 cm with a left-pocket IMU and 10.05 cm with both pocket IMUs; right-leg MPJVE is 10.4 cm and 9.75 cm, respectively.The authors report only a modest gain from two leg IMUs because the legs tend to move in tandem.Source: §6.2 · p. 8
- End-effectors average 20.28 cm MPJVE on IMUPoser and 17.29 cm on DIP-IMU, versus 12.92 cm and 11.23 cm for non-end-effector joints.The paper notes that error accumulates along the kinematic chain.Source: §6.2 · p. 8
6.3 Comparison to Prior Work
- On DIP-IMU, IMUPoser’s MPJVE is 3.2 cm worse than DIP [21] and 5.0 cm worse than TransPose [65], while Jitter is 1.9 vs. 1.4 for TransPose.The comparison evaluates the systems on the same DIP-IMU dataset; systems using six sensors are more accurate than IMUPoser’s one-to-three-sensor setup.Source: §6.3 · p. 8
7 ACTIVE DEVICE TRACKING
IMUPoser tracks which consumer devices are present and where they are located, then supplies available IMU data to its pose model. Its preliminary evaluation found perfect earbud and smartwatch tracking and 90.8% instance-wise smartphone tracking accuracy.
7 ACTIVE DEVICE TRACKING
- The tracker runs alongside the pose model to identify available devices and their body locations before pose estimation.Users first specify their usual phone pocket, phone-holding hand, and watch arm; automated heuristics then refine device locations.Source: §7 · p. 8, §7.1 · p. 9
7.1 Implementation
- Phone-state heuristics combine screen, motion, proximity, and—when a watch is available—UWB distance; otherwise, setup preferences supply the location.A moving phone with its screen on is treated as hand-held; a screen-off phone with triggered proximity sensing is treated as pocketed.Source: §7.1 · p. 9
- Moving, connected watches are assigned to the user-specified wrist, while connected AirPods are treated as worn in the ear.AirPods in their charging case sleep and stop transmitting IMU data.Source: §7.1 · p. 9
7.2 Evaluation
- The evaluation tested 15 device combinations in random order, with participants walking, sitting, standing, and returning to their starting position during each trial.The Figure 9 combinations specify which devices were present; an experimenter marked trial boundaries and ground-truth labels.Source: §7.2 · p. 9, Figure 9 · p. 9
- Across all participants and data instances, earbud and smartwatch tracking accuracy was 100%, while smartphone instance-wise accuracy was 90.8% across five possible states: not present, left or right pocket, and left or right hand.Source: §7.2 · p. 9
8 REAL-TIME IMPLEMENTATION
The real-time proof of concept combines active device tracking with pose estimation using consumer-device IMUs. It streams data to a MacBook Air, which produces a visualized SMPL mesh at a rate capped by the slowest IMU.
8 REAL-TIME IMPLEMENTATION
- Active device tracking identifies available devices and predicts their body locations, then passes those inferences and IMU data to the pose model.The two processes work together in the real-time implementation.Source: §8 · p. 9
8.1 Proof-of-Concept Device Ecosystem
- The proof of concept uses an iPhone 11 Pro, Apple Watch Series 6, and AirPods Pro, whose devices exchange 6DOF IMU data at different rates.AirPods report at roughly 25 FPS and fuse their two IMUs into one 6DOF head estimate.Source: §8.1 · p. 9
8.2 Output
- The live tracking display illustrates which devices are active while the pose output is visualized.The foreground laptop shows the live tracking result, with active devices highlighted by white circles.Source: Figure 10 · p. 10
- 26.8 ms mean inference time on a MacBook Air, with output capped at 25 FPS by the slowest IMU.The system visualizes a SMPL mesh and requires the same calibration used during data collection; running the model on a phone would require additional engineering effort.Source: §8.2 · p. 10
10 LIMITATIONS AND FUTURE WORK
The system estimates unsensed limbs only approximately and may regress to the mean pose when their motion is independent of instrumented joints. Its current implementation also omits global root translation and has a proof-of-concept device tracker that depends on a homogeneous ecosystem.
10 LIMITATIONS AND FUTURE WORK
- Uninstrumented limb estimates are approximate and tend toward the mean pose when their motion is completely independent of instrumented joints.The authors describe the resulting pose as low fidelity for applications such as special-effects motion capture or VR avatars.Source: §10 · p. 10
- Future extensions could add placements such as a back or coat pocket and devices such as smart shoes, eyewear, or rings to expand supported poses and activities.The authors also envision execution on whichever capable device the user carries, potentially including a watch and wireless headphones without a phone.Source: §10 · p. 10
- The current system does not predict global root translation; visual odometry or contextual cues are proposed as possible future inputs for improving translation or accuracy.The cited examples include smartphone visual odometry, activity, and user location.Source: §10 · p. 10
- Active device tracking remains a proof-of-concept needing refinement for consumer deployment, and the devices must currently share a homogeneous ecosystem such as Apple’s.The authors suggest an industry-wide device-networking standard could mitigate the ecosystem constraint.Source: §10 · p. 10
11 CONCLUSION
IMUPoser estimates real-time full-body pose from streaming IMUs in consumer phones, watches, and earbuds while tracking their availability and body locations. The evaluations report natural, temporally coherent estimates with as little as one device despite noisy consumer IMU signals.
11 CONCLUSION
- IMUPoser combines automatic device and location tracking with streaming consumer-device IMUs to estimate natural, temporally coherent pose from as little as one device.The conclusion frames this as a route to whole-body applications without additional user instrumentation.Source: §11 · p. 10