PaperlayerView exact revision on arXiv

Source-linked AI summary

One Sensor, Whole Body - 3D Body Pose from a Single Consumer Earbud IMU

Zhilin Guo, Boqiao Zhang, Oszkár Urbán, Josef Bengtson, Hakan Aktas, Wenzhao Li, Siyu Hong, Kyle Fogarty, Chenliang Zhou, Ali Senguel, Cengiz Oztireli

arXiv:2609.34978v1cs.CVcs.HC

TL;DR

The paper asks how much 3D body pose one consumer earbud IMU can recover and whether adding consumer sensors helps. It builds a real-data benchmark and finds the head stream effective for lower-body pose and contact, while added foot sensors do not improve pose.

  • Problem

    How much 3D body pose can one consumer head IMU recover, and does adding consumer sensors help?

    Source: §1 · p. 1

  • Method

    The authors capture four-view RGB-D video and head and foot IMUs across 35 takes, align streams post-hoc, and evaluate adapted recurrent models using SAM 3D Body pseudo-ground-truth.

    Source: §1 · pp. 1–2

  • Results

    79.0 mm rigid-MPJPE; adding two foot IMUs never significantly improves pose and significantly degrades it in two of four model–split combinations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    For the consumer devices tested, sensor reliability rather than sensor count is the design constraint, and the head stream is the sweet spot.

    Source: §8 · p. 4

    The pilot covers one subject and one capture day and uses SAM 3D Body pseudo-ground-truth rather than marker-based motion capture, so it establishes within-participant feasibility rather than population-level generalization.

    Source: §7 · p. 4

Abstract

from arXiv · show

Consumer earbuds already stream inertial motion data from the head, one of the most widely worn sensor locations on the body. We ask how much of the 3D body pose a single such head IMU can recover, and whether adding more consumer sensors actually helps. We build a multimodal capture pipeline that records four-view RGB-D video together with an AirPods head IMU and two Striv insole IMUs, synchronize the streams post-hoc, and generate pseudo-ground-truth with SAM 3D Body, yielding a 35-take single-subject benchmark spanning gait, turning, vertical, everyday, and clinically inspired motions. Adapting two recurrent model families (IMUPoser and MobilePoser), we show that one head IMU recovers lower-body pose at 79.0 mm rigid-MPJPE and per-foot ground contact at 0.809 macro-F1, and that a causal variant retains most of this accuracy at streaming latency. In paired per-take significance tests across both families, adding the consumer foot IMUs never significantly improves pose and significantly degrades it in two of four model-split combinations; a mounting-bias probe and feet-only ablation identify insole orientation quality, not foot placement, as the mechanism. Extending the output to a 20-joint full-body skeleton maps the boundary: gross distal-arm motion is partially recoverable from the head alone, proximal upper-body pose is not, and staged fine-tuning recovers the leg accuracy that naive joint training sacrifices to multi-task dilution. For learned pose from consumer wearables, sensor reliability, not sensor count, is the binding constraint here. For the devices tested, the earbud is its sweet spot. Code is available at https://github.com/ZhilinGuo/one-sensor-whole-body.

1 Introduction

The paper tests whether one consumer earbud IMU can recover useful 3D body pose and whether adding consumer sensors helps. It builds a real-data capture benchmark and finds the head stream strongest among the tested configurations.

1 Introduction

  • The benchmark pairs four-view RGB-D video with head and foot IMUs across 35 takes covering entertainment and clinically relevant movements.Streams are aligned post-hoc, and SAM 3D Body supplies pseudo-ground-truth labels.Source: §1 · pp. 1–2
  • The study evaluates adapted recurrent models on lower-body pose, foot contact, streaming inference, and a 20-joint full-body extension.Its claims are limited to one subject, one capture day, one earbud model, and one insole product.Source: §1 · p. 2
  • 79.0 mm rigid-MPJPE; the head alone also reaches 0.809 macro-F1 for foot contact, while adding two foot IMUs never significantly improves pose.Adding them significantly degrades pose in two of four model–split combinations.Source: §1 · p. 2

2 Related Work

Prior inertial pose systems generally use more sensors or richer tracking than a consumer earbud provides. This work addresses a real-data gap for single-earbud sensing and consumer insole reliability.

2 Related Work

  • Most sparse-IMU methods use six-IMU suits or pelvis sensors, while commodity-device evaluations have largely relied on simulated data.The study instead evaluates real captured streams, where consumer noise and mounting variability are present.Source: §2 · p. 2
  • Head-mounted XR pose systems often use 6-DoF head positions or hand trackers, whereas this setting uses one earbud IMU without positions or hands.Ear2Pos uses two earbuds and personalized bone lengths; this study uses one unified head stream without personalization.Source: §2 · p. 2
  • Whether consumer insole IMUs remain reliable beyond controlled straight-line locomotion is largely untested on real hardware.Feet are established cues for gait phase and contact.Source: §2 · p. 2
  • Unlike vision-plus-IMU work that uses inertial signals to stabilize pose, this pipeline uses vision to create pseudo-ground-truth labels.The passage identifies no prior benchmark targeting a single consumer earbud IMU with real multimodal capture and explicit sensor-reliability diagnostics.Source: §2 · p. 2

3 Capture System and Dataset

The capture system combines four-view RGB-D video with consumer head and foot IMUs. Its 35-take protocol spans varied movement scripts and yields aligned labels and sensor streams.

3 Capture System and Dataset

  • The rig records four Intel RealSense D455e RGB-D cameras, an AirPods fused head-motion stream, and left/right Striv insole IMUs.The head stream includes fused attitude, device-frame acceleration, and gyroscope; the insoles provide fused Euler orientation and acceleration at about 30 Hz, without raw gyroscope.Source: §3 · p. 2
  • Five repeated runs of seven motion scripts yield 35 takes spanning gait, stepping, turning, vertical motion, everyday composites, clinically inspired gait, and upper-limb coordination.The protocol enables leave-one-run-out and leave-one-motion-out evaluation.Source: §3 · p. 2
  • The benchmark contains 82,417 frames across 45.8 minutes, with usable lower-body labels for every frame and per-foot contact labels.Each foot is in ground contact for 65% of frames; at least one foot is in contact for 73%.Source: §3 · p. 2

4 Alignment and Pseudo-Ground-Truth

The pipeline aligns streams post-hoc using motion-based offsets, then derives lower-body and full-body pseudo-ground-truth from SAM 3D Body. The labels are acknowledged as less accurate than marker-based mocap.

4 Alignment and Pseudo-Ground-Truth

  • Streams are placed on a common 30 Hz grid and offsets are refined by matching head gyroscope magnitude and foot acceleration energy to SAM-derived motion.Across 35 takes, recovered offsets cluster at −2.69 ± 0.06 s for the head and −3.05 ± 0.10 s for the feet.Source: §4 · pp. 2–3
  • SAM 3D Body output is converted into nine lower-body joint targets, while its full 70-keypoint output is retained for the full-body extension.The labels are pseudo-ground-truth because single-view mesh recovery is less accurate than marker-based mocap.Source: §4 · p. 3

5 Benchmark Models and Protocol

The study adapts two recurrent model families for lower-body pose and contact from head-only or head-plus-feet IMUs, with synthetic pretraining and real-data fine-tuning. Its protocol compares held-out runs and motions using sequence-aligned pose metrics and paired per-take tests.

5 Benchmark Models and Protocol

  • IMUPoser-adapted predicts nine-joint rotations from 12-D sensor inputs, while MobilePoser-adapted uses a joint-position RNN followed by a pose RNN and adds a contact head to the IMUPoser backbone.Both are adapted to the lower-body task; IMUPoser-adapted uses forward kinematics to recover joint positions.Source: §5 · p. 3
  • All models are pretrained on synthetic AMASS IMUs and fine-tuned on real pseudo-GT, with leave-one-run-out testing seen motion types and leave-one-motion-out testing unseen classes.The latter is described as the harder generalization test.Source: §5 · p. 3
  • Rigid-MPJPE applies one similarity transform per sequence, while per-joint motion R² is measured against the temporal mean, giving static predictions approximately zero.PA-MPJPE and MPJVE are also reported for completeness.Source: §5 · p. 3
  • Head-only versus head-plus-feet rigid-MPJPE is paired across 35 takes and tested with Wilcoxon signed-rank tests, Holm-corrected across four model-by-split comparisons.The paired plot uses head-only on x and head-plus-feet on y; points above the diagonal favor head-only.Source: §5 · p. 3, Figure 2 · p. 3
  • The causal variant uses a same-capacity unidirectional LSTM with past observations only; the full-body variant predicts 20 joints and weakly supervises three spine joints with interpolated targets.Fine-tuning uses per-window rotation-plus-scale alignment to address global frame and scale ambiguity.Source: §5 · p. 3

6 Results

Across the benchmark, one head IMU performs strongly, while adding the consumer foot IMUs does not significantly improve pose and can degrade it. Full-body results show partial distal-arm recovery, and staged fine-tuning limits the leg-accuracy trade-off from joint training.

6 Results

  • 79.0 mm rigid-MPJPE and 0.49 distal R² are achieved by head-only IMUPoser-adapted on leave-one-run-out; head-only never loses significantly to head-plus-feet across four model-split comparisons.On leave-one-motion-out, IMUPoser-adapted reaches 92.6 mm versus 107.4 mm for the best multi-sensor configuration; paired tests find significant head-only wins in two IMUPoser-adapted comparisons.Source: §6 · p. 3, Table 2 · p. 4
  • The causal head-only model reaches 92.9 mm on leave-one-run-out and 91.5 mm on leave-one-motion-out at one-frame latency, beating bidirectional head-plus-feet results on both splits.It processes approximately 3,500 CPU frames per second.Source: §6 · p. 3
  • Foot-stream degradation tracks orientation reliability: no calibration worsens head-plus-feet to 107.0 mm, and injected mounting bias raises error to 118.9±1.2 mm at σ=40°.The insoles lack raw gyroscope data, unlike the AirPods stream; even the luckiest mounting-bias draw remains worse than discarding the feet.Source: §6 · p. 4
  • Head-only contact prediction reaches 0.809 macro-F1 across held-out takes while retaining 77.0 mm rigid-MPJPE, with contact easiest in upper-limb motions and hardest in turning.The corresponding F1 values are 0.968 for upper-limb motions and 0.694 for turning.Source: §6 · p. 4
  • In the 20-joint extension, distal arm motion is partially recovered, but neck, head, and shoulders are at or below the prior; four-view label fusion shifts proximal errors by under 2 mm.Elbow R² is 0.45±0.06 and wrist R² is 0.30±0.02; naive joint training gives 107.5±13.1 mm for the legs, near the 105.5 mm prior.Source: §6 · p. 4
  • Staged fine-tuning recovers leg error to 86.2±7.2 mm while retaining distal-arm signal, whereas freezing the trunk protects leg accuracy but eliminates that signal.On unseen motion classes the staged arm signal largely vanishes (R²=0.06); adding feet raises leg error to 130.0 mm.Source: §6 · p. 4

7 Discussion and Limitations

The paper frames its findings as within-participant feasibility for the tested hardware, not population-level generalization. The pilot uses one subject, one capture day, and SAM 3D Body pseudo-ground-truth rather than marker-based motion capture.

7 Discussion and Limitations

  • The pilot covers one subject, one capture day, and one earbud and insole product, so its claims are limited to within-participant feasibility for that hardware.Labels come from SAM 3D Body pseudo-ground-truth rather than marker-based motion capture.Source: §7 · p. 4
  • The labels are pseudo-ground-truth, not marker-based motion capture, and the four-view label check shares the SAM 3D Body backbone.The paper notes that the shared backbone remains a limitation of that label-quality control.Source: §7 · p. 4
  • Population-level generalization is not established by this single-subject pilot.The authors identify multi-subject capture as a next step.Source: §7 · p. 4

8 Conclusion

A single consumer earbud IMU supports lower-body pose and foot-contact estimation, and can recover gross arm swing with staged fine-tuning. For the devices tested, the head stream is the more reliable choice than adding foot IMUs.

  • 79.0 mm rigid-MPJPE for lower-body pose and 0.809 macro-F1 for foot contact show what one head IMU recovers.Source: §8 · p. 4
  • With staged fine-tuning, the head-only setup also recovers gross arm swing at streaming latency.Source: §8 · p. 4
  • For the consumer devices tested, the head IMU is never less and often significantly more accurate than adding two foot IMUs.Source: §8 · p. 4

Related papers

Loading 2609.34978v1…