PaperlayerView exact revision on arXiv

Source-linked AI summary

Reliability-Gated Fusion of Consumer Head and Foot IMUs for Lower-Body 3D Pose

Zhilin Guo, Boqiao Zhang, Oszkár Urbán, Josef Bengtson, Hakan Aktas, Wenzhao Li, Siyu Hong, Kyle Fogarty, Chenliang Zhou, Ali Senguel, Cengiz Oztireli

arXiv:2609.35764v1cs.CVcs.HC

TL;DR

Sparse pose estimation from consumer IMUs must contend with unreliable sensor channels, including biased fused orientations. This paper learns temporal gates for each stream’s channel blocks and finds the gated model strongest among its learned-fusion arms across clean data and simulated faults. Its deployment analysis also identifies per-take mounting calibration as a remaining boundary.

  • Problem

    Consumer IMU pose estimation uses a sparse layout without pelvis sensing or raw foot gyroscopes, while foot orientation can undermine the useful acceleration signal.

    Source: §1 · pp. 1–2

  • Method

    The model applies temporal reliability gates to each stream’s orientation and acceleration blocks, trained with an auxiliary reliability objective on synthetically corrupted pretraining data.

    Source: §1 · p. 2, §4 · p. 4

  • Results

    69.4 / 77.8 / 69.6 / 72.8 mm clean/bias/drift/dropout, channel gating is the most accurate learned-fusion arm across the reported conditions.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results locate consumer-sensor reliability at the channel level: gating can retain useful acceleration while suppressing unreliable orientation without test-time supervision.

    Source: §1 · p. 2, §7 · p. 8

    Per-take mounting calibration remains necessary; without it, closing the gap without per-take vision is identified as a deployment blocker.

    Source: §6.4 · p. 8

Abstract

from arXiv · show

Sparse inertial pose estimation promises camera-free motion capture from consumer devices, but consumer sensors are unreliable: firmware-fused orientations are biased, mounting varies between sessions, and streams drift or drop out. On a new 35-take single-subject benchmark pairing an earbud head inertial measurement unit (IMU) with two smart-insole foot IMUs (SAM-3D-Body pseudo-ground-truth labels), we show the reliability problem is channel-level: a channel ablation isolates foot acceleration as the most informative input (66.6 mm vs. 79.0 mm head-only) and the firmware-fused foot orientation as the liability that destroys the gain. We therefore let the model learn how much to trust each channel of each stream: one temporal gate per stream per channel block, trained with an auxiliary reliability objective on synthetically corrupted pretraining data. The channel-gated model is the most accurate of our learned fusion arms on clean data (69.4 mm vs. 83.7 static, 86.6 ungated) and under every simulated fault (bias in training; drift, dropout eval-only); its gates suppress the natively biased foot-orientation channels on clean real data without test-time supervision and flag dropout bursts at 0.92-0.999 AUROC. Two contrasts: dropping a channel known a priori to fail is flat across foot faults but collapses when an unanticipated stream fails (head dropout: 92.9 vs. 79.3 mm); and a fine-tuned HMD-Poser is more accurate on clean data (64.4 mm) and nominally under drift, with no significant paired difference under bias or dropout, but a larger worst-case degradation from clean (+16.1 vs. +3.5 mm, single seed). Learning to gate reliability instead of sensor count is the lever for deployable sparse inertial capture. Code is available at https://github.com/ZhilinGuo/reliability-gated-imu-fusion.

1. Introduction

The paper studies sparse lower-body pose estimation from consumer head and foot IMUs, where useful foot acceleration is coupled to unreliable fused orientation. It introduces channel-reliability gating and evaluates it on a 35-take benchmark.

1. Introduction

  • The consumer-device layout has no pelvis sensor or raw foot gyroscope, leaving root translation and hip axial rotation weakly observed.The paper also compares a priori channel dropping with learned gating and reports external-baseline and deployment contrasts.Source: §1 · pp. 1–2
  • Foot acceleration alone reaches 66.6 mm, significantly better than head-only, while adding foot orientation destroys most of that gain (+12.9 mm, p<10−5).Foot orientation alone is worse than no feet on average (p=0.054).Source: §1 · p. 2
  • The model learns temporal reliability gates for each stream’s orientation and acceleration channels using corruption-supervised pretraining.On clean real data, gates partially close on natively biased foot orientations; stream-level gates add little over static weights.Source: §1 · p. 2
  • The 35-take single-subject benchmark pairs four RGB-D views for labels with an earbud head IMU and two smart-insole foot IMUs.It includes released capture, alignment, and pseudo-GT labeling protocols, with run- and motion-held-out evaluation.Source: §1 · p. 2
  • 69.4 / 77.8 / 69.6 / 72.8 mm clean/bias/drift/dropout, channel-reliability gating is the most accurate learned-fusion arm across these conditions.The gates suppress natively biased foot-orientation channels and flag dropout bursts at 0.92–0.999 AUROC without test-time supervision.Source: §1 · p. 2

2. Related Work

Prior sparse-IMU methods establish the methodological backbone but often assume pelvis sensing and raw gyroscopes. This work situates head-plus-feet consumer sensing among foot, robustness, and vision-assisted approaches.

2. Related Work

  • Learning-based six-IMU pipelines [13], real-time physics-aware optimization [33] [34], and Transformers [16] form prior sparse-IMU methods, but assume pelvis sensors and raw gyroscopes.Commodity-device systems include IMUPoser [21] and MobilePoser [29], while HMD-Poser [3] infers legs from upper-body inputs.Source: §2 · p. 2
  • Reduced-IMU lower-limb estimation reports roughly 6 cm segment error and degrades on turns and dynamic motion [26], while insole pressure has also been paired with a physics digital twin [10].Wearables are also used in Parkinsonian gait assessment.Source: §2 · p. 3
  • Prior robustness work includes dynamic calibration [39], magnetic hardening [23], and uncertainty propagation [18]; WHIP uses raw IMU channels and hardware-synchronized mocap [2].The paper claims no new gating mechanism [1] [8] [11] [22], but instantiates and validates gating for test-time inertial-sensor faults.Source: §2 · p. 3
  • DeepFuse fuses body-worn IMUs with multi-view images [12], whereas this work uses vision to create labels and reports no prior benchmark of head-plus-two-feet sensing without pelvis or foot gyroscopes.Multi-view alignment without hardware synchronization has prior precedent.Source: §2 · p. 3

3. Capture, Alignment, and Pseudo-Ground-Truth

The benchmark combines consumer wearables with four-view RGB-D capture, then aligns the streams after capture and derives lower-body pseudo-ground-truth labels. Its labels and per-take sensor calibration are explicitly bounded by the study’s setup.

3. Capture, Alignment, and Pseudo-Ground-Truth

  • The capture includes four RGB-D cameras, an AirPods head IMU, and left/right Striv insole IMUs across 35 takes and 82,417 frames.The motion protocol spans gait, turns, balance, everyday composites, clinical gait, and upper-limb motions.Source: §3 · p. 3
  • Streams are resampled to 30 Hz and aligned per take by motion agreement; SAM 3D Body supplies nine-joint lower-body pseudo-GT labels.Head and shared-foot offsets are estimated by cross-correlating wearable motion against SAM-derived motion.Source: §3 · p. 3
  • The authors treat SAM-derived labels as biased pseudo-GT rather than mocap-grade truth, and the four-view control tests label-view choice rather than absolute correctness.Each take also receives a spatial sensor-to-bone rotation fit using that take’s own SAM-derived bone frames.Source: §3 · p. 3

4. Method

The method adds channel-specific temporal reliability gates to a recurrent inertial pose model. Synthetic sensor corruption supervises gate reliability during pretraining, followed by real pseudo-GT fine-tuning without gate loss.

4. Method

  • The model uses per-stream stems, gated fusion, a bidirectional LSTM, and forward kinematics to regress nine lower-body joints.Its inputs are acceleration and orientation; the pose head predicts 6D rotations.Source: §4 · pp. 3–4, Figure 2 · p. 4
  • Each stream’s acceleration and orientation features receive separate temporal scalar gates, allowing one channel block to be retained while the other is suppressed.The gated features are concatenated before projection into the pose trunk.Source: §4 · p. 4
  • The channel-level temporal gate uses a short temporal receptive field, while baseline variants include frozen, static, per-frame, and stream-level temporal gates.The temporal gate uses a kernel of 9 with symmetric padding.Source: §4 · p. 4
  • Pretraining applies bias, drift, and dropout to random foot streams and supervises gates with binary cross-entropy; real-data fine-tuning turns gate loss off.Bias and drift mark orientation corrupted, while dropout marks both channel blocks; corruption augmentation remains active during fine-tuning.Source: §4 · p. 4, Figure 2 · p. 4

5. Experimental Setup

Evaluation holds out either a run or an entire motion class, with the latter testing harder generalization. Metrics are root-relative and emphasize sequence-aligned pose error, motion variance explained, and fault robustness.

5. Experimental Setup

  • Leave-one-run-out tests unseen repetitions of seen motions, while leave-one-motion-out holds out an entire motion class as the harder generalization test.A constant mean-pose prior and a zero-input control serve as sanity floors.Source: §5 · p. 4
  • Root-relative sim-MPJPE uses a single per-sequence scale and rotation, while model positions come from forward kinematics with fixed bone lengths.Root translation is neither predicted nor evaluated; per-frame PA-MPJPE is deemphasized because a static mean pose scores well after alignment.Source: §5 · pp. 4–5
  • Per-joint motion R2 measures the fraction of joint-motion variance explained, and distal variants focus on knees, ankles, and feet.MPJVE measures finite-difference velocity error, while foot-contact quality is scored against SAM-derived contacts.Source: §5 · p. 5
  • Foot faults include constant orientation bias, eval-only random-walk drift, and eval-only packet dropout in bursts covering about 10% of frames.Three-seed run-split results use paired per-take tests, with script-level sign-consistency tests reported as cluster-robust evidence because takes cluster by motion class.Source: §5 · p. 5

6. Results

The results isolate foot-orientation reliability—not sensor count—as the key issue, then show channel-level gates outperform other learned fusion arms across the tested faults. Gates respond to native bias and dropout without test-time supervision, while calibration requirements and external-baseline comparisons bound deployment claims.

6.1. The problem is channel-level

  • 66.6 mm with foot acceleration alone is the best tested configuration; adding foot orientation raises error to 80.3 mm.The foot-orientation bias dose-response is monotonic across seeds, from 86.6 to 114.7 mm as bias increases from 0° to 40°.Source: §6.1 · pp. 5–6, Table 1 · p. 6

6.2. Reliability-gated fusion

  • 69.4 mm clean and 77.8/69.6/72.8 mm under bias/drift/dropout make the channel gate the best learned fusion arm across regimes.It beats the ungated control by 16.8 mm on clean data and by 22.1/16.6/20.5 mm under bias/drift/dropout.Source: §6.2 · p. 6, Table 2 · p. 6

6.3. What the gates learn

  • On clean real data, foot-orientation gates are partially closed without test-time supervision, while foot-acceleration gates close inside dropout bursts and reopen afterward.Foot-orientation gate values are 0.41 and 0.15; foot-acceleration gates average 0.08 inside bursts versus 1.00 outside.Source: §6.3 · p. 7, Figure 4 · p. 7

6.4. External baselines and deployment contrasts

  • The channel gate is the most accurate learned arm under each reported external-baseline fault, with 4.6× HMD-Poser’s CPU throughput and 37% fewer parameters.The Table 3 comparison uses eval-only foot corruption and a single seed.Source: Table 3 · p. 7, §6.4 · p. 7
  • HMD-Poser scores 64.4 mm on clean data but has a larger worst-case degradation than the channel gate: +16.1 versus +3.5 mm.The paired comparison favors HMD-Poser on clean data and nominally under drift, but finds no significant difference under bias or dropout.Source: §6.4 · pp. 7–8, Table 3 · p. 7
  • The gated model outperforms the a priori foot-acceleration-only configuration when the unanticipated head stream drops out: 79.3 versus 92.9 mm.The gate partially closes on head orientation during bursts, despite head faults being absent from training.Source: §6.4 · p. 8
  • A one-time opening T-pose calibration is worse than no calibration, while full per-take calibration reaches 80.3 mm; without per-take vision, the best camera-free arm reaches 94.0 mm.The authors identify closing the gap to calibrated head-only performance as a deployment blocker.Source: §6.4 · p. 8
  • Slow drift is not detected directly: the channel gate’s drift margin comes from suppressing the orientation block before drift accumulates.White noise in the trusted acceleration channel instead degrades the gated model more than the ungated one, 143 versus 118 mm.Source: §6.4 · p. 8

7. Discussion and Conclusion

Consumer sensor reliability is channel-level: foot acceleration is valuable while fused orientation is harmful, and fixed remedies fail. Channel-reliability gating resolves this trade-off without test-time supervision, but the evidence and deployment setting remain bounded.

7. Discussion and Conclusion

  • The same foot stream couples valuable acceleration with toxic fused orientation, making consumer sensor reliability a channel-level property.Augmentation without gating, oracle hard masking, and one-time static calibration all fail.Source: §7 · p. 8
  • Channel-reliability gating resolves the trade-off without test-time supervision.Source: §7 · p. 8
  • Millimeter claims are relative to SAM-3D-Body labels, and the benchmark covers one subject and day with synthetic corruptions and no pelvis sensor.Per-take vision-derived mounting calibration is also required.Source: §7 · p. 8

Related papers

Loading 2609.35764v1…