PaperlayerView exact revision on arXiv

Source-linked AI summary

MuyBridge: Mobile Human Center-of-Mass Estimation from Monocular Video via Sparse Fusion

Aidan Bradshaw, Marco Giordano, David Rode, Andreas Habersack, Elif Basokur, Annika Kruse, Markus Tilp, Michele Magno, Peter Wolf, Luca Benini, Christoph Leitner

arXiv:2609.02854v1cs.CV

TL;DR

Metric CoM is difficult to measure from a single camera because existing approaches either lack anatomical constraints or require heavy capture infrastructure. MuyBridge addresses this with compact monocular pose and depth fused through anatomical and physical priors, achieving 33–41 mm vertical CoM error and 2.3–6.6% AbsRel range error across three athletic movement regimes.

  • Problem

    Metric CoM measurement commonly requires metric segment positions and anthropometric quantities from laboratory or dedicated capture systems, limiting single-camera use in athletic environments.

  • Method

    MuyBridge combines compact 2D pose, distilled single-step monocular depth, anthropometric constraints, and physical range cues through analytic metric fusion without task-specific 3D or CoM supervision.

  • Results

    33–41 mm vertical CoM error and 2.3–6.6% absolute-relative range error (AbsRel) are achieved across running, track and field, and figure skating.

  • Takeaways & Limitations

    MuyBridge demonstrates mobile whole-body CoM estimation on an iPhone 15, producing CoM estimates at the 63 FPS pose-estimation rate with asynchronous 2.86 Hz depth updates.

Abstract

from arXiv · show

The 3D center of mass (CoM) is a primary quantity in the biomechanical analysis of sport, rehabilitation, and clinical movement, yet existing 3D pose tracking, mesh recovery, and multi-view triangulation methods either optimize 3D keypoint accuracy without anatomical constraints or carry compute and capture infrastructure too heavy to deploy where CoM tracking is most useful. As a result, the metric CoM remains difficult for coaches and movement analysts to measure from a single camera where athletes train and compete. In this work, we introduce MuyBridge, an on-device system that estimates the athlete's segmental center of mass trajectory from a single phone camera video stream. MuyBridge couples a compact 2D pose network and a distilled single-step monocular depth network through an analytic metric fusion that uses anatomical and physical priors to anchor the metric CoM, requiring no 3D or task-specific supervision. Evaluated on the athletic movements of AthletePose3D (running, track and field, and figure skating), MuyBridge achieves 33-41 mm vertical CoM error and 2.3-6.6% absolute-relative range error (AbsRel) under a one-time calibration, and produces CoM estimates at the 63 FPS pose-estimation rate using asynchronous 2.86 Hz depth updates on iPhone 15. Code is available at: https://github.com/Abradshaw1/Muybridge

1. Introduction

MuyBridge targets mobile metric CoM estimation by combining compact monocular pose and depth with anatomical and physical constraints. It is evaluated across athletic movements with reported accuracy and mobile throughput.

  • Whole-body CoM is important for analyzing balance, acceleration, and landing, but conventional metric segment measurements often require laboratories or dedicated capture facilities.
  • Existing vision methods estimate keypoints, 3D pose, meshes, or scene depth but generally optimize those outputs rather than directly recovering anatomical quantities.
  • 63 FPS pose estimation with asynchronous 2.86 Hz depth updates supports CoM estimation at the pose rate on an iPhone 15.
  • The evaluation characterizes performance across viewpoint, motion phase, and individual range cues while the mobile implementation profiles latency, throughput, memory, and energy.
  • MuyBridge combines image-space pose, subject-specific anthropometry, sparse monocular depth, and geometric range constraints without task-specific 3D-pose or CoM supervision.

2. Related Work

Related work advances biomechanically constrained vision and efficient inference, but prior approaches differ in their target quantities, reconstruction settings, and computational demands.

  • Biomechanically grounded methods incorporate skeletal, center-of-mass, center-of-pressure, body–floor, or musculoskeletal constraints into monocular or multi-view human analysis.
  • Kaichi et al. estimate CoM from multi-view visual-hull reconstruction but do not define the metric segmental quantities required for whole-body CoM estimation.
  • Efficient visual inference reduces the cost of 2D pose, temporal 3D pose, and monocular scene-geometry estimation under resource constraints.

3. Method

MuyBridge reconstructs metric joints from monocular pose and sparse depth, then computes whole-body CoM using segmental anthropometry. Analytic fusion combines anatomical measurements with physical range cues and temporal modeling.

  • Segmental Center-of-Mass Model: Whole-body CoM is computed as a mass-weighted sum of segment centers defined between anatomical joint endpoints.Each segment center uses a longitudinal CoM position, and sex-specific mass fractions and positions are applied.
  • System Overview: MuyBridge uses three stages: compact 2D pose, single-step monocular depth, and analytic metric fusion for metric joint positions and whole-body CoM.
  • 2D Pose Network: The pose branch predicts 26 Halpe-26 anatomical keypoints from each RGB frame and is trained without AthletePose3D or sport-specific supervision.
  • Monocular Depth Network: The depth branch predicts a dense affine-invariant normalized depth field, whose keypoint samples provide relative range information while fusion resolves metric scale.Latent consistency distillation reduces inference to a single UNet evaluation.
  • Sparse Keypoint-Depth Measurements: Keypoint depths are filtered using pose confidence and local depth consistency, then summarized by the median of retained samples as a robust relative range signal.
  • Metric Fusion: Known segment lengths and endpoint rays provide geometric range estimates, while ground contact and ballistic motion supply additional metric observations.Range estimates across visible segments use a fixed low quantile because foreshortening biases individual estimates upward.
  • Metric Reconstruction: Physical and depth-derived measurements are fused with a constant-velocity range model to obtain final athlete range, which places keypoint rays in metric camera coordinates.

4. On-Device Deployment

MuyBridge compresses its pose and depth networks for mobile execution, then deploys them as a Core ML pipeline on iPhone 15. Compression improves pose throughput while depth remains the dominant computational and memory cost, with modest depth-accuracy degradation.

  • Pose Network Compression: Pose accuracy is evaluated with AUC and PCK@0.1, while Table 1 compares the compressed INT8 model against full-size RTMPose.PCK@0.1 measures keypoints within 10% of the reference scale.
  • Pose Network Compression: 1.82× RTMPose throughput and a 7.0-point PCK@0.1 gain are achieved by the compressed pose model.The reported accuracy difference reflects Halpe-26 retraining rather than quantization.
  • Depth Network Compression: Eight-bit quantization is applied to convolution and projection layers, while attention and normalization remain in FP16 because full INT8 attention degrades depth accuracy.Weights use symmetric per-channel quantization and activations use asymmetric per-tensor quantization.
  • Depth Network Compression: 0.008–0.023 AbsRel increases and 0.011–0.076 δ1 decreases quantify the compressed depth network’s accuracy loss across datasets.The largest degradation occurs on KITTI-Depth, while NYU-Depth remains comparatively stable.
  • Pipeline Deployment and Profiling: Both networks run as compiled Core ML graphs in a native Swift iPhone 15 application, with CPU-based metric fusion and CoM computation.The depth model is split into encoder, denoiser, and decoder graphs, and camera intrinsics plus one-time scene calibration support deployment.
  • Pipeline Deployment and Profiling: 349.9 ms depth updates dominate the 15.58 ms pose inference cost, model size, energy, and memory in the deployed pipeline.The full pass costs 349.9 ms and 638 mJ, while peak memory reaches 1,185 MB for the complete pipeline versus 126.8 MB for pose alone.

5. Experiments

MuyBridge is evaluated across running, track and field, and figure skating using synchronized AthletePose3D data, with analyses of accuracy, viewpoints, motion phases, range cues, and body representations. Errors are dominated by camera-depth localization, while sparse depth, physical cues, and segmental anatomy jointly improve metric CoM estimation.

  • Evaluation protocol: The evaluation uses eight athletes and approximately 1.3M synchronized multi-view frames, partitioned into 1,028 sequence-camera pairs and 235,183 frames without AthletePose3D fine-tuning.Errors are averaged within sequence-camera pairs and reported as medians over pairs.
  • Overall accuracy: Camera-depth error contributes 166 of 187 mm in running, 117 of 185 mm in track and field, and 672 of 707 mm in figure skating.Lateral and vertical errors remain substantially smaller, identifying monocular range recovery as the dominant source of metric localization error.
  • Cyclic locomotion: 187 mm 3D CoM MAE and 3.6% range AbsRel are achieved in running at a mean athlete range of 4.4 m.Median per-axis errors are 44 mm lateral, 33 mm vertical, and 166 mm in depth.
  • Confined rotational movement: 185 mm 3D CoM MAE and 2.3% range AbsRel are achieved in track and field at a mean range of 5.4 m, while javelin spans 200–429 mm across cameras.Among fixed-circle events, 21 of 32 camera–discipline combinations remain below 200 mm 3D CoM MAE.
  • Ballistic movement: 707 mm 3D CoM MAE and 6.6% range AbsRel occur in figure skating at a 10.1 m athlete range, where airborne motion removes the ground-contact cue.The overall jump trajectory is recovered, but small image-space perturbations produce larger metric displacements at this range.
  • Motion phase: Contact error ranges from 3.6–5.6% of athlete range versus 7.0–8.7% during flight, a 1.5–2.3× increase across all six jump types.The consistent phase pattern associates degradation with airborne motion rather than a specific jump.
  • Ablation study: Holding athlete range constant increases error from 187 to 721 mm in running and from 707 to 1890 mm in skating, while torso-only reconstruction reaches 284, 252, and 733 mm.Removing either the ground-contact or keypoint-depth cue also increases error across all motions.

6. Conclusion

MuyBridge estimates metric center of mass from monocular video through keypoint depth, anatomical constraints, and physical range cues. Across three athletic motion regimes, it delivers millimeter-scale vertical CoM accuracy on mobile hardware, while range recovery remains the primary limitation.

  • MuyBridge uses three-stage fusion of keypoint depth, anatomical constraints, and physical range cues for monocular metric CoM estimation.
  • 33–41 mm vertical CoM error and 2.3–6.6% AbsRel range error were achieved across running, track and field, and figure skating.
  • 63 FPS CoM output was sustained on an iPhone 15 using asynchronous 2.86 Hz depth updates.
  • Absolute localization is primarily limited by camera-to-athlete range, especially during sustained translation and long-range airborne motion.
  • The system assumes one-time scene calibration and uses stature-scaled, sex-specific population segment proportions, while AthletePose3D contains eight athletes with markerless reference trajectories.

B. Extended Pose Network Evaluation

The extended pose evaluation attributes most pose-processing cost to the keypoint backbone and reports stable throughput during continuous execution.

  • 12.53 of 15.58 ms and 100.2 of 126.8 MB are attributed to the keypoint backbone.
  • The p99 latency remains within 2.5 ms of the median, indicating stable throughput under continuous execution.

C. Extended Depth Network Evaluation

The extended depth evaluation finds consistent metric ordering and identifies the UNet as the dominant cost of the depth branch.

  • The ordering reported in Table 2 holds across every metric in the full monocular-depth metric suite.
  • The UNet contributes 210.7 ms and 845.0 MB, and sets the full 1,185 MB peak memory of the depth branch.

D. Fused Pipeline Profiling

Peak memory follows the depth branch across all evaluated resolutions because the pipeline branches do not peak concurrently.

  • Peak memory tracks the depth branch across all three evaluated resolutions.
  • The branches do not peak concurrently, explaining why peak memory follows the depth branch.

E. Running Partition by Camera View

Camera view changes the error profile: Camera 4 has the best lateral and vertical components but the worst depth component, with view variation dominated by camera-depth localization.

  • Camera 4 achieves the best lateral and vertical components while producing the worst depth component.
  • Camera-depth localization dominates variation in running results across camera views.

F. Jump Height in Figure Skating

MuyBridge recovers figure-skating jump height from vertical center-of-mass displacement across hundreds of sequence–camera pairs. The evaluation reports both accuracy and correlation with reference jump heights, while qualitative reconstructions show the end-to-end outputs.

  • 62 mm MAE recovers jump height across 719 sequence–camera pairs spanning 0.26–1.31 m reference height.Jump height is defined as the takeoff-to-apex rise of the vertical CoM.
  • The recovered jump heights correlate with reference height at r = 0.84 and have a mean signed error of +12 mm.
  • Qualitative end-to-end reconstructions show RGB frames, predicted depth, 2D keypoints, image-plane CoM, and recovered metric CoM.
  • Table 15 reports median absolute error over sequence–camera pairs, with bias as mean signed error and GT range as the reference span.
  • Jump-height recovery is a vertical-displacement metric and is not directly comparable to absolute 3D CoM MAE.
Loading 2609.02854v1…