PaperlayerView exact revision on arXiv

Source-linked AI summary

Biomechanical 3D Body: Self-Supervised Distillation of Biomechanical Pose from a 3D Body Foundation Model

R. James Cotton, J. D. Peiffer, Lucinda Williamson, John Leske, Georgios Pavlakos

arXiv:2608.29928v1cs.CV

TL;DR

Monocular body-recovery models do not directly provide clinically defined biomechanical joint angles, and paired image–biomechanics data are limited. The paper adds a biomechanical head to frozen SAM-3D-Body and trains it through self-distillation with differentiable kinematics and in-loop inverse-kinematics targets. B3D outperforms an existing skeleton-regression model and is competitive with a state-of-the-art optimization-based method, while enabling faster feed-forward inference.

  • Problem

    Monocular mesh-recovery outputs lack biomechanically defined joint angles, while large paired image–biomechanics datasets are unavailable for direct regression.

  • Method

    B3D grafts a trainable biomechanical head onto frozen SAM-3D-Body and distills its outputs using differentiable MuJoCo kinematics with inline Levenberg–Marquardt IK pseudo-labels.

  • Results

    B3D outperforms an existing human skeleton regression model and is competitive with a state-of-the-art monocular biomechanics method requiring more expensive inference-time optimization.

  • Takeaways & Limitations

    Self-distillation can produce high-quality monocular biomechanical estimates with a single feed-forward inference pass and fewer outliers than cold-start inverse kinematics.

  • Takeaways & Limitations

    B3D is faster than PBL but does not exceed its accuracy; PBL remains ahead on several joint-center comparisons, especially in the clinical cohort’s upper limb.

Abstract

from arXiv · show

State-of-the-art monocular body recovery methods predict mesh vertices and angles on the corresponding kinematic tree, but their outputs lack biomechanically defined joint angles that downstream applications like clinical and biomechanical analyses require. We extend an existing foundation model, SAM-3D-Body, with an additional biomechanical prediction head that, from a single RGB image, regresses the joint angles and scales of a biomechanical model. Training this model presents a challenge, as there are limited datasets of paired images and biomechanical fits. To overcome this, we supervise biomechanical outputs with in-loop optimized targets from a Levenberg-Marquardt solver performing inverse kinematics fits against markers from the mesh predictions. This allows distilling the biomechanical head from the mesh head, even from unlabeled images. To make this work with GPU-optimized biomechanical models in MuJoCo, the entire model was implemented in JAX using Equinox. We trained this distilled output head on the publicly released SAM-3D-Body dataset. We then validated this model on biomechanical fits to two publicly available marker-based datasets, MoVi and BioCV, as well as movements from a clinical cohort captured with multiview markerless motion capture. The resulting model outperforms existing models for direct regression of biomechanics from images while only slightly underperforming the state-of-the-art monocular biomechanics method that performs more costly inference-time optimization of entire trajectories.

1 Introduction

Monocular mesh models do not directly provide clinically defined biomechanical joint angles, and paired image–biomechanics data are scarce. The paper addresses both obstacles by adding a trainable biomechanical head to frozen SAM-3D-Body and distilling it with in-loop optimization.

  • Motivation: SAM-3D-Body predicts high-quality MHR meshes, but its kinematic tree and joint angles do not match clinical biomechanical definitions.This mismatch limits direct use of mesh-recovery outputs for clinical and biomechanical analyses.
  • Challenges: No large paired image–biomechanics dataset exists for directly training a biomechanical regressor.The paper therefore builds on biomechanical fits obtained from SAM-3D-Body MHR estimates.
  • Approach: A small trainable biomechanical head grafted onto a frozen SAM-3D-Body backbone adds clinically interpretable, MuJoCo-compatible outputs with minimal parameter and compute cost.The head reads the foundation model representation while preserving the frozen backbone and decoder.
  • Approach: The head is trained without paired biomechanical labels by distilling from the frozen MHR head through an EM-fitted marker mapping and differentiable MuJoCo forward kinematics.Ground-truth MHR pseudo-labels from the training dataset are not used; only SAM-3D-Body predictions supervise training.
  • Approach: An inline Levenberg–Marquardt inverse-kinematics solver supplies cold-to-warm pseudo-labels, allowing the head to learn seeds that drive optimization beyond cold-start performance.This model-optimization-in-the-loop strategy is used to train the biomechanical output head.

2 Related Work

Prior work progressively connects monocular mesh recovery with biomechanical estimation, but existing approaches generally rely on model fitting or trajectory optimization rather than direct feed-forward regression.

  • Surface-mesh body recovery: SMPLify established optimization-based fitting of surface meshes to images, HMR demonstrated direct parameter regression, and SPIN placed optimization inside training.These methods form the methodological lineage for self-supervised mesh-to-body-model distillation.
  • Markerless biomechanical estimators: OSSO and SKEL introduce biomechanically accurate internal skeletons and joints into surface-based human representations.HSMR builds on SKEL while converting SMPL pseudo-ground truth and refining labels with iterative optimization.
  • Markerless biomechanical estimators: PBL achieves state-of-the-art monocular biomechanical kinematics by optimizing trajectories against 3D and reprojection errors, while prior work recovers MHR-based joint angles with Levenberg–Marquardt IK.The present paper uses that IK solver inside training rather than relying only on inference-time optimization.

3 Methods

The method adds a biomechanical prediction head to frozen SAM-3D-Body, maps its outputs to MHR-derived marker targets, and trains with differentiable MJX forward kinematics plus inline IK pseudo-labels.

  • Architecture: A person crop passes through frozen SAM-3D-Body and MHR components, while a trainable head predicts biomechanical parameters from the pose token.At inference, the system emits both the MHR mesh and biomechanical skeleton in one feed-forward pass.
  • Biomechanical model: MJX implements the biomechanical model’s forward kinematics as a differentiable operator returning world-frame joint centers and site-marker positions.This allows gradients from marker-position discrepancies to train the biomechanical head.
  • Biomechanical model: The biomechanical model uses generalized coordinates, per-segment scales, and per-marker offsets to represent clinically defined anatomy.Its MuJoCo humanoid has a flexible lumbar spine, articulated hands, 99 generalized coordinates, and 102 anatomical marker sites.
  • MHR-to-biomechanical mapping: Because MHR and MuJoCo differ in marker sites and joint definitions, an EM procedure learns a marker correspondence and marker-specific confidence weights.Matched vertices, keypoints, and joint centers are gathered into a common marker order for distillation.
  • Training objective and inline IK: The inline Levenberg–Marquardt solver fits q and β in stages, progressing from neutral initialization to warm starts from the biomechanical head as training improves.The resulting pseudo-labels are stop-gradiented before entering the training objective.
  • Training and evaluation: Training uses 1,486,720 distinct images and 3,722,776 person-crop samples from SAM-3D-Body, with evaluation on separate biomechanical datasets.The external evaluation includes BML-MoVi, BioCV, and a clinical cohort, while inference can optionally apply warm IK refinement.

4 Results

Across marker-based and clinical evaluations, the proposed method remains competitive with optimization-based PBL and outperforms HSMR and OpenCap-Monocular, while offering substantially faster feed-forward inference.

  • Evaluation setup: The method uses bias-corrected metrics across marker-based references and clinical data to compare methods despite differing skeleton formats.A single per-joint median offset is subtracted before scoring.
  • Qualitative comparison: Qualitatively, the method aligns slightly better than HSMR in some regions, while its gait waveforms track the reference more tightly than HSMR and OpenCap-Monocular.The waveform comparison uses an example clinical walking trial.
  • Marker-based references: On BML-MoVi and BioCV, the method tracks dominant lower-limb angles, recovers joint centers within several centimetres, and outperforms HSMR and OpenCap-Monocular across all metrics.It remains competitive with PBL, and inline IK at inference has minimal impact on BML-MoVi.
  • Clinical MMMC cohort: On the clinical cohort, the method remains highly competitive except against PBL, while its advantage over HSMR and OpenCap-Monocular becomes more pronounced in several populations.OpenCap produced no scored fit for 400 of 1693 inputs, potentially biasing its reported accuracy favorably.
  • Inference speed: The feed-forward method runs at roughly 15–20 frames/s in batched mode, whereas PBL runs at roughly 0.03–0.14 frames/s.PBL is a whole-clip optimization and is two to three orders of magnitude slower.
  • Training ablations: SPIN improves marker fit from 21.1 to 12.9 mm and biomechanical joint-center PA-MPJPE from 27.5 to 17.3 mm.The SPIN-disabled baseline required additional regularization to prevent divergence.
  • Training ablations: The feed-forward head reaches similar median joint errors to a cold solver while reducing per-frame cost from 180 ms to 41 ms.The cold solve has a much heavier error tail.

5 Discussion

B3D produces biomechanical estimates through self-distillation and performs competitively against existing methods, while remaining faster than optimization-based PBL. Its limitations include cross-model evaluation differences, shoulder and arm accuracy gaps, and frame-wise rather than temporal inference.

  • Performance: B3D outperforms an existing human skeleton regression model and is competitive with a state-of-the-art method requiring more expensive inference-time optimization.The model was validated against marker-based and multiview markerless references.
  • Evaluation: Joint-angle error was used as the primary metric alongside MPJPE, revealing substantial limitations in HSMR and OpenCap.HSMR tends to keep many joints overly flexed.
  • Accuracy patterns: All methods were less accurate in the arms than the legs, with shoulder-plane and axial-rotation estimates performing worst.Elbow flexion and shoulder elevation were recovered within a few degrees, whereas shoulder-plane and axial-rotation errors were several times worse.
  • Training: The in-loop Levenberg–Marquardt IK solver enabled self-distillation without SAM-3D-Body labels and may support refinement on additional external videos where MHR regression remains robust.The external-video refinement possibility remains untested.
  • Limitations: IK-fit accuracy is bounded by the MHR-alignment loss, with some bones extending slightly beyond the MHR mesh.More degrees of freedom, refined scaling, or a more complete shoulder model could improve the fits.
  • Comparison with PBL: Compared with PBL, B3D is faster but does not exceed its accuracy, with PBL retaining clearer advantages on the clinical cohort.Against marker-based references, the margins are small enough that the methods are best read as comparable overall.
  • Limitations: Because B3D estimates each frame independently, it ignores inter-frame cues and does not enforce temporal continuity.PBL gains accuracy partly through bundle adjustment with skeleton scale constrained across frames.

6 Conclusion

B3D uses model-in-the-loop self-distillation to train a biomechanical head on SAM-3D-Body. It achieves performance competitive with state-of-the-art optimization-based methods while substantially accelerating inference.

  • Conclusion: B3D trains a biomechanical output head on a SAM-3D-Body backbone through model-in-the-loop self-distillation.The approach provides performance competitive with optimization-based methods while accelerating inference.

A.1 Full per-joint metric tables

The supplementary metric tables define how position and joint-angle errors are pooled, reported, and compared across methods and datasets. They also specify cohort sizes and OpenCap’s accepted-trial subset.

  • Reporting: Per-joint tables report MJAE in degrees with nIQR, while PA-MPJPE and Rigid MPJPE are reported in millimeters.Lower values are better throughout the supplementary tables.
  • Position metrics: PA-MPJPE pools 19 joint centers shared by all methods, whereas Rigid MPJPE spans each method’s full exposed joint set.The two position metrics are therefore not strictly comparable across methods.
  • Joint-angle metrics: Avg MJAE is bias-corrected and pooled over shared hip, knee, ankle, elbow, and shoulder degrees of freedom.The uncorrected row reports each method’s raw per-group mean, and † marks optimization-with-kinematic-fitting methods.
  • Dataset coverage: The BML-MoVi cohort contains 85 subjects and the clinical MMMC cohort contains 1693 trials.OpenCap is scored on 1293 trials accepted by its WHAM front-end.

A.2 Qualitative skeleton overlays

The supplementary figures show cross-method biomechanical skeleton overlays for clinical MMMC and held-out BML-MoVi frames. B3D generally registers limb segments closely to reference fits, while HSMR retains knee and trunk offsets.

  • Figure scope: Figures S1 and S2 visualize predicted biomechanical skeletons against references for the clinical MMMC cohort and held-out BML-MoVi data.The overlays provide qualitative cross-method comparisons.
  • Related quantitative table: Table S2 provides a per-joint cross-method comparison on the clinical gait cohort against the MMMC reference.It reports MJAE with nIQR and PA-MPJPE/Rigid MPJPE, with lower values preferred.
  • Clinical MMMC: Figure S1 compares Input, Ours, Ours+IK, PBL, HSMR, and OpenCap on representative clinical-gait frames.Ours and PBL register the lower limb tightly to the MMMC reference, while HSMR shows a visible knee/trunk offset.
  • Held-out BML-MoVi: Figure S2 compares the same methods on held-out BML-MoVi frames with frontal, unoccluded views.Ours, Ours+IK, and PBL closely register limb segments to the optical-marker fit, while HSMR’s residual offset persists.

A.3 HSMR knee-flexion (crouch) bias

HSMR exhibits a persistent knee-flexion bias in clinical gait, but bias correction removes the constant offset rather than the remaining waveform-shape error.

  • A.3 HSMR knee-flexion (crouch) bias: A +30.4° pooled knee offset accounts for much of HSMR’s error, so headline comparisons remove it before scoring.The offset is pooled across all trials for the method and joint.
  • A.3 HSMR knee-flexion (crouch) bias: After subtracting the offset, HSMR’s knee waveform remains misaligned because a constant correction cannot repair its per-frame trajectory.The reference knee reaches near-full extension during stance, whereas HSMR maintains a crouched posture.

A.4 Why the MHR head is kept frozen: deceptive in-distribution metrics

Fine-tuning the MHR head improved in-distribution metrics while causing catastrophic off-axis mesh collapse, so the MHR head remains frozen.

  • A.4 Why the MHR head is kept frozen: deceptive in-distribution metrics: Every in-distribution training curve improved monotonically, yet the recovered mesh collapsed for off-axis viewpoints.The affected curves tracked online MHR keypoint MPJPE, biomechanical marker RMS, and inline-IK residual.
  • A.4 Why the MHR head is kept frozen: deceptive in-distribution metrics: The fine-tuned mesh appeared plausible frontally but folded and twisted when viewed off-axis.Thus, in-distribution metrics provided no warning of the off-axis failure.

A.5 What the biomechanical head adds over IK alone

The biomechanical head provides a consistent image-conditioned alternative to cold-start IK while preserving the frozen foundation-model trunk and supporting efficient feed-forward inference.

  • A.5 What the biomechanical head adds over IK alone: A feed-forward head avoids bypassing the learned prediction with cold-start Levenberg–Marquardt IK at inference.The comparison uses the identical solver on frozen MHR output, initialized from the neutral pose.
  • A.5 What the biomechanical head adds over IK alone: Cold-started IK produces substantially more outlier frames than the head-based variant, despite similar performance on typical frames.Free marker offsets make inverse kinematics computationally demanding and prone to local minima.
  • A.5 What the biomechanical head adds over IK alone: Bias-corrected gait waveforms show that PBL and B3D track the reference most closely across the illustrated clinical trials.The correction removes each method’s pooled per-joint offset, isolating waveform shape.
  • A.5 What the biomechanical head adds over IK alone: Fine-tuning the MHR head improved in-distribution curves but caused catastrophic off-axis mesh degradation, motivating the frozen-head design.The off-axis failure was not revealed by the in-distribution metrics.
  • A.5 What the biomechanical head adds over IK alone: The study compares feed-forward and optimization methods using different inference sampling and frame-cap constraints, which are not matched across methods.Feed-forward methods are subsampled, whereas PBL and OpenCap run at full frame rate by design.
  • A.5 What the biomechanical head adds over IK alone: The JAX/Equinox port adds only +0.08 mm MPJPE and +0.13 mm PA-MPJPE relative to the PyTorch teacher before distillation.The reported disagreement is two orders of magnitude below the biomechanical errors in the main comparison.
  • A.5 What the biomechanical head adds over IK alone: The biomechanical head is a lightweight approximately 13.6 M-parameter module that remaps frozen MHR predictions into MuJoCo coordinates.It reads a single 1024-d pose token and uses MHR conditioning to form a 2048-d input.
Loading 2608.29928v1…