Source-linked AI summary
Cross-Model Distillation of a Human-Pose Foundation Model from Unannotated Infant Video for Markerless 3D Pose Estimation
R. James Cotton, Divya Joshi, Colleen Peyton
TL;DR
Infant movement assessment is clinically valuable but difficult to scale, and existing pose models split 2D fidelity from direct 3D recovery. This paper distills a frozen Sapiens 2 teacher into SAM 3D Body using unlabeled infant video and differentiable rendering. On held-out infants, the adapted model improves agreement with the teacher and reduces Procrustes-aligned 3D error.
Problem
Clinical infant movement assessment is time-consuming and variable, while adult-trained pose models do not jointly provide strong 2D accuracy and direct 3D recovery for infants.
Method
The method distills dense Sapiens 2 pseudo-labels into SAM 3D Body using unlabeled infant video and a differentiable render-in-the-loop objective.
Results
Body PCK@10px improves from 0.216 to 0.418, face PCK@10px from 0.219 to 0.422, and PA-MPJPE from 25.5 to 22.2 mm.
Takeaways & Limitations
Cross-model distillation combines Sapiens 2D fidelity with SAM 3D Body’s 3D kinematic output using unannotated infant video alone.
Takeaways & Limitations
The mesh remains mismatched to infant shape, especially at the head, because MHR’s shape space was designed for adults.
Abstract
from arXiv · showhide
Spontaneous movement is one of the earliest windows onto an infant's neuromotor health, and structured clinical instruments that score it are validated early predictors of cerebral-palsy risk. However, they require specially trained raters, are time-consuming, and carry inter-rater variability. This motivates automated, video-based markerless assessment, especially as marker-based motion capture is impractical in infants. Yet the foundation models that make markerless capture possible are trained almost entirely on adults: our recent multi-view infant study found that no single model is jointly best, with strong 2D keypoint accuracy and direct 3D body recovery split across different models. While that study identifies this trade-off, it does not resolve it. Here, we perform cross-model distillation from the Sapiens 2 pose model into the SAM 3D Body model, using unannotated infant video alone. A frozen teacher supplies dense pseudo-labels, and a differentiable renderer aligns the predicted mesh to them in the training loop. On eleven held-out infants (18 sessions, 173 recordings) under our prior study's multi-view protocol, fine-tuning improves same-view 2D keypoint agreement with the Sapiens reference (median body percentage of correct keypoints @ 10px 0.22 -> 0.42, face 0.22 -> 0.42) and Procrustes-aligned mean per joint 3D position error (25.5 -> 22.2 mm). This demonstrates how cross-model distillation improves SAM 3D Body model performance on infants.
1 Introduction
Infant movement can reveal early neuromotor impairment, but current clinical assessment is labor-intensive and marker-based capture is impractical. Because existing foundation models trade off 2D accuracy against direct 3D recovery, this work distills Sapiens 2 into SAM 3D Body using unlabeled infant video.
- Motivation: Infant spontaneous movement is an early observable sign of motor impairment and supports structured assessments predicting later motor outcomes.Examples include the Prechtl General Movements Assessment, the Test of Infant Motor Performance, and the Baby Observational Selective Control AppRaisal.
- Motivation: Clinical observational assessments require specially trained raters, take time, and exhibit inter-rater variability, motivating automated video-based assessment.
- Motivation: Markerless motion capture is non-invasive, scalable, and feasible for infants, whereas affixing markers is impractical and often contraindicated.
- Problem: Infant pose estimation is constrained by adult-focused training data, scarce labeled infant poses, and the lack of foundation models predicting infant-specific shape parameters.Existing infant-specific shape spaces such as SMIL and Anny do not provide image-to-parameter foundation models.
- Problem: Sapiens achieved the best 2D multi-view consistency without direct 3D estimates, while SAM 3D Body produced reasonable 3D kinematics with less geometrically consistent keypoints.
- Contribution: The paper distills Sapiens 2 pseudo-labels into SAM 3D Body using unlabeled infant video, with differentiable rendering aligning the predicted mesh to keypoints, depth, normals, and silhouettes.Held-out infants show improvements in both 2D agreement and Procrustes-aligned 3D error.
2 Methods
The method adapts SAM 3D Body through render-in-the-loop distillation from a frozen Sapiens 2 teacher on unlabeled infant recordings. Only a small pose head is optimized, while evaluation uses held-out multi-camera recordings and model-derived 2D and 3D references.
- Teacher and student: A shared per-frame infant crop is processed by a frozen Sapiens 2 teacher and SAM 3D Body student during training.The teacher supplies dense pseudo-labels, while the student predicts an MHR body whose rendered outputs are compared with those labels.
- Teacher and student: SAM 3D Body recovers MHR pose, orientation, hand, face, identity-shape, and skeletal-scale parameters from a single top-down crop.
- Trainable surface: Only the pose head projection network is optimized; the backbone, decoder, and forward kinematics remain fixed, while direct shape and scale readout gradients are masked.The features feeding the masked rows can still adapt, so predicted shape and scale are not held constant.
- Render-in-the-loop objective: The differentiable renderer compares predicted depth, normals, silhouette, and keypoints with teacher channels so pixel-space losses backpropagate into the pose head.Depth and normals use differentiable rasterization, while a soft silhouette rasterizer provides smooth outline gradients.
- Render-in-the-loop objective: The composite loss combines keypoint, normal, depth, silhouette, and regularization terms, with keypoint, normal, and depth terms active and silhouette lightly weighted.Keypoint supervision is confidence-gated and down-weights dense facial landmarks by an order of magnitude relative to major joints and head anchors.
- Data: The unlabeled monocular training cohort contains 543 recordings from 311 infants, while validation infants are excluded from the corpus to prevent leakage.
- Evaluation: Validation uses eleven infants recorded with eight synchronized RGB cameras at 29 frames per second under the prior multi-view protocol.Sessions contain 8–10 trials of roughly one minute, with calibrated camera geometry supporting multi-view triangulation.
3 Results
On 11 held-out infants, distillation improved SAM-3D-Body’s agreement with Sapiens and its aligned 3D error, while gains were generally consistent across infants. The fine-tuned model also improved benchmark reprojection and geometric consistency, but retained adult-like shape artifacts and relied mainly on keypoint losses.
- Qualitative results: Fine-tuning consistently reduced 2D distance to Sapiens keypoints in four qualitative held-out examples.The fine-tuned meshes remained affected by adult-proportioned faces and overly narrow torsos.
- Quantitative results: 0.216 → 0.418 body PCK@10px and 0.219 → 0.422 face PCK@10px approximately doubled after distillation.These teacher-agreement medians were significant under a paired Wilcoxon signed-rank test (p ≤0.005).
- Quantitative results: 25.5 →22.2 mm PA-MPJPE improved over the primary 11-joint core-body set.The core set excludes hips because occlusions make their triangulated reference unreliable.
- Quantitative results: 39.8 →36.7 px reprojection error and 0.31 →0.36 GC@10 improved over the SAM-3D-Body base.Off-the-shelf Sapiens remained better on these 2D metrics, with reprojection error 22.8 px and GC@10 0.82.
- Per-infant consistency: The two PCK panels improved for all 11 infants, while pixel-error and PA-MPJPE panels had one to two regressions each.Figure 4 shows per-infant changes with cohort means and ±1 SEM.
- Shape–scale regularization: Training shape and scale readout rows did not improve accuracy, and removing the shape–scale prior inflated body proportions, especially the head.The masked model was best or tied for best on every metric in the reported sweep.
- Training loss composition: The keypoint terms accounted for 98.1% of the objective, while normals, silhouette, and depth together contributed 1.86%.The dense render terms contributed little and did not converge.
4 Discussion
Cross-model distillation combines Sapiens 2's 2D fidelity with SAM 3D Body's 3D kinematics using unannotated infant video. The resulting student improves 2D agreement and scale-invariant 3D accuracy, while adult-oriented shape modeling and limited evaluation contexts remain important boundaries.
- Contribution: Sapiens 2's 2D fidelity and SAM 3D Body's 3D kinematics can be combined through label-free cross-model distillation.The method uses a render-in-the-loop objective on unannotated infant video, avoiding manual annotation and marker-based ground truth.
- Results: Fine-tuning significantly improved 2D agreement with the Sapiens reference alongside the reduction in scale-invariant 3D error.This narrows the trade-off identified in the prior study between precise 2D estimates and direct 3D kinematic recovery.
- Implications: Whole-body 3D kinematics could support viewpoint-invariant and anatomically interpretable representations for future video-based assessment of early motor development.The paper notes that the accuracy needed to classify pathological movements remains unknown.
- Results: 25.5 to 22.2 mm: Procrustes-aligned 3D error decreased by about 10% on held-out infant recordings.The evaluation included eleven held-out infants and 173 recordings under a multi-view protocol.
- Limitations: The mesh still differs from infant shape, particularly at the head, because MHR's shape space was designed for adults.The authors identify infant-oriented shape modeling, such as ANNY, and further hyperparameter tuning as future directions.
- Limitations: The evaluation uses multi-view triangulation as a high-quality but imperfect reference because infant ground-truth annotations and marker-based capture are impractical or contraindicated.Accuracy estimates may therefore contain reference error, especially at frequently occluded joints such as the hips.
A.1 Shape/scale constraint: does training the readout help?
Training the shape and scale readout does not improve held-out accuracy: the masked model is best or tied on every metric, while unconstrained fitting produces implausible deformations. The reported masked model therefore preserves cleaner meshes, although it does not recover true infant-specific body shape.
- A.1 Shape/scale constraint: does training the readout help?: The masked model is best on every held-out metric, or tied with λ_ss = 0.005 on geometric consistency.All four runs use final checkpoints and the same evaluation harness on eleven held-out infants.
- A.1 Shape/scale constraint: does training the readout help?: Training the shape and scale rows does not improve accuracy despite allowing the mesh to fit teacher-derived pixel losses more closely.With λ_ss = 0, the run reaches the tightest in-loop teacher-fidelity fit, but this flexibility is spent deforming shape rather than improving 3D joint accuracy.
- A.1 Shape/scale constraint: does training the readout help?: The unregularized run enlarges and rounds the head, narrows the forehead, and produces anatomically implausible shape changes.Its torso becomes somewhat less narrow, moving toward infant proportions, while the head deformation remains implausible.
- A.1 Shape/scale constraint: does training the readout help?: The strong prior λ_ss = 0.05 is worst, while the light prior λ_ss = 0.005 is the best trained-readout run but trails the masked model on every metric except geometric consistency.The light prior matches the masked model on geometric consistency.
- A.1 Shape/scale constraint: does training the readout help?: Masking the readout keeps meshes clean but does not reach true infant-specific body shape, motivating an infant-aware shape basis such as Anny.This limitation is not fully captured by joint-position metrics alone.