Source-linked AI summary
UT-HMR: Uncertainty-Guided Temporal Modeling for Robust 3D Human Mesh Recovery
Zhibo Wang, Sirisha Rambhatla, Yuhao Chen, Maria Koshkina, David Clausi
TL;DR
Weak observations make monocular one-stage HMR ambiguous because different 3D poses can produce similar 2D projections. UT-HMR models correlated pose uncertainty and adds neighboring-frame evidence before decoding; it reports state-of-the-art results on two sports benchmarks and remains competitive on 3DPW.
Problem
Monocular HMR is ambiguous under weak observations, especially for one-stage methods that recover meshes from full images without person crops or external 2D cues.
Source: §1 · p. 1, §3.2 · p. 3Method
UT-HMR predicts a low-rank-plus-diagonal Gaussian over full pose for joint-wise precision-weighted training and adds temporal early fusion before pose decoding.
Source: §1 · pp. 1–2Results
Up to 4.7%, 3.8%, and 5.6% lower MPJPE, PA-MPJPE, and PVE; state-of-the-art on both sports benchmarks and competitive on 3DPW.
Source: §1 · p. 2Takeaways & Limitations
Predicted covariance provides a per-person reliability signal that could weight tracking data association or down-weight unreliable joints in action analysis.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Monocular 3D human mesh recovery (HMR) remains unreliable under weak observations caused by long camera-to-person distances, occlusion, and motion blur. In challenging scenarios such as sports, these factors make the problem highly ambiguous, as different 3D poses may yield similar 2D projections. This ambiguity is especially severe for one-stage HMR, which recovers all meshes directly from the full image without person crops or external 2D cues. To handle this ambiguity, we present UT-HMR ( Uncertainty-guided Temporal HMR), to our knowledge the first one-stage HMR framework to capture correlated pose uncertainty across joints. Each person query predicts a Gaussian distribution over the full SMPL pose with a compact low-rank-plus-diagonal covariance, which is converted into joint-wise precision weights for heteroscedastic supervision while keeping inference deterministic. Temporal early fusion further injects neighboring-frame evidence before pose decoding, supplying the visual evidence needed to resolve the ambiguity that the uncertainty captures. With a lightweight ViT-B backbone, UT-HMR achieves state-of-the-art results on VIP-HockeyPose-v2 (ice hockey) and WorldPose (soccer), reducing MPJPE, PA-MPJPE, and PVE by up to 4.7%, 3.8%, and 5.6%, and remains competitive on 3DPW against methods with larger backbones and higher input resolutions.
1. Introduction
One-stage HMR is attractive for efficient multi-person processing but faces severe pose ambiguity under weak visual evidence. UT-HMR combines full-pose uncertainty modeling with temporal early fusion and reports state-of-the-art results on two sports benchmarks.
1. Introduction
- Monocular HMR is ill-posed, and ambiguity is especially severe in one-stage methods that estimate meshes from full images without cropped-person evidence.Different 3D poses can yield similar 2D projections.Source: §1 · p. 1
- The top two directions explain 39% of pose-residual variance, motivating a compact low-rank-plus-diagonal Gaussian for correlated and residual uncertainty.The residual analysis covers 24 joints × 6 pose dimensions.Source: Figure 1 · p. 1, §1 · p. 1
- One-stage HMR jointly processes people without person-level detection, supporting efficient multi-person inference in crowded scenes where per-person cropping is costly.Its single-pass, end-to-end design also keeps the pipeline simple and fast.Source: §1 · p. 1
- Prior uncertainty methods rely on crops, tracking, explicit 2D inputs, or multiple hypotheses, which do not directly transfer to joint one-stage estimation of all people.UT-HMR is presented as the first one-stage HMR framework to capture correlated pose uncertainty across joints.Source: §1 · pp. 1–2
- UT-HMR predicts a pose Gaussian from each person query and converts its covariance into joint-wise precision weights for training, while inference remains deterministic.Temporal early fusion injects neighboring-frame evidence before pose decoding, using full-image features rather than crop-based aggregation.Source: §1 · pp. 1–2
- UT-HMR achieves state-of-the-art performance on both sports benchmarks, ice hockey and soccer [12], reducing MPJPE, PA-MPJPE, and PVE by up to 4.7%, 3.8%, and 5.6% over the strongest baseline.It remains competitive on 3DPW against larger models with higher input resolutions.Source: §1 · p. 2
2. Related Work
Related work spans multi-person mesh recovery, video-based temporal modeling, and probabilistic pose estimation. UT-HMR builds on one-stage recovery while adding temporal context and deterministic uncertainty-guided training.
2.1. Multi-Person Human Mesh Recovery
- Two-stage methods crop detected people, often preserving image resolution but depending heavily on detector quality; one-stage methods estimate people jointly from the full image.One-stage approaches include map regression and transformer set prediction, and UT-HMR extends this paradigm with temporal context and uncertainty modeling.Source: §2.1 · p. 2
2.2. Video-Based Human Mesh Recovery
- Video-based HMR methods model temporal dynamics or long-range dependencies, while UT-HMR targets weak-observation projection ambiguity with lightweight early fusion before pose decoding.Its neighboring-frame information is incorporated while preserving an efficient one-stage multi-person framework.Source: §2.2 · pp. 2–3
2.3. Probabilistic Human Mesh Recovery
- Probabilistic HMR includes distributions and multiple-hypothesis methods, whereas UT-HMR keeps inference deterministic and uses low-rank-plus-diagonal Gaussian covariance for joint-wise precision-weighted training.Other approaches explicitly sample, rank, or aggregate hypotheses, or use uncertainty in 2D pose heatmaps.Source: §2.3 · p. 3
3. Method
UT-HMR combines temporal early fusion with full-pose uncertainty modeling in a one-stage mesh-recovery pipeline. It uses neighboring-frame evidence before pose decoding and trains a deterministic estimator with uncertainty-derived joint weights.
3.1. Preliminaries
- SMPL converts predicted pose and shape parameters into a mesh; a linear regressor then obtains 3D joint locations.The pose uses 6D joint rotations, converted to the axis-angle representation required by SMPL.Source: §3.1 · p. 3
3.2. Overall Pipeline
- A ViT-B encoder and DETR-style decoder produce person queries, while weak observations can leave plausible 2D projections paired with incorrect 3D poses.The queries regress pose and shape parameters that SMPL converts into meshes.Source: §3.2 · p. 3
3.3. Temporal Early Fusion
- Three-frame temporal early fusion concatenates neighboring-frame memories, letting center-frame person queries aggregate feature-level evidence before pose regression.Only the center frame is supervised; the sliding clip window produces frame-wise predictions without an explicit smoothness loss or motion model.Source: §3.3 · p. 4
3.4. Pose Uncertainty Modeling
- Each query predicts a Gaussian pose distribution with low-rank correlated and diagonal residual uncertainty, requiring 432 parameters at rank 2 versus 10,440 for full covariance over 144 pose dimensions.Source: §3.4 · p. 4
3.5. Model Training
- Training adds Gaussian negative log-likelihood for covariance and an auxiliary joint loss that weights more reliable joints while retaining uniform 3D supervision.The distribution loss explains residual pose uncertainty without directly updating the deterministic pose branch; the heteroscedastic weights are detached to prevent uncertainty inflation.Source: §3.5 · p. 5
4.1. Experimental Setup
The experiments fine-tune a SAT-HMR-initialized ViT-B/14 model on two sports datasets and 3DPW, using a three-frame input configuration and benchmark-specific training.
4.1.2. Implementation Details
- The model starts from a SAT-HMR checkpoint pretrained on six listed datasets and is fine-tuned on all three evaluation datasets.Fine-tuning uses 50 epochs on VIP-HockeyPose-v2 and WorldPose, and 20 epochs on 3DPW.Source: §4.1.1 · pp. 5–6, §4.1.2 · p. 6
4.1.1. Datasets and Metrics
- VIP-HockeyPose-v2 and WorldPose use sports data, while 3DPW assesses generalization beyond sports.VIP-HockeyPose-v2 samples frames from 24 NHL games; WorldPose uses 2022 FIFA World Cup footage and pseudo ground truth from calibrated-camera keypoint triangulation and SMPL optimization.Source: §4.1.1 · p. 6
- Evaluation reports MPJPE, PA-MPJPE, PVE, and per-frame inference time on a single NVIDIA RTX 6000 Ada GPU.PA-MPJPE is measured after Procrustes alignment; PVE measures average vertex error.Source: §4.1.1 · p. 6
4.1.2. Implementation Details
- The setup follows SAT-HMR with input resolution 644 and a ViT-B/14 backbone, encoding three frames independently for center-frame decoding.Center-frame queries cross-attend to concatenated token memories from the three frames.Source: §4.1.2 · p. 6
- Competing methods are fine-tuned on each benchmark’s training split, except DepthHMR’s 3DPW results, which are included as reported because its protocol is unspecified.DepthHMR code was unavailable.Source: §4.1.2 · p. 6
4.2. Overall Performance
UT-HMR improves on SAT-HMR across three benchmarks and leads tested models on both sports datasets. Qualitative examples show robustness to weak observations and generalization to in-the-wild scenes.
4.2.1. Quantitative Results
- UT-HMR improves over SAT-HMR across all three benchmarks with moderate overhead, leads tested models on all three metrics on both sports datasets, and remains competitive on 3DPW.On 3DPW, it remains competitive despite using a lighter backbone and lower input resolution than top-performing competitors.Source: §4.2.1 · p. 6
- The single-frame T=1 variant outperforms competing single-frame methods on both sports benchmarks and remains competitive on 3DPW against substantially larger backbones.Among multi-frame methods, three-frame UT-HMR also outperforms PSVT’s best nine-frame setting on 3DPW.Source: §4.2.1 · p. 6
4.2.2. Qualitative Results
- Figure 4 overlays UT-HMR reconstructions on input frames from VIP-HockeyPose-v2, WorldPose, and 3DPW.VIP-HockeyPose-v2 referees are excluded because ground-truth meshes are unavailable.Source: §4.2.2 · p. 6, Figure 4 · p. 7
- Across sports and general-scene examples, UT-HMR produces stable, plausible meshes and remains robust to small person scale, occlusion, and motion blur.The results also show generalization to in-the-wild 3DPW scenes.Source: §4.2.2 · p. 7
- Compared with SAT-HMR, UT-HMR reduces failure cases for unclear or occluded sports players, while visualized uncertainty is higher around limbs under occlusion or weak evidence.The paper presents this uncertainty pattern as suggesting that learned uncertainty captures meaningful pose ambiguity.Source: §4.2.2 · p. 7
4.3. Ablation Studies
On VIP-HockeyPose-v2, temporal early fusion and pose uncertainty modeling each improve the baseline, and their combination performs best. Further analyses support uncertainty-based joint weighting and identify settings used in the final model.
4.3.1. Effectiveness of Proposed Modifications
- Temporal early fusion and pose uncertainty modeling each improve the baseline, while their combination achieves the best performance on VIP-HockeyPose-v2.Table 2 compares the modifications against a SAT-HMR baseline fine-tuned on the same data.Source: §4.3.1 · p. 7, Table 2 · p. 8
- MPJPE, PA-MPJPE, and PVE rise monotonically with predicted uncertainty, supporting its use as a reliability ranking for joint reweighting.The analysis groups instances into four equal-size uncertainty percentiles using joint-level marginal variance averaged over each joint's six rotation dimensions.Source: §4.3.1 · p. 7, Figure 6 · p. 8
4.3.2. Hyperparameter Sensitivity
- A ±1 temporal window improves over a single frame, longer ranges add no gain and may degrade performance, and low-rank covariance beats diagonal-only while larger ranks hurt.The final model uses temporal range ±1 and covariance rank r = 2.Source: §4.3.2 · p. 8, Table 3 · p. 8
4.3.3. Effectiveness of Heteroscedastic Joint Loss
- Per-joint uncertainty weighting outperforms per-person weighting, and the final model uses per-joint weights.On VIP-HockeyPose-v2, uncertainty-based weighting improves over uniform or error-based weighting; up-weighting uncertain joints helps but remains inferior to down-weighting them.Source: §4.3.3 · p. 8, Table 4 · p. 8
5. Conclusion
UT-HMR combines temporal early fusion, low-rank-plus-diagonal Gaussian pose uncertainty, and covariance-based heteroscedastic joint supervision. It improves over baselines on three benchmarks, remains competitive with larger models despite a lighter backbone, and offers a low-cost reliability signal with potential uses in tracking and action analysis.
5. Conclusion
- UT-HMR combines temporal early fusion, low-rank-plus-diagonal Gaussian pose uncertainty modeling, and covariance-based heteroscedastic joint supervision.The conclusion reports experiments on VIP-HockeyPose-v2, WorldPose, and 3DPW.Source: §5 · p. 8
- Experiments on VIP-HockeyPose-v2, WorldPose, and 3DPW show consistent baseline improvements, while UT-HMR remains competitive with larger models despite its lighter backbone.Source: §5 · p. 8
- The predicted covariance provides a per-person reliability signal at negligible inference cost, with tracking and action-analysis applications left for future quantitative study.Source: §5 · p. 8