PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 3D Body: Robust Full-Body Human Mesh Recovery

Xitong Yang, Devansh Kukreja, Don Pinkus, Anushka Sagar, Taosha Fan, Jinhyung Park, Soyong Shin, Jinkun Cao, Jiawei Liu, Nicolas Ugrinovic, Matt Feiszli, Jitendra Malik, Piotr Dollar, Kris Kitani

arXiv:2602.15989v1cs.CV

TL;DR

Monocular full-body mesh recovery remains fragile under difficult poses, occlusion, viewpoints, and diverse appearances, while high-quality full-body supervision is limited. SAM 3D Body combines promptable body-and-hand decoding, MHR, and a diverse annotation engine, achieving state-of-the-art body and hand estimation, strong generalization, and a 5 : 1 visual-quality win rate. Its scope remains limited by separate-person processing and hand accuracy below specialized hand-only methods.

  • Problem

    Existing monocular HMR systems struggle with challenging poses, occlusion, uncommon viewpoints, unified body-hand-foot estimation, and high-quality diverse supervision.

  • Method

    3DB combines a promptable encoder–decoder with separate body and hand decoders, MHR, and a VLM-driven data engine with multi-stage high-quality annotation.

  • Results

    3DB achieves state-of-the-art body and hand pose estimation, outperforms prior HMR methods with stronger unseen-dataset generalization, and wins visual-quality comparisons 5 : 1 among 7, 800 participants.

  • Takeaways & Limitations

    Prompted, editable full-body recovery extends robust single-image estimation across difficult in-the-wild conditions and supports nuanced evaluation by pose and appearance categories.

  • Takeaways & Limitations

    3DB processes individuals separately and does not model multi-person or human-object interactions; its hand accuracy remains below specialized hand-only methods.

Abstract

from arXiv · show

We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.

1 Introduction

SAM 3D Body addresses the limited robustness of monocular full-body mesh recovery in challenging in-the-wild conditions. It combines a promptable full-body model, the MHR representation, and diverse high-quality supervision to improve body and hand estimation.

  • Existing HMR methods remain unreliable on challenging poses, severe occlusion, uncommon viewpoints, and unified estimation of body, hands, and feet.
  • High-quality, diverse 3D supervision is difficult and costly because existing datasets trade off laboratory diversity against pseudo-label mesh quality.
  • 3DB introduces a promptable encoder–decoder architecture that accepts optional 2D keypoints, masks, or camera information for controllable pose estimation.
  • 3DB uses separate body and hand decoders with MHR, which decouples skeletal structure from body shape for richer control and interpretability.
  • 7 million images receive high-quality annotation, while a VLM-based data engine mines rare poses, difficult viewpoints, and varied appearances.
  • 3DB achieves state-of-the-art performance across body and hand pose estimation, with a significant 5 : 1 visual-quality win rate in a study of 7, 800 participants.

2 Related Work

Prior work uses parametric human-body models, full-body or part-specific HMR systems, promptable inference, and increasingly sophisticated annotation pipelines. SAM 3D Body combines these directions with MHR, prompt tokens, and scalable challenging-case mining.

  • SMPL parameterizes pose and shape, while SMPL-X adds articulated hands and faces; MHR instead decouples skeletal structure and body shape.
  • 3DB belongs to the full-body HMR paradigm that estimates body, hands, and feet, unlike earlier body-only or hand-specific methods.
  • Promptable inference uses user- or system-provided inputs such as 2D keypoints or masks to guide predictions.
  • HMR annotation pipelines address noisy monocular pseudo-ground truth by combining multi-view or synthetic supervision with dense keypoints, parametric priors, and robust optimization.

3 SAM 3D Body Model Architecture

The architecture encodes cropped human images and optional prompts, then decodes combined query tokens into MHR and camera predictions. Separate body and hand decoding supports automatic or user-guided full-body recovery with enhanced hand estimates.

  • A normalized human crop passes through a vision backbone to produce dense image features, with optional hand crops producing hand-specific feature maps.
  • 2D keypoint prompts become positional-and-learned embedding tokens, while segmentation masks are embedded by convolutions and added to image embeddings.
  • The body decoder predicts the full-body rig, while an optional hand decoder uses hand crops to provide enhanced hand pose results.
  • MHR and camera tokens, keypoint prompts, auxiliary keypoint tokens, and optional hand-position tokens form the decoder query set.
  • The body decoder can operate automatically or with prompts, fusing query tokens and image features through cross-attention for editable mesh recovery.
  • The first body-decoder output is mapped by an MLP to MHR pose, shape, camera-pose, and skeleton parameters, with hand outputs optionally merged into the body prediction.

4 Model Training and Inference

3DB uses multi-task, prompt-aware training to supervise body and hand predictions, with inference that combines dedicated body and hand decoding for full-body output.

  • Model Training: 3DB trains with task-specific losses for prediction heads and anatomical structures, using empirically set loss weights and warm-up scheduling for selected terms.Training randomly samples prompts in multiple rounds to simulate interactive inference.
  • Model Training: 2D and 3D keypoint supervision uses uncertainty-weighted L1 losses, anatomical normalization, availability-based hand weighting, and extra weight for user-provided keypoints.The loss is computed in cropped image spaces for 2D keypoints.
  • Model Training: MHR pose and shape use L2 regression with joint-limit penalties, while hand detection uses GIoU and L1 losses with predicted box uncertainty.The hand decoder is turned off for hand-occluded samples during inference.
  • Full-body Inference: During inference, 3DB uses the body decoder by default and merges hand-decoder output when hands are detected to improve hand pose estimation.The dedicated hand head supports a free-moving wrist and uses hand-specific training data.

5 Data Engine for Diversity

The data engine improves dataset diversity by routing difficult images for annotation through VLM-driven mining and iterative failure analysis.

  • Data Engine for Diversity: An automated data engine selectively routes difficult images for annotation to increase diversity while enabling scalable dataset curation.The motivation is that large video collections can contain similar poses, appearances, imaging conditions, and backgrounds.
  • Data Engine for Diversity: A VLM-driven strategy identifies occlusion, unusual poses, interaction, extreme scale, low visibility, and hand-body coordination as challenging scenarios.The strategy replaces simple heuristics or random sampling with automatically generated and updated mining rules.
  • Data Engine for Diversity: Mining rules adapt through semi-manual failure analysis that evaluates 3DB, visualizes challenging images using keypoint error, and converts annotations into VLM text prompts.Newly selected images are routed for manual annotation.
  • Data Engine for Diversity: The annotation tool compares dense thin keypoints with sparse thick keypoints for pseudo annotation.The supplied figure caption identifies the left panel as a 2D-keypoint annotation GUI.

6 Data Annotation and Mesh Fitting

3DB’s annotation pipeline combines manual keypoints, dense detection, optimization, and geometric or temporal constraints to produce mesh supervision from single-view and multi-view data.

  • Data Annotation and Mesh Fitting: The multi-stage pipeline produces accurate 3D mesh pseudo-ground truth using manual 2D keypoints, sparse and dense detection, geometric constraints, and temporal constraints.It is applied to in-the-wild single-image datasets and several multi-view datasets.
  • Manual Annotation: Annotators review and correct initial 2D joint estimates and assign visibility labels, marking joints as not visible when occlusion or blur prevents accurate placement.The rubric gives substantial occlusion, such as 50%, as an example of non-visibility.
  • Multi-View Mesh Fitting: The mesh-fitting examples include multi-view fitting and scan-based fitting, with body and hand data used in the fitting process.The cited figure caption identifies EgoExo4D and Re:Interhand as the respective sources.
  • Single-Image Mesh Fitting: Single-image MHR fitting refines initial parameters against 595 dense 2D keypoints using weighted task losses, including reprojection, initialization regularization, and anatomical priors.The priors constrain pose and shape toward anatomical plausibility and the fitted meshes become training supervision.
  • Single-Image Mesh Fitting: The dense keypoint detector uses sparse manual-keypoint guidance and iterative training that projects fitted MHR meshes back to dense keypoints.The iterative scheme is applied twice after training on 3D and in-the-wild datasets.

7 Training Datasets

3DB is trained on single-view, multi-view, synthetic, and hand-focused datasets spanning general body pose, interactions, and in-the-wild conditions.

  • Training Datasets: The training mixture combines single-view, multi-view, and synthetic datasets to cover general body pose, hands, interactions, and in-the-wild conditions.The stated aim is to ensure training-data quality, quantity, and diversity.
  • Training Datasets: Single-view in-the-wild datasets include AIChallenger, MS COCO, MPII, 3DPW, and a subset of SA-1B.These datasets provide unconstrained environments with diverse appearance, pose, and scene conditions.
  • Training Datasets: Multi-view datasets provide geometric consistency through Ego-Exo4D, Harmony4D, EgoHumans, InterHand2.6M, DexYCB, and Goliath.They are used for more reliable annotations.
  • Training Datasets: A photorealistic synthetic Goliath extension supplies millions of frames with ground-truth MHR parameters across diverse identities, clothing, and contexts.Synthetic data provides accurate supervision complementary to real-world diversity.
  • Training Datasets: Hand datasets such as Re:Interhand train both decoders, with wrist-truncated samples used specifically for the hand decoder.The hand datasets are marked with a star in Table 1.

8 Evaluation

Across standard, out-of-domain, categorical, qualitative, and user-preference evaluations, 3DB consistently outperforms prior HMR methods and generalizes to challenging conditions. It also delivers strong hand and full-body recovery, including difficult poses, truncation, occlusion, and fine details.

  • Common datasets: 3DB outperforms all other single-image methods on five standard benchmarks and is competitive with video-based approaches using temporal information.The model also exceeds NLF on all datasets in 3D metrics except RICH, where NLF was trained.
  • Common datasets: 3DB shows superior results on the out-of-domain EMDB and RICH datasets, indicating better generalization than previous state-of-the-art methods.The comparison uses five standard datasets and reports stronger performance on datasets excluded from training.
  • New datasets: 3DB’s leave-one-out model shows strong generalization across five new datasets, while prior methods exhibit significant performance drops and dataset-specific biases.The new testbed contains 38.6K images spanning skilled activities, multi-human interaction, studio motion, synthetic camera variation, and SA1B-Hard.
  • Hand pose estimation: 3DB achieves strong hand shape and pose accuracy despite the usual gap between full-body and hand-only pose estimation methods.The model combines hand and body data, uses a separate hand decoder, and aligns body wrists with hand predictions through prompting.
  • Categorical performance: 3DB outperforms all baselines across 24 SA1B-Hard categories and performs particularly well on truncation, inverted-body, and split-limb cases.For 3D categorical evaluation, it also leads in very hard pose categories and improves over baselines for severe truncation and top-down viewpoints.
  • Qualitative and preference results: 3DB consistently produces more accurate body pose and shape recovery than six state-of-the-art methods, especially for limbs and hands under complex poses, shapes, and occlusions.In the human preference study, 3DB achieves an 83.8% win rate against the strongest baseline, NLF.

9 Conclusion

The conclusion presents 3DB as a robust body-and-hand HMR model built from MHR, promptable encoder–decoder components, and a high-quality, diverse supervision pipeline. These design choices support generalization beyond curated benchmarks and strengthen hand-pose estimation.

  • Conclusion: 3DB combines the Momentum Human Rig representation, a flexible encoder–decoder architecture, and optional 2D keypoint or mask prompts for guided inference.The model is designed for body and hand mesh recovery.
  • Conclusion: A separate hand decoder uses hand crops to enhance hand pose estimation and make 3DB comparable to state-of-the-art hand pose methods.The conclusion connects this decoder to the model’s full-body inference design.
  • Conclusion: The supervision pipeline combines multi-view capture, synthetic sources, and a scalable data engine that mines and annotates challenging samples instead of relying on noisy monocular pseudo-ground truth.The stated goal is cleaner and more diverse training signals.
  • Conclusion: 3DB’s data and model design supports generalization beyond curated benchmarks.This conclusion is stated as the consequence of the supervision strategy.

B Evaluating 3DB Prompt Following

3DB follows 2D keypoint and mask prompts for controllable pose estimation, with performance improving as reliable keypoint prompts are added. Prompting is especially useful for wrist and hand alignment and for distinguishing people during close interaction.

  • 2DKeypointPrompt: 3DB’s 2D keypoint prompting improves both 2D and 3D performance as more prompts are provided.The inference ablation selects the keypoint with the largest error for prompting.
  • 2DKeypointPrompt: 3DB is robust to small keypoint inaccuracies below a noise scale of 0.05, but performance degrades when prompts become more inaccurate.At larger noise levels, the model tends to follow incorrect keypoint prompts.
  • 2DKeypointPrompt: Keypoint prompting improves wrist and hand-joint alignment compared with inference without keypoint prompts.The full-body pipeline uses prompting to improve hand pose estimation quality.
  • Mask conditioning: Mask conditioning helps 3DB disambiguate people during close interactions with substantial occlusion when bounding boxes are insufficient.Segmentation masks identify the corresponding person for prediction.
  • Mask conditioning: On SA1B-Hard, mask conditioning improves the Multi-person subset by +4.4% versus +0.9% on the overall dataset.The larger subset gain indicates the particular value of mask prompts in multi-person scenarios.

C Limitations

3DB has limitations in multi-person interaction, specialized hand accuracy, training data coverage, and modeling body shapes across all age groups.

  • Multi-person and interaction limitations: 3DB processes each individual separately, so it does not account for multi-person or human-object interactions.The authors identify relative positions and physical interactions as the affected capabilities.
  • Hand estimation: 3DB improves hand pose estimation within full-body recovery but does not surpass specialized hand-only methods.The authors attribute this boundary partly to limited high-quality full-body training data.
  • Hand estimation: The body decoder’s hand estimation remains suboptimal because high-quality full-body training data are limited.The proposed next step is incorporating more diverse full-body data into training.
  • Age coverage: 3DB and MHR do not model human body shapes equally well across all age groups, which can affect children’s pose and shape estimates.The paper specifically identifies children as a group for which outputs may be suboptimal.
Loading 2602.15989v1…