Source-linked AI summary
Sapiens: Foundation for Human Vision Models
Rawal Khirodkar, Timur Bagautdinov, Julieta Martinez, Su Zhaoen, Austin James, Peter Selednik, Stuart Anderson, Shunsuke Saito
TL;DR
Human-centric vision needs models that generalize across fundamental tasks despite difficult in-the-wild annotation and task-specific systems. Sapiens uses large-scale self-supervised pretraining on human images, then adapts a shared architecture with task-specific heads and high-quality or synthetic labels. The resulting models support high-resolution inference, generalize broadly, and achieve state-of-the-art performance across four human-centric benchmarks.
Problem
Robust in-the-wild estimation of pose, body parts, depth, and surface normals remains difficult, while accurate annotations are hard to scale.
Method
Sapiens pretrains vision transformers with masked autoencoding on large-scale human images, then fine-tunes a consistent encoder-decoder architecture for individual tasks.
Results
Sapiens achieves state-of-the-art performance across 2D pose, body-part segmentation, depth, and normal estimation benchmarks.
Takeaways & Limitations
Large-scale human-image pretraining combined with limited high-quality or synthetic labels supports robust in-the-wild generalization across human-centric tasks.
Takeaways & Limitations
Complex or rare poses, crowding, and severe occlusion remain challenging for the models.
Abstract
from arXiv · showhide
We present Sapiens, a family of models for four fundamental human-centric vision tasks -- 2D pose estimation, body-part segmentation, depth estimation, and surface normal prediction. Our models natively support 1K high-resolution inference and are extremely easy to adapt for individual tasks by simply fine-tuning models pretrained on over 300 million in-the-wild human images. We observe that, given the same computational budget, self-supervised pretraining on a curated dataset of human images significantly boosts the performance for a diverse set of human-centric tasks. The resulting models exhibit remarkable generalization to in-the-wild data, even when labeled data is scarce or entirely synthetic. Our simple model design also brings scalability -- model performance across tasks improves as we scale the number of parameters from 0.3 to 2 billion. Sapiens consistently surpasses existing baselines across various human-centric benchmarks. We achieve significant improvements over the prior state-of-the-art on Humans-5K (pose) by 7.6 mAP, Humans-2K (part-seg) by 17.1 mIoU, Hi4D (depth) by 22.4% relative RMSE, and THuman2 (normal) by 53.5% relative angular error. Project page: https://about.meta.com/realitylabs/codecavatars/sapiens.
1. Introduction
Sapiens develops unified, high-fidelity human-centric vision models that generalize across tasks and in-the-wild conditions. The approach combines large-scale human-image pretraining with lightweight task adaptation and high-quality annotations.
- Motivation: The framework addresses 2D pose, body-part segmentation, depth, and surface normal estimation within a unified model family.It is intended to replace complicated task-specific systems and support human-centric applications in the wild.
- Motivation: Sapiens targets generalization, broad applicability, and high fidelity for human-centric vision models.These criteria correspond to robustness across conditions, versatility across tasks, and precise high-resolution outputs.
- Approach: Sapiens combines large-scale human-image pretraining with pretrain-then-finetune adaptation using a consistent encoder-decoder architecture.The pretrained encoder is paired with a lightweight task-specific decoder, and both are finetuned end-to-end.
- Approach: High-quality supervision includes 308 whole-body keypoints, 28 segmentation classes, and synthetic data for depth and normal estimation.Multi-view capture supports pose and segmentation annotations, while detailed scans provide human-centric synthetic data for depth and normals.
- Results: Domain-specific large-scale pretraining with limited, high-quality annotations produces robust in-the-wild generalization without costly, diverse annotation collection.The paper presents this combination as an effective strategy for precise discriminative models in real-world scenarios.
- Results: The first 1K-resolution model natively supports high-fidelity inference for human-centric tasks and achieves state-of-the-art benchmark performance.The contribution spans pose, body-part segmentation, depth, and normal estimation.
2. Related Work
Prior work established large-scale pretraining and progress on human vision tasks, but unconstrained settings still motivate versatile models that generalize in the wild. Vision architectures have also lagged behind language models in scale.
- Pretraining at Scale: Large-scale pretraining has become standard in language modeling and is increasingly adopted in computer vision as larger datasets become available.Examples include LAION-5B, Instagram-3.5B, JFT-300M, LVD-142M, Visual Genome, and YFCC100M.
- Human Vision Tasks: Human digitization has advanced in controlled or studio environments, while extending robust methods to unconstrained settings remains challenging.This motivates models for essential human vision tasks that generalize to natural images.
- Human Vision Tasks: Versatile models for keypoint estimation, body-part segmentation, and surface normal prediction are important for human vision in natural settings.The related-work discussion frames these tasks as essential components of in-the-wild human understanding.
- Scaling Architectures: Vision Transformers have not yet been scaled as successfully as language models, despite sharing a similar architecture.The discussion contrasts publicly accessible language models exceeding 100B parameters with comparatively smaller vision efforts.
3. Method
Sapiens combines large-scale human-image pretraining with a consistent encoder-decoder design and task-specific heads for four human-centric vision tasks. The method uses curated real and synthetic data, high-resolution inputs, and fine-grained annotations to support detailed prediction.
- 3.1. Humans-300M Dataset: Sapiens pretraining uses approximately 1 billion filtered in-the-wild human images, retaining detected people above specified confidence and size thresholds.The dataset discards watermarks, text, artistic depictions, and unnatural elements before person-detection filtering.
- 3.2. Pretraining: MAE pretraining reconstructs complete human images from randomly masked patches, using both single- and multi-human images.The masking ratio is fixed during training, and Figure 3 shows reconstruction on unseen human images.
- 3.3. 2D Pose Estimation: The pose transformer predicts K keypoint heatmaps from a resized bounding-box image and is fine-tuned with a pretrained encoder and randomly initialized decoder.The model supports skeletons including 17, 133, and 308 keypoints, with 1 million high-resolution pose annotations for the detailed format.
- 3.4. Body-Part Segmentation: The segmentation model predicts per-pixel class-probability maps using the same encoder-decoder initialization scheme as pose estimation.Fine-tuning covers 20-class and 28-class vocabularies, including detailed limb, lip, teeth, and tongue categories.
- 3.5. Depth Estimation: Depth estimation uses a segmentation-style architecture with a one-channel regression decoder, normalized relative depths, and synthetic supervision from 500,000 rendered images.The synthetic data uses 600 high-resolution human scans, randomized HDRI backgrounds, and varied virtual-camera parameters.
- 3.6. Surface Normal Estimation: Surface-normal estimation uses a three-channel decoder for xyz normal components and trains on synthetic supervision with loss computed only over human pixels.The synthetic-data pipeline is shared with depth estimation.
4. Experiments
Experiments evaluate Sapiens across four human-centric tasks, showing strong benchmark performance, scalable improvements, broad generalization, and the value of human-centric pretraining.
- 2D Pose Estimation: Sapiens-2B reaches 61.1 AP on Humans-5K, improving 7.6 AP over prior art for whole-body pose estimation.
- Depth Estimation: Sapiens-2B outperforms prior monocular depth estimators across single-human scales and multi-human scenarios, reducing RMSE by 20% versus Depth-Anything on Hi4D.The model is fine-tuned solely on synthetic data, and RMSE is the primary metric because δ1 poorly reflects subtle human-scene depth variations.
- Surface Normal Estimation: All Sapiens models outperform existing human-specific surface-normal estimators, with Sapiens-2B reaching approximately 12° mean error on THuman2.0 and Hi4D.Evaluation reports mean and median angular error plus thresholded pixel accuracy.
- Pretraining and Generalization: Human300M pretraining yields superior performance across pose, segmentation, depth, and normal metrics under identical training schedules and computational budgets.Increasing the diversity and size of human images is associated with improved downstream generalization without observed saturation.
- Pretraining and Generalization: Large-scale pretraining supports zero-shot generalization from limited single-human, third-person segmentation data to varied subject counts, ages, and egocentric views.The paper states that similar observations hold for the other tasks.
- Limitations: Complex or rare poses, crowding, and severe occlusion remain challenging, although augmentation and detect-and-crop strategies may mitigate these issues.
5. Conclusion
Sapiens combines curated human-image pretraining, scalable high-resolution vision transformers, and high-quality annotations to generalize across human-centric tasks. The authors position these models as potential building blocks for downstream applications, with future work extending them to 3D and multimodal data.
- Sapiens combines curated human-centric pretraining, scaled high-resolution and high-capacity vision transformers, and high-quality annotations.
- The models demonstrate strong generalization across a variety of human-centric tasks and could provide high-quality vision backbones for downstream work.
- The authors identify extending Sapiens to 3D and multimodal datasets as a potential direction for future work.