PaperlayerView exact revision on arXiv

Source-linked AI summary

Sapiens2

Rawal Khirodkar, He Wen, Julieta Martinez, Yuan Dong, Su Zhaoen, Shunsuke Saito

arXiv:2604.21681v1cs.CV

TL;DR

Human-centric vision needs models that generalize across tasks and imagery while preserving high-fidelity details. Sapiens2 combines masked reconstruction and contrastive learning with large-scale human data and high-resolution architectures, achieving state-of-the-art performance across benchmarks, including 82.3 mAP for dense 308-keypoint pose estimation.

  • Problem

    Masked reconstruction preserves visual details but often provides limited semantic understanding, motivating features that support both dense prediction and broader human-task generalization.

  • Method

    Sapiens2 pretrains high-resolution transformers on 1 billion human images using masked reconstruction plus contrastive objectives, then fine-tunes them with high-quality annotations across human tasks.

  • Results

    Sapiens2 achieves state-of-the-art performance across benchmarks and reaches 82.3 mAP for dense 308-keypoint pose predictions in-the-wild.

  • Takeaways & Limitations

    Sapiens2 provides a high-resolution foundation for human-centric vision spanning tasks from pose estimation to albedo recovery.

Abstract

from arXiv · show

We present Sapiens2, a model family of high-resolution transformers for human-centric vision focused on generalization, versatility, and high-fidelity outputs. Our model sizes range from 0.4 to 5 billion parameters, with native 1K resolution and hierarchical variants that support 4K. Sapiens2 substantially improves over its predecessor in both pretraining and post-training. First, to learn features that capture low-level details (for dense prediction) and high-level semantics (for zero-shot or few-label settings), we combine masked image reconstruction with self-distilled contrastive objectives. Our evaluations show that this unified pretraining objective is better suited for a wider range of downstream tasks. Second, along the data axis, we pretrain on a curated dataset of 1 billion high-quality human images and improve the quality and quantity of task annotations. Third, architecturally, we incorporate advances from frontier models that enable longer training schedules with improved stability. Our 4K models adopt windowed attention to reason over longer spatial context and are pretrained with 2K output resolution. Sapiens2 sets a new state-of-the-art and improves over the first generation on pose (+4 mAP), body-part segmentation (+24.3 mIoU), normal estimation (45.6% lower angular error) and extends to new tasks such as pointmap and albedo estimation. Code: https://github.com/facebookresearch/sapiens2

1 INTRODUCTION

SAPIENS2 advances human-centric vision across task coverage, image diversity, and prediction fidelity. It combines high-resolution transformer backbones, unified masked-reconstruction and contrastive pretraining, and large-scale human imagery to support generalizable dense prediction.

  • Any human image: 1B high-quality human images span diverse ages, ethnicities, backgrounds, and real-world conditions, with each image containing at least one prominent person.Pretraining uses no task labels and injects no human-specific priors.
  • Highest fidelity: 4K hierarchical backbones use windowed self-attention and decode dense predictions at 2K resolution across tasks.The 4K backbone is pretrained and post-trained for dense prediction, making higher-resolution processing tractable.
  • Contributions: SAPIENS2 is a 0.4B–5B-parameter transformer family pretrained on 1 billion high-quality human images for high-resolution dense prediction.The models support native 1K resolution and hierarchical 4K resolution.
  • Contributions: Masked reconstruction combined with contrastive objectives learns features that generalize in zero-shot human tasks while preserving fine details for dense prediction.This unifies semantic representation learning with reconstruction-based preservation of spatial detail.

2 RELATED WORK

Prior work spans self-supervised learning through masked image modeling and contrastive learning, human-centric models that incorporate structural priors, and rapidly scaling vision transformers compared across parameters, resolution, and data.

  • Self-Supervised Learning: Self-supervised learning at scale primarily follows masked image modeling or contrastive learning, with image representations remaining denser and more ambiguous than language representations.MIM is commonly viewed as compressing input tokens.
  • Human-Centric Vision Models: Human-centric vision models often outperform similarly sized general models on human-related tasks by incorporating body structure or geometric information.Examples include keypoint-guided masking, diffusion-based geometry estimation, and 3D skeleton supervision.
  • Human-Centric Vision Models: Unlike prior human-centric pretraining approaches, Sapiens2 injects no explicit human priors beyond the data during pretraining.The passage describes this as a prior-free inductive approach.
  • Vision Transformers at Scale: Vision transformer scaling is commonly characterized along three axes: parameters, resolution, and data.The largest cited Perception Encoder backbone has 2B parameters, uses 448 px resolution, and is trained on 5.4B samples.

3 PRETRAINING

Sapiens2 pretraining combines human-centric data curation with masked image reconstruction and student–teacher contrastive learning. The joint objective is designed to preserve low-level output fidelity while strengthening view-invariant semantic understanding.

  • Data curation: ∼4B images are filtered for human-centric content using bounding boxes, head pose, aesthetics, realism, CLIP features, and text-overlay detection.Images failing realism, quality, or other checks are removed.
  • Masked Image Modeling: Masked image modeling reconstructs all patches from visible tokens and learned mask tokens, with targets normalized before loss computation.The loss averages mean squared error over masked tokens and views.
  • Joint objective: Figure 4 combines masked reconstruction loss Lmae with global contrastive loss Lcl, associating reconstruction with low-level detail and contrastive learning with semantic understanding.The figure describes Lmae as supporting high-fidelity dense tasks and Lcl as improving semantic understanding across human images.
  • Contrastive Learning: The student–teacher contrastive scheme uses an EMA teacher and cross-view global↔global and global↔local positive pairs.Local↔local pairs are skipped, and the objective averages teacher-to-student cross-entropy over the selected pairs.
  • Joint objective: The final objective is L = LMAE + λLCL, combining human-centric low-level fidelity with view-invariant semantics.For the illustrated setup, V = 2 and the figure shows only the global contrastive objective for clarity.

4 MODEL ARCHITECTURE

Sapiens2’s backbone is redesigned to scale stably to 5B parameters and support 4K input resolution while retaining compatibility with sparse masked pretraining. The architecture combines block-specific attention, gated SwiGLU feed-forward layers, and QK-Norm for long-schedule stability.

  • The revised backbone scales to 5B parameters, increases input resolution from 1K to 4K, and remains compatible with sparse masked pretraining.
  • Mid-depth blocks use grouped-query attention, whereas early and late blocks use standard multi-head self-attention.This assigns different attention mechanisms to different backbone stages.
  • The feed-forward layers are replaced with gated SwiGLU-FFN variants, and QK-Norm is applied for long-schedule stability.QK-Norm normalizes queries and keys before attention.

5 POST-TRAINING

Sapiens2 post-trains a fixed pretrained backbone across five human-centric tasks using lightweight task-specific heads, broader supervision, and refined objectives. The task-specific designs support dense full-body pose, precise segmentation, 3D pointmaps, normals, and illumination-invariant albedo.

  • Post-training overview: Post-training fine-tunes lightweight task-specific heads for pose, body-part segmentation, depth, surface normals, and albedo while leaving the backbone unchanged.Supervision and task objectives are broadened relative to Khirodkar et al. (2024).
  • Pose Estimation: 308 keypoints define the full-body pose skeleton, including 243 facial and 40 hand keypoints, with 100K newly annotated in-the-wild high-resolution images added to capture-studio supervision.The top-down pose estimator predicts keypoint heatmaps.
  • Body-Part Segmentation: 29 segmentation classes add eyeglasses to the previous 28, using 20K in-the-wild labeled images and weighted cross-entropy plus Dice loss for sharper boundaries.The vocabulary targets precise localization of semantic human body parts.
  • Pointmap (Depth) Estimation: The depth task regresses focal-normalized per-pixel 3D pointmaps with a scalar scale head, using entirely synthetic supervision with higher-fidelity assets.The formulation addresses metric-scale ambiguity from unknown camera intrinsics.
  • Normal Estimation: Normal estimation predicts per-pixel unit normals for human pixels from high-fidelity synthetic assets, with multiple PixelShuffle layers enabling artifact-free upsampling.The decoder uses PixelShuffle layers for upsampling.
  • Albedo Estimation: Albedo estimation predicts per-pixel diffuse RGB albedo from high-fidelity synthetic pairs, encouraging illumination-invariant recovery of skin tone and clothing through value, gradient, and mean-alignment losses.The loss aligns spatial RGB means in addition to per-pixel and gradient errors.

6 EXPERIMENTS

Sapiens2 is evaluated for feature generalization and post-training performance across challenging in-the-wild human-centric tasks. It outperforms comparable baselines broadly, with strong gains in pose, segmentation, pointmap, normal, and albedo estimation.

  • Evaluation: Evaluation uses task-specific, challenging in-the-wild test sets designed to assess fidelity and generalization beyond existing benchmarks.The pose set contains 11K images with 308 keypoints, while segmentation uses 5K images with 29 classes.
  • Dense probing: Sapiens2 outperforms baselines at comparable model sizes, while Sapiens2-5B surpasses all baselines across every dense-probing task.Dense probing freezes each backbone and trains a lightweight task-specific decoder with identical hyperparameters across methods.
  • Pose: Sapiens2-0.8B outperforms larger models despite its smaller parameter count, and performance gains increase predictably with model scale.The comparison uses state-of-the-art whole-body top-down pose estimators retrained on the new keypoint set.
  • Segmentation: 27.9% mIoU and 16.9% mAcc separate Sapiens2–1B from Sapiens-1B on 5K segmentation, supported by in-the-wild supervision and 1K rather than 0.5K output resolution.Both models use the same 1K input resolution, and all baselines are trained on the same training set.
  • Pointmap and normal: Sapiens2 outperforms all pointmap baselines, while Sapiens2-0.4B achieves 8.63◦ mean angular error and 94.76% of human pixels below 30◦.Pointmap evaluation uses focal-length-normalized canonical coordinates; normal evaluation uses whole-body scans with 4K ground-truth normals.
  • Albedo: Sapiens2-5B achieves the lowest albedo MAE of 0.012 and highest PSNR of 32.6 dB, while recovering skin tone under varying lighting and generalizing in the wild.Despite synthetic-only training, the feedforward model is significantly more efficient at inference than diffusion-based methods.

7 CONCLUSION

SAPIENS2 introduces high-resolution, human-centric models pretrained on 1 billion images, combining masked reconstruction and contrastive objectives to learn appearance cues and semantics. The models outperform general-purpose models on human images, support tasks from pose estimation to albedo recovery, and provide a foundation for high-fidelity dense prediction.

  • Contributions: SAPIENS2 uses a 1-billion-image pretraining dataset for high-resolution, human-centric models.The models are designed for human images and high-fidelity dense predictions.
  • Contributions: Masked reconstruction and contrastive objectives jointly teach SAPIENS2 appearance cues and semantics.This unified objective supports both visual detail and semantic understanding.
  • Contributions: SAPIENS2 consistently outperforms general-purpose models on human images and spans tasks from pose estimation to albedo recovery.Its task range includes both pose estimation and albedo recovery.
  • Contributions: Sapiens2-1B effectively encodes low-level details for albedo estimation and generalizes to in-the-wild images despite limited synthetic training data.The qualitative result concerns albedo estimation using Sapiens2-1B.

A APPENDIX · A.1 PRETRAINING · A.1.1 IMPLEMENTATION DETAILS

SAPIENS2 pretraining uses dense-probing evaluations to guide design, combining masked reconstruction and contrastive learning in a multi-view setup. Training specifies detailed optimization, scheduling, teacher–student, and checkpoint-evaluation procedures.

  • A.1.1 IMPLEMENTATION DETAILS: Dense-probing evaluations serve as the final metrics guiding pretraining design decisions.This evaluation criterion is used to select and assess pretraining choices.
  • A.1.1 IMPLEMENTATION DETAILS: SAPIENS2–1B is pretrained at 1024×768 with embed dim 1536, 40 layers, 24 heads, patch size 16, and a final norm with [CLS].The model configuration is specified for the implementation described.
  • A.1.1 IMPLEMENTATION DETAILS: Pretraining combines a joint MAE and contrastive objective with an 8-layer MAE decoder and a [CLS] projection head.The MAE decoder has dim 512 and uses ℓ2 reconstruction.
  • A.1.1 IMPLEMENTATION DETAILS: The loss weights are MAE: 1.0, CLS: 0.4, and KoLeo: 0.04.These weights define the relative contributions of the listed objectives.
  • A.1.1 IMPLEMENTATION DETAILS: Multi-view training is adopted as part of the pretraining procedure.The supplied implementation passage introduces multi-view training but does not provide its full specification.
  • A.1.1 IMPLEMENTATION DETAILS: Optimization uses fused AdamW with lr 1×10−4, (β1, β2)=(0.9, 0.95), wd 0.05, and zero-decay for norms, biases, positional and special tokens.The optimizer configuration includes distinct decay handling for the listed parameter categories.
  • A.1.1 IMPLEMENTATION DETAILS: Training runs for 5×105 iters with 103 warmup, cosine decay to 10−7, and global grad-norm clip 5.0.The schedule and gradient clipping parameters are specified explicitly.
  • A.1.1 IMPLEMENTATION DETAILS: The contrastive teacher EMA is 0.992, center momentum is 0.9, student temperature is 0.1, and teacher temperature warms from 0.065 to 0.07 over the first 103 iters.Every checkpoint is evaluated on downstream tasks with a frozen encoder, and results use the best checkpoint.

A.1.2 MASKING STRATEGY … A.6 ALBEDO ESTIMATION

The appendix details Sapiens2’s high-resolution masking strategy and downstream human-centric vision implementations, spanning pose estimation, body-part segmentation, pointmaps, surface normals, and albedo.

  • A.1.2 MASKING STRATEGY: Sapiens2 mixes blockwise and patchwise masking with a 75% mask ratio at patch size 16 for high-resolution pretraining.At 1024×768, this masks approximately 2304 of 3072 patches per image, producing coarse occlusions while preserving context for contrastive learning.
  • A.2 POSE ESTIMATION: The pose estimator uses ground-truth bounding boxes and predicts 308 keypoints with a feature-map encoder and a two-stage deconvolutional heatmap decoder.The decoder uses 1536 input channels, 308 output channels, 4× upsampling, and convolutional channels of (768, 768, 512).
  • A.2 POSE ESTIMATION: Pose fine-tuning uses AdamW with layer-wise learning-rate decay, gradient clipping, linear warmup for 500 iterations, and polynomial decay thereafter.The learning rate is 5×10^-4, weight decay is 0.1, and gradients are clipped to a global ℓ2 norm of 1.0.
  • A.3 BODY-PART SEGMENTATION: Sapiens2-1B performs body-part segmentation over 29 classes on real-world images.The appendix illustrates predictions from the Sapiens2-1B model for body-part segmentation.
  • A.4 POINTMAP ESTIMATION: Sapiens2-1B estimates XYZ pointmaps and visualizes their derived absolute depth as heatmaps together with surface normals computed from depth.The appendix presents pointmap visualizations for each image.
  • A.5 NORMAL ESTIMATION: Sapiens2-1B predicts surface normals in the normal-estimation task.The appendix provides a visualization of surface-normal predictions.
  • A.6 ALBEDO ESTIMATION: Sapiens2-1B predicts albedo, or base color, at 1024 × 768 resolution.The appendix visualizes albedo predictions from the Sapiens2-1B model.
Loading 2604.21681v1…