PaperlayerView exact revision on arXiv

Source-linked AI summary

SMPLer-X: Scaling Up Expressive Human Pose and Shape Estimation

Zhongang Cai, Wanqi Yin, Ailing Zeng, Chen Wei, Qingping Sun, Yanjun Wang, Hui En Pang, Haiyi Mei, Mingyuan Zhang, Lei Zhang, Chen Change Loy, Lei Yang, Ziwei Liu

arXiv:2309.17448v3cs.CV

TL;DR

EHPS methods remain limited by dependence on a confined set of training datasets, restricting coverage across varied scenarios. SMPLer-X scales data and ViT-based model capacity using diverse datasets, then finetunes the foundation model into specialists. It achieves strong performance across seven benchmarks, including 107.2mm NMVE on AGORA, while the evaluation remains bounded by benchmark coverage and architecture exploration.

  • Problem

    EHPS methods depend largely on a confined set of training datasets, despite diverse scenarios requiring broader coverage.

  • Method

    SMPLer-X systematically benchmarks 32 EHPS datasets and scales diverse training data with ViT-based model capacity, using selected data for specialist finetuning.

  • Results

    107.2mm NMVE on AGORA, an 11.0% improvement, accompanies state-of-the-art performance on EgoBody, UBody, and EHF.

  • Takeaways & Limitations

    The benchmark and benchmarking-finetuning paradigm provide guidance for transferable EHPS training and rapid adaptation to specific scenarios.

  • Takeaways & Limitations

    The five evaluation benchmarks may not sufficiently represent the real-world distribution, and architecture effects are not fully investigated because foundation-model training is costly.

Abstract

from arXiv · show

Expressive human pose and shape estimation (EHPS) unifies body, hands, and face motion capture with numerous applications. Despite encouraging progress, current state-of-the-art methods still depend largely on a confined set of training datasets. In this work, we investigate scaling up EHPS towards the first generalist foundation model (dubbed SMPLer-X), with up to ViT-Huge as the backbone and training with up to 4.5M instances from diverse data sources. With big data and the large model, SMPLer-X exhibits strong performance across diverse test benchmarks and excellent transferability to even unseen environments. 1) For the data scaling, we perform a systematic investigation on 32 EHPS datasets, including a wide range of scenarios that a model trained on any single dataset cannot handle. More importantly, capitalizing on insights obtained from the extensive benchmarking process, we optimize our training scheme and select datasets that lead to a significant leap in EHPS capabilities. 2) For the model scaling, we take advantage of vision transformers to study the scaling law of model sizes in EHPS. Moreover, our finetuning strategy turn SMPLer-X into specialist models, allowing them to achieve further performance boosts. Notably, our foundation model SMPLer-X consistently delivers state-of-the-art results on seven benchmarks such as AGORA (107.2 mm NMVE), UBody (57.4 mm PVE), EgoBody (63.6 mm PVE), and EHF (62.3 mm PVE without finetuning). Homepage: https://caizhongang.github.io/projects/SMPLer-X/

1 Introduction

SMPLer-X addresses EHPS generalization by systematically benchmarking diverse datasets and scaling both training data and model capacity. The resulting foundation and specialist models achieve strong, balanced performance across varied benchmarks.

  • Dataset scaling: The study benchmarks 32 EHPS datasets across five major benchmarks to guide robust and transferable training.The benchmark reveals significant inconsistencies among benchmarks and motivates data scaling to address domain gaps.
  • Dataset scaling: Dataset transferability depends on complementary data sources, with usefulness emerging once datasets exceed approximately 100K instances.The study also finds varying indoor scenes, synthetic data, and pseudo-SMPL-X labels can support transferability.
  • Foundation model: SMPLer-X is a minimalist generalist foundation model designed to facilitate massive data and parameter scaling across EHPS scenarios.Its experiments with varied data combinations and model sizes produce a well-rounded model intended as a baseline for future work.
  • Specialist models: 107.2mm NMVE marks an 11.0% improvement on AGORA, while finetuned specialists achieve state-of-the-art performance on EgoBody, UBody, and EHF.The specialist models are obtained by finetuning the foundation model using the selected data strategy.
  • Contributions: The contributions combine a comprehensive dataset benchmark, data and model scaling for a generalist model, and data-selected finetuning for specialists.The framework extends successfully to unseen datasets and targets varied benchmark scenarios.

2 Related Work

Prior EHPS methods span optimization-based fitting, complex multi-stage learning pipelines, and broader multi-dataset human-centric pretraining. These approaches leave open challenges in efficient, coherent whole-body estimation and comprehensive EHPS scaling.

  • Expressive Human Pose and Shape Estimation: EHPS estimates body, hands, and face motion and shape using parametric human models such as SMPL-X.The task supports applications including animation, gaming, and fashion.
  • Expressive Human Pose and Shape Estimation: Multi-stage whole-body models crop and resize hands and face regions, but can produce inconsistent articulation and implausible wrist rotations in difficult scenes.These issues are associated with occluded, truncated, and blurry inputs.
  • Expressive Human Pose and Shape Estimation: OSX introduces a one-stage ViT-based framework intended to relieve problems in earlier multi-stage pipelines and provide a concise route to model scaling.The passage presents OSX as a promising approach rather than a complete solution to broader dataset-scaling challenges.
  • Multi-dataset Training for Human-centric Vision: Existing multi-dataset human-centric models evaluate mainly 2D tasks, while prior 3D work recovers skeletons without estimating body shapes and meshes.HumanBench uses 37 datasets, UniHCP 33, and Sárándi et al. 28 datasets.

3 Benchmarking EHPS Datasets

The benchmark evaluates 32 EHPS datasets across five diverse test benchmarks and analyzes which dataset attributes support cross-domain generalization. Results show that dataset diversity, scene coverage, synthetic data, and accurate or pseudo-SMPL-X annotations matter more than scale alone once datasets exceed roughly 100K instances.

  • Benchmark setup: 32 datasets are benchmarked by training a standardized SMPLer-X model on each dataset and evaluating it across AGORA, UBody, EgoBody, 3DPW, and EHF.Datasets are ranked using mean primary error, excluding certain in-domain test results from MPE computation.
  • Benchmark findings: Single-dataset training performs well in-domain but often transfers poorly, with AGORA ranking 1st on AGORA yet 24th on EHF.This cross-domain variation motivates scaling training data for robust and transferable EHPS models.
  • Dataset attributes: Above approximately 100K instances, dataset size is not strongly associated with ranking; MSCOCO outperforms datasets with 10× larger scales.The analysis suggests prioritizing diversity and quality after a dataset becomes adequately large.
  • Dataset attributes: In-the-wild datasets rank highest overall, indoor datasets occupy much of the top 20, and fixed-scene studio datasets rank lower.In-the-wild data has the broadest distribution, while indoor scene variety can outperform a single-scene indoor collection.
  • Dataset attributes: Most contemporary synthetic datasets rank highly, and their learned knowledge transfers to real benchmarks because they cover in-the-wild-like and other real-world feature distributions.Four of the five evaluation benchmarks are real datasets.
  • Dataset attributes: Accurate or pseudo-SMPL-X annotations characterize most top-10 datasets, whereas SMPL annotations tend to rank lower for SMPL-X estimation.InstaVariety ranks third despite using pseudo-SMPL-X annotations, supporting pseudo-label fitting even when labels are noisy.

4 Scaling up EHPS

SMPLer-X uses a minimalist, scalable architecture to study data and model scaling for EHPS. More datasets improve performance and transferability, while larger backbones help with diminishing returns beyond ViT-L; finetuning further adapts the foundation model.

  • Model Architecture: SMPLer-X combines a scalable ViT backbone, a neck for hand and face feature cropping, and separate heads for body parts.The architecture retains only essential components to support efficient large-scale training and decouple the framework from specific algorithm designs.
  • Training Setup: Balanced sampling standardizes selected datasets to equal lengths, preventing larger datasets from overshadowing smaller ones during training.The training pipeline also standardizes datasets into COCO or HumanData formats.
  • Scaling Results: More training datasets gradually improve MPE, with their diverse scenarios contributing beyond the increase in training instances.At a fixed data amount, larger foundation models perform better, but ViT-H provides limited additional gain beyond ViT-L despite having more than twice its parameters.
  • Efficiency: SMPLer-X-S and SMPLer-X-B achieve real-time inference, while SMPLer-X-L approaches real-time speed and all variants exceed OSX’s 12.2 FPS under the same setting.Inference speed is measured on a single V100 GPU with batch size 1, excluding data loading.
  • Generalization: Foundation models show strong, balanced performance across benchmarks and transfer effectively to unseen ARCTIC and DNA-Rendering environments.The models outperform SOTAs using conventional data-sampling strategies, while visual comparisons cover truncation, difficult poses, and rare camera angles.
  • Specialization: Finetuning the ViT-L foundation model for five epochs consistently produces substantial performance improvements and yields specialist models with strong benchmark results.The finetuning strategy selects five high-ranking datasets using the target training set rather than the inaccessible test set.

5 Conclusion

The work benchmarks EHPS datasets and uses the resulting insights to train and adapt a generalist foundation model. It provides reusable resources and a benchmarking-finetuning paradigm, while acknowledging limits in benchmark coverage, architecture exploration, and potential misuse.

  • Contributions: The released pretrained model, benchmark, and benchmarking-finetuning paradigm support future EHPS generalization studies and rapid adaptation to specific scenarios.Users can evaluate pretrained models on a collected training set and select relevant datasets for finetuning.
  • Limitations: The five evaluation datasets may still be insufficient to represent the full real-world distribution.This limits how comprehensively the reported generalization capability can be gauged.
  • Limitations: The experiments do not fully investigate the impact of different model architectures because foundation-model training is prohibitively expensive.The limitation concerns architectural coverage rather than the reported scaling experiments themselves.
  • Societal Impact: Released strong EHPS models could be used for unwarranted surveillance or privacy violations.The paper identifies this as a potential negative societal impact of the work.

B.1 Dataset Descriptions

The study covers 32 public academic datasets spanning studio, in-the-wild, synthetic, egocentric, crowded, and human-object-interaction settings. These datasets vary substantially in annotation type, scale, capture setup, and scenario diversity.

  • In-the-Wild and Mobile Capture: 3DPW is an in-the-wild dataset captured with a moving phone camera and IMU sensors, providing accurate SMPL annotations across diverse environments.It contains 60 video sequences and uses official train, validation, and test splits.
  • Synthetic and Rendered Data: AGORA provides accurate SMPL-X annotations for diverse poses and appearances using high-quality scans rendered in realistic 3D scenes.It contains 14K training images, 3K test images, and 173K instances.
  • Interaction Datasets: ARCTIC captures whole-body hand-object interaction with SMPL-X annotations from eight static cameras and one egocentric camera.The dataset includes 10 subjects manipulating 11 objects and 210K video frames.
  • Interaction Datasets: BEHAVE contains multi-view RGB-D frames with SMPL-H parameters, object fits, and contact information for interactions with 20 common objects.It covers about 15K frames, eight subjects, and five locations.
  • Egocentric and Scene-Centric Data: EgoBody combines third-person multi-view and head-mounted egocentric recordings across 125 sequences, 36 subjects, and 15 indoor scenes.Its two capture modes are called MVSet and EgoSet.
  • Studio Motion Capture: FIT3D and HumanSC3D are large studio motion-capture datasets with multi-view sequences and SMPL-X pose and shape annotations.FIT3D contains 2,964,236 images, while HumanSC3D contains 1.2M ground-truth instances.
  • Pseudo-Annotated Data: In-the-wild datasets such as MPII, MSCOCO, CrowdPose, InstaVariety, and LSPET provide varied images or crowded scenes but often rely on pseudo-annotations fitted from 2D joints.Their annotated scales range from 2,946 instances in LSPET to approximately 149.8K in MSCOCO.
  • Specialized Pose and Capture Settings: MTP targets difficult self-contact poses, while MPI-INF-3DHP combines constrained indoor and complex outdoor capture across 14 camera views.MTP provides pseudo-ground-truth SMPL-X parameters and 2D keypoints for 3.2K training instances; MPI-INF-3DHP provides 939,847 annotated instances.

C.1 Architecture

SMPLer-X uses a minimalist transformer-based architecture for expressive human pose and shape estimation, with specialized processing for body, hands, and face. Its annotation adaptation converts gendered SMPL-X parameters efficiently but does not transfer reliably across SMPL and SMPL-X topologies.

  • Architecture: SMPLer-X crops and resizes the whole-body image, tokenizes it into 32×24 patches, adds task tokens, and processes the tokens with a ViT backbone.The input resolution is 512×384 with patch size 16.
  • Architecture: Predicted hand and face bounding boxes drive ROI feature crops, while separate body, hand, and face heads estimate keypoints and model parameters.Hand and body heads use positional and regressor modules; the face head uses a regressor module.
  • Annotation adaptation: The gendered-to-neutral SMPL-X adapter achieves 8.4 mm vertex-to-vertex error on AGORA and converts annotations in 0.09 seconds per frame.The adapter is a three-layer fully connected network applied to AGORA, EgoBody, DNA-Rendering, and RICH.
  • Annotation adaptation: Across SMPL and SMPL-X topologies, adapter training produces 27.1 mm vertex-to-vertex error, so SMPL annotations supervise only global orientation and body pose.The authors nevertheless report performance gains from this supervision strategy.
  • Data distributions: UMAP distributions show distinct hand-pose and body-shape clusters for several datasets, while facial-expression coverage remains insufficiently diverse.DNA-Rendering, CHI3D, HumanSC3D, and Talkshow form distinct clusters for hand poses and betas.

D.2 Training Schemes

The training ablation compares dataset selection and finetuning scope. Higher-ranked datasets improve foundation-model performance, while finetuning only the neck and heads can match or slightly exceed full-network finetuning with fewer learnable parameters.

  • Dataset selection: Top 5 or Top 10 datasets produce much better performance than Bottom 5 or Bottom 10 datasets when trained with the same number of instances.The comparison uses SMPLer-X-B models and rankings from the individual dataset benchmark.
  • Dataset selection: Selecting higher-ranked datasets benefits the foundation model, whereas lower-ranked datasets are less effective at improving performance.The authors describe this as evidence that their training scheme is efficient.
  • Finetuning scope: Finetuning only the neck and head is slightly better than finetuning the entire network while using many fewer learnable parameters.The models are evaluated on the UBody test set, with finetuned variants marked separately in Table 12.
  • Finetuning scope: The authors speculate that extensive multi-dataset training makes the backbone strong and generalizable, leaving limited improvement from backbone finetuning.This explanation is presented as a speculation rather than a demonstrated mechanism.

D.4 Data Sampling Strategies

The sampling ablation compares balanced and ranking-weighted dataset lengths. Foundation-model performance is not sensitive to the sampling strategy, while balanced sampling is simpler and more efficient to implement.

  • Sampling comparison: Balanced sampling gives every dataset the same length, whereas weighted sampling adjusts lengths according to individual dataset benchmark rankings.Weighted sampling upsampled or downsampled datasets toward an arithmetic sequence of lengths.
  • Sampling comparison: Foundation-model performance is not sensitive to the sampling strategy.The comparison evaluates balanced, weighted, and another sampling strategy in the sampling ablation.
  • Sampling comparison: Balanced sampling is more intuitive, easier to implement, and efficient, while weighted sampling may have greater potential with further weight tuning.The latter is presented as a possible direction rather than a reported established advantage.

D.5 Training Domains

Training-domain experiments show strong benefits from both seeing a benchmark during training and adding many out-of-domain datasets. Using 32 datasets achieves the best reported MPE for the SMPLer-X-L model.

  • Domain exposure: In-domain training is highly effective because seeing a benchmark’s dataset during training brings significant performance improvement.The experiment includes a benchmark’s training split and evaluates on its test split.
  • Out-of-domain transfer: More training datasets particularly improve unseen-benchmark performance, reducing errors on EHF, ARCTIC, and DNA-Rendering-HiRes.These results support evaluating dataset generalization beyond in-domain performance.
  • Out-of-domain transfer: Training on 32 datasets gives SMPLer-X-L its best reported performance, with 66.2 mm MPE.The authors characterize it as a strong and effective SMPL-X estimator.
  • Benchmark construction: The supplementary benchmark ranks EHPS datasets using training sets from AGORA, UBody, EgoBody, and 3DPW, omitting EHF because it has no training set.This benchmark avoids selecting datasets using rankings that leak information from evaluation test sets.

E.2 Complete Results of Foundation Models on Evaluation Benchmarks

The section reports complete foundation-model results across six evaluation benchmarks and describes dataset-selection rankings used to identify training data for key benchmarks.

  • Complete results for SMPLer-X-H32 are reported on AGORA, UBody, EgoBody-EgoSet, EHF, ARCTIC, and DNA-Rendering-HiRes.
  • UBody results distinguish methods finetuned on the UBody training set from those evaluated without that finetuning.
  • ARCTIC and DNA-Rendering-HiRes results separately mark methods finetuned on their respective training sets.
  • EgoBody-EgoSet results identify methods finetuned on the EgoBody-EgoSet training set.
  • The 3DPW table lists only whole-body SMPL-X methods, marks 3DPW-finetuned models, and reports results in millimeters.
  • Training datasets are ranked by MPE using evaluations on their own training sets and the training sets of AGORA, UBody, EgoBody-EgoSet, and 3DPW.
Loading 2309.17448v3…