PaperlayerView exact revision on arXiv

Source-linked AI summary

BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motion

Michael J. Black, Priyanka Patel, Joachim Tesch, Jinlong Yang

arXiv:2306.16940v1cs.CV

TL;DR

HPS estimation needs realistic, diverse training data with accurate 3D supervision, which prior synthetic datasets lacked. BEDLAM supplies animated clothed humans with ground-truth SMPL-X bodies and shows that synthetic-only training can achieve strong real-image performance, while also revealing practical limits in motion and hair realism.

  • Problem

    HPS research needs training data with real ground-truth 3D bodies rather than only 2D or pseudo-ground-truth annotations, while prior synthetic data lacked sufficient realism and diversity.

  • Method

    BEDLAM provides monocular RGB videos of diverse animated, clothed people with ground-truth SMPL-X bodies, realistic clothing simulation, varied scenes, lighting, and cameras.

  • Results

    Training only on BEDLAM produces state-of-the-art accuracy on real-image benchmarks for 3D pose and body-shape estimation, with basic HMR approaching CLIFF.

  • Takeaways & Limitations

    BEDLAM suggests realistic synthetic data can generalize across real-image domains and may matter more for HPS accuracy than increasingly sophisticated architecture.

  • Takeaways & Limitations

    BEDLAM has relatively few sitting, lying, and complex sports poses, and lacks hair physics, long hairstyles, and diverse hair colors.

Abstract

from arXiv · show

We show, for the first time, that neural networks trained only on synthetic data achieve state-of-the-art accuracy on the problem of 3D human pose and shape (HPS) estimation from real images. Previous synthetic datasets have been small, unrealistic, or lacked realistic clothing. Achieving sufficient realism is non-trivial and we show how to do this for full bodies in motion. Specifically, our BEDLAM dataset contains monocular RGB videos with ground-truth 3D bodies in SMPL-X format. It includes a diversity of body shapes, motions, skin tones, hair, and clothing. The clothing is realistically simulated on the moving bodies using commercial clothing physics simulation. We render varying numbers of people in realistic scenes with varied lighting and camera motions. We then train various HPS regressors using BEDLAM and achieve state-of-the-art accuracy on real-image benchmarks despite training with synthetic data. We use BEDLAM to gain insights into what model design choices are important for accuracy. With good synthetic training data, we find that a basic method like HMR approaches the accuracy of the current SOTA method (CLIFF). BEDLAM is useful for a variety of tasks and all images, ground truth bodies, 3D clothing, support code, and more are available for research purposes. Additionally, we provide detailed information about our synthetic data generation pipeline, enabling others to generate their own datasets. See the project page: https://bedlam.is.tue.mpg.de/.

1. Introduction

BEDLAM addresses the difficulty of separating architecture improvements from training-data improvements in 3D human pose and shape estimation by providing realistic synthetic data with ground-truth 3D bodies. Training on BEDLAM alone produces strong performance on real-image benchmarks, including state-of-the-art results with both HMR and CLIFF.

  • HPS progress confounds neural-network architecture changes with training-data improvements, making it difficult to determine which matters most.
  • BEDLAM provides realistic synthetic monocular RGB videos with ground-truth SMPL-X bodies, diverse people, clothing, motions, scenes, lighting, and cameras.The dataset includes 271 body shapes, 27 hairstyles, 111 outfits, 1691 clothing textures, and 2311 motions.
  • Synthetic data can provide perfect ground truth and controlled diversity, whereas prior synthetic datasets were insufficient because they lacked realism and diversity.
  • Training only on BEDLAM produces methods with state-of-the-art accuracy on real-image benchmarks for both 3D pose and 3D body shape estimation.
  • HMR trained on BEDLAM approaches CLIFF’s accuracy on real images, while CLIFF trained on BEDLAM achieves the best overall performance across HBW and SSP-3D.
  • BEDLAM supports meta-ablation studies of architectures, backbones, and training data, and is released with assets and tools for further research.

2. Related work

Existing HPS training data trades off diversity against ground-truth quality, while prior synthetic datasets remain limited in realism, scale, clothing, motion, or avatar variation. BEDLAM is positioned to address these limitations with realistic, diverse clothed bodies and exact 3D supervision.

  • In-the-wild datasets commonly provide 2D annotations or pseudo-ground-truth SMPL parameters, while controlled captures are costly and limited in subjects and variety.
  • SSP-3D and HBW address body shape but are too small to train a general body-shape regressor.
  • Real HPS data offers either diverse natural images with low-quality ground truth or limited variability with high-quality ground truth.
  • Prior synthetic datasets used crude naked-body textures, silhouettes, rigged clothing, limited scenes, or small numbers of models and motions.
  • Physics-based clothing simulation improves realism but has constrained previous datasets because simulation is challenging and limits size and complexity.
  • Prior datasets were insufficient across properties including body shapes, textures, poses, motions, backgrounds, clothing, physical realism, and cameras.

3. Dataset

BEDLAM is constructed as a large synthetic dataset of animated, clothed people with diverse bodies, appearances, motions, scenes, and cameras. Its rendering pipeline provides realistic clothing deformation and additional annotations for HPS and related tasks.

  • Dataset overview: The pipeline renders sampled body shapes, skin textures, simulated clothing, cloth textures, hair, poses, scenes, illumination, and motion blur.
  • Appearance diversity: BEDLAM uses 50 female and 50 male skin albedo textures spanning seven ethnic groups and multiple variations to support inclusive HPS estimation.
  • Clothing: CLO3D simulates 111 complex real-world outfits with realistic details and deformations across body motions; the outfits receive 1691 artist-designed textures.
  • Clothing: For high-BMI bodies where cloth simulation frequently fails, BEDLAM transfers simulated-garment textures onto body UV maps as a fallback.
  • Motion: Motions are sampled from AMASS using BABEL labels to improve diversity, with GRAB hand motions added because AMASS usually lacks hand motion.
  • Scenes and rendering: Scenes contain 1–10 non-colliding subjects, with randomized viewpoints, camera extrinsics, cinematic orbits, and selected camera intrinsics.
  • Additional annotations: BEDLAM also provides depth maps and semantic segmentation masks for hair, clothing, and skin.
  • Dataset overview: BEDLAM combines 271 bodies, 27 hairstyles, 111 clothing types, 1691 clothing textures, 2311 motions, 95 HDRI scenes, and 8 3D scenes.The dataset contains 10K motion clips, 380K RGB frames, and about 1M person bounding boxes.

4. Experiments

The experiments evaluate HMR and CLIFF trained on synthetic data across real-image benchmarks, testing architectures, training sources, backbones, and dataset composition. BEDLAM-trained models generalize strongly, with simple HMR approaching CLIFF and BEDLAM-CLIFF achieving the strongest overall results.

  • 4.1. Implementation Details: BEDLAM-CLIFF-X extends the evaluation to full-body estimation by regressing body and hand poses.Hand training uses BEDLAM hand crops retained when MediaPipe detects the hand with confidence greater than 0.8.
  • 4.3. Comparison with the State-of-the-Art: BEDLAM-CLIFF* achieves the most accurate results on 3DPW and RICH when pretrained on BEDLAM and fine-tuned with mixed 3DPW and BEDLAM data.The comparison includes CLIFF†* and the prior CLIFF result.
  • 4.3. Comparison with the State-of-the-Art: BEDLAM-HMR* is nearly as accurate as BEDLAM-CLIFF* on 3DPW and more accurate than CLIFF†* on RICH under the same mixed training setup.This supports strong performance from a simpler architecture when trained on suitable data.
  • 4.3. Comparison with the State-of-the-Art: BEDLAM-CLIFF ranks second on both SSP-3D and HBW without training on either dataset, achieving the best average rank across the two body-shape benchmarks.SHAPY performs best on HBW, while Sengupta et al. performs best on SSP-3D, but each performs poorly on the other dataset.
  • 4.4. Ablation Studies: Using only 5% of BEDLAM, or 38K crops, gives reasonable 3DPW performance, and adding more BEDLAM data improves performance monotonically.The 38K-crop subset outperforms 85K AGORA crops, while combining BEDLAM and AGORA gives the best results.
  • 4.4. Ablation Studies: Realistic clothing simulation yields significantly better results than textured bodies, with a larger effect when using a weaker backbone.The experiments train HMR and CLIFF with synthetic data and compare different backbones, training sources, and data quantities.

5. Limitations and Future Work

The paper identifies limitations in BEDLAM’s assets, motion and scene realism, physical coverage, appearance diversity, footwear, and hand-and-face motion. Future work targets more realistic interactions, hair, body shapes, textures, shoes, and full-body motion capture.

  • Open source assets: Commercial asset licensing prevented use of many high-quality assets, making more open-source assets important for research progress.The authors identify licensing restrictions as a significant impediment to research progress.
  • Motion and scenes: BEDLAM’s randomly sampled motions do not model real-world correlations between clothing, motions, scenes, people, and objects.The dataset also contains relatively few sitting, lying, and complex sports poses, which challenge cloth simulation.
  • Hair: BEDLAM lacks hair physics, long hairstyles, and broad hair-color diversity, while hair cards can produce lighting artifacts.The authors propose strand-based hair grooming for flowing hair, hair-body interaction, and more varied lighting.
  • Body shape diversity: Body-shape coverage is non-uniform and omits children and several diverse body types, while high-BMI clothing can cause cloth-simulation failures.Retargeting AMASS motions to high-BMI subjects is also problematic.
  • More realistic body textures: BEDLAM’s skin textures lack fine detail and realistic reflectance, and suitable high-quality textures are difficult to license.The licensing difficulty constrains access to improved appearance assets.
  • Shoes: Barefoot bodies omit footwear effects; shoes such as high heels alter posture and gait and may require retargeting or inverse kinematics.Adding basic shoes is described as straightforward, but general footwear modeling is more complex.
  • Hands and Faces: Full-body and hand motion-capture data are scarce, especially for hands interacting with objects, and BEDLAM ignores facial motion.The authors note that current datasets do not evaluate full-body and facial motion together.

6. Discussion and Conclusions

BEDLAM-trained methods generalize to substantially varying real scenes, while experiments suggest architecture may matter less than commonly thought. A caveat is that HPS accuracy depends on backbone pre-training.

  • BEDLAM-trained methods generalize to real scenes across SSP-3D, HBW, 3DPW, and RICH.
  • BEDLAM-HMR outperforms many recent, more sophisticated methods, suggesting architecture may be less important than commonly thought.
  • HPS accuracy depends on backbone pre-training, with COCO 2D pose pre-training helping generalization.
  • The authors expect pre-training may eventually become unnecessary as synthetic data improves in realism.

Supplementary Material

The supplementary material documents dataset creation, statistics, examples, experiments, and qualitative results. It also directs readers to the supplemental video and project website for related materials.

  • The supplement provides dataset-creation details, dataset statistics, additional images, experimental results, and qualitative results.
  • The supplemental video presents the motions in BEDLAM, with the video, data, and related materials available on the project website.
  • The dataset name refers to independently animated synthetic humans whose resulting motions have a chaotic feel.

A. Dataset creation

BEDLAM combines diverse body shapes, clothing, motions, and multi-person scenes with additional rendered ground truth. Its construction includes practical compromises and limitations involving high-BMI simulation, motion placement, hair, and body-model compatibility.

  • Body shape diversity: BEDLAM uses AGORA and additional high-BMI CAESAR bodies to broaden body-shape diversity.
  • Body shape diversity: BEDLAM converts gendered source bodies into a gender-neutral SMPL-X shape space with 11 shape components.
  • Body shape diversity: The dataset’s BMI distribution has a gap at 25–30, yet the authors report good body-shape prediction and suggest generalization.
  • Body shape diversity: The appropriate BMI training distribution remains unclear, and high-BMI clothing and motion generation can cause simulation failures or interpenetration.
  • Motion and scenes: Motion sampling uses BABEL labels to obtain wider coverage of the motion space.
  • Clothing and rendering: Physics-based simulation models complex clothing deformation, while simple motion placement remains a future-work limitation.
  • Motion and scenes: BEDLAM places 1 to 10 subjects per sequence and selects animations and trajectories to avoid frozen poses and collisions.
  • Clothing and rendering: Hair-card assets can produce ghosting, illumination issues, hue shifts, and black hairstyles under certain rendering conditions.

B. Comparison to other datasets

Prior synthetic datasets vary widely in subjects, frames, image-generation realism, clothing, and motion diversity. BEDLAM is positioned against limitations in realism, clothing simulation, and comparable motion coverage.

  • Dataset scale and realism: Synthetic datasets range from a handful of bodies to over 1000 subjects, with substantial variation in frame counts and effective image diversity.
  • Dataset scale and realism: Most prior datasets composite rendered bodies onto backgrounds, limiting realism, while mixed-reality methods use captured people or simple graphics in real scenes.
  • Clothing and motion: Previous clothing representations include textures, scans, rigged models, and physics simulation, but simulation datasets often contain few garments.
  • Clothing and motion: Motion diversity is difficult to compare because reported counts are approximate and may conceal repeated or insufficiently varied motions.
  • Additional ground truth: BEDLAM additionally provides depth maps and semantic segmentation masks for hair, clothing, and skin.
  • Additional ground truth: BEDLAM’s third-party rendering assets include Unreal Marketplace environments.
  • Additional ground truth: 3DPW contains roughly 51K frames and about 100K unique bounding boxes across 60 moving-camera sequences.

C. Implementation Details

BEDLAM-CLIFF-X extends the body estimator to full-body SMPL-X prediction by combining body and hand networks, with a refinement module for elbow and wrist pose. Training uses multiple parameter, keypoint, joint, and vertex losses plus extensive image augmentation.

  • BEDLAM-CLIFF-X: BEDLAM-CLIFF-X combines BEDLAM-CLIFF body predictions with a separate MANO-compatible hand network for full-body SMPL-X estimation.The body and hand outputs are merged into a full-body prediction.
  • BEDLAM-CLIFF-X: The refinement network updates elbow and wrist pose using features and pose estimates from both body and hand networks.Learning only wrist updates helps prevent unnatural wrist bending.
  • Training: Training combines MSE losses on model parameters, projected keypoints, and 3D joints with an L1 loss on 3D vertices.
  • Training: Training augmentation varies crops, scale, blur, compression, image appearance, noise, grayscale conversion, and downscaling.

D.1. Ablation of training data and backbones

The ablations show that backbone pretraining and synthetic-data composition strongly affect HPS accuracy. COCO pretraining performs best, BEDLAM outperforms AGORA alone, and combining both synthetic datasets improves results consistently.

  • Backbones: COCO-pretrained backbones produce the best results, while training from scratch on BEDLAM underperforms ImageNet and COCO pretraining.
  • Training data: BEDLAM-only training performs better than AGORA-only training on 3DPW.
  • Training data: BEDLAM+AGORA training consistently outperforms using either synthetic dataset alone.Both datasets are synthetic.
  • Training data: Using 50% or 25% of BEDLAM gives good performance, although training error continues decreasing through 100% of the dataset.

D.2. Ablation on losses

The loss ablations identify SMPL parameter supervision as particularly important, while projected-vertex supervision can hurt performance under camera mismatch. Loss choice also depends on the evaluation task and body-shape distribution.

  • Loss design: The loss ablation evaluates SMPL, 3D-joint, 2D-joint, 3D-vertex, and 2D-vertex supervision, with 2D-joint loss included throughout for image alignment.
  • Loss supervision: 3D-joint or 3D-vertex losses alone provide insufficient supervision, whereas combining SMPL loss with both substantially reduces error.
  • Loss supervision: Including the 2D-vertex loss slightly worsens performance, plausibly because camera mismatch during inference harms 3D pose.The authors suggest better camera estimation could make this loss useful.
  • Loss choice: L1 loss yields lower 3DPW error, but MSE performs better for body-shape estimation on SSP-3D and HBW.The authors therefore use MSE for the final model and main-paper results.
  • Dataset attributes: Clothing simulation consistently outperforms texture-only clothing, while hair provides a modest MPJPE and MVE improvement.With COCO pretraining, training-data differences matter less; the authors note that the hair models are not yet ideal or diverse.

D.4. Experiment on Human3.6M

Human3.6M results illustrate the importance of dataset overlap and training-set choice for evaluation. BEDLAM-CLIFF generalizes better than CLIFF without Human3.6M training, while qualitative comparisons show improved 3D alignment under camera variation and occlusion.

  • Human3.6M evaluation: CLIFF and its reimplementation use Human3.6M data, which contributes to their low Human3.6M test errors.
  • Human3.6M evaluation: BEDLAM-CLIFF outperforms CLIFF when Human3.6M is excluded from training, indicating better generalization under the matched evaluation protocol.
  • Human3.6M evaluation: Human3.6M training substantially reduces Human3.6M test error without reducing 3DPW error, reflecting similarity between its training and test conditions.The authors caution that methods can effectively overfit to Human3.6M.
  • SMPL-X evaluation: BEDLAM-CLIFF-X performs well on AGORA face and hand metrics, although body-shape overlap with AGORA gives it an advantage over methods without AGORA fine-tuning.
  • Qualitative comparison: BEDLAM-CLIFF produces better 3D body alignment than PARE and CLIFF in qualitative RICH and 3DPW comparisons.The side views reveal improved alignment despite camera variation, camera angle changes, and frame occlusion.
  • SMPL-X evaluation: BEDLAM-CLIFF-X qualitative results cover AGORA-test, BEDLAM-test, 3DPW-test, and RICH-test images, including hand poses and gendered body shapes.
Loading 2306.16940v1…