PaperlayerView exact revision on arXiv

Source-linked AI summary

PoseDreamer: Scalable and Photorealistic Human Data Generation Pipeline with Diffusion Models

Lorenza Prospero, Orest Kupyn, Ostap Viniavskyi, João F. Henriques, Christian Rupprecht

arXiv:2603.28763v1cs.CV

TL;DR

Labeled 3D human data is difficult to acquire because monocular annotation is ambiguous and existing real or rendered datasets have scale, realism, diversity, or cost limitations. PoseDreamer uses diffusion-based controllable generation with alignment, hard-sample mining, and filtering to produce mesh-annotated data, achieving performance comparable to or better than existing alternatives and stronger dataset-combination results.

  • Problem

    Acquiring large labeled 3D human datasets is difficult because monocular 3D annotation is ambiguous, while real and rendered datasets have important scale, diversity, realism, or cost limitations.

  • Method

    PoseDreamer uses diffusion models for controllable human-image generation with 3D-2D consistency, Direct Preference Optimization, curriculum-based hard-sample mining, and quality filtering.

  • Results

    Models trained on PoseDreamer perform comparably to or better than models trained on real-world and traditional synthetic datasets, while combined datasets outperform real-world-plus-synthetic combinations.

  • Takeaways & Limitations

    PoseDreamer provides a scalable, photorealistic alternative to rendering-based human-data acquisition with complementary value when combined with synthetic datasets.

  • Takeaways & Limitations

    PoseDreamer trades some label precision for greater flexibility, scalability, and photorealism, and generated samples may contain subtle inconsistencies absent from rendering engines.

Abstract

from arXiv · show

Acquiring labeled datasets for 3D human mesh estimation is challenging due to depth ambiguities and the inherent difficulty of annotating 3D geometry from monocular images. Existing datasets are either real, with manually annotated 3D geometry and limited scale, or synthetic, rendered from 3D engines that provide precise labels but suffer from limited photorealism, low diversity, and high production costs. In this work, we explore a third path: generated data. We introduce PoseDreamer, a novel pipeline that leverages diffusion models to generate large-scale synthetic datasets with 3D mesh annotations. Our approach combines controllable image generation with Direct Preference Optimization for control alignment, curriculum-based hard sample mining, and multi-stage quality filtering. Together, these components naturally maintain correspondence between 3D labels and generated images, while prioritizing challenging samples to maximize dataset utility. Using PoseDreamer, we generate more than 500,000 high-quality synthetic samples, achieving a 76% improvement in image-quality metrics compared to rendering-based datasets. Models trained on PoseDreamer achieve performance comparable to or superior to those trained on real-world and traditional synthetic datasets. In addition, combining PoseDreamer with synthetic datasets results in better performance than combining real-world and synthetic datasets, demonstrating the complementary nature of our dataset. We will release the full dataset and generation code.

1 Introduction

PoseDreamer addresses the scarcity and limitations of labeled 3D human datasets by using diffusion models to generate photorealistic, diverse images with precise 3D-2D consistency. Its controllable generation, alignment, hard-sample mining, and large-scale dataset improve the practicality of synthetic training data.

  • Motivation: 3D human annotation is costly, time-consuming, ambiguous from monocular images, and limited in existing real-world datasets.Rendering-based alternatives also require substantial expertise and assets, while often producing a synthetic look that creates a real-world domain gap.
  • Approach: PoseDreamer uses diffusion models to generate synthetic human-mesh data while ensuring precise 3D-2D consistency.The pipeline is presented as an alternative to traditional rendering, with greater scene diversity and visual realism.
  • Approach: Direct Preference Optimization aligns controllable image generation with 3D pose constraints through an enhanced mesh-to-RGB encoding scheme.The control-alignment component is designed to improve correspondence between generated images and their 3D annotations.
  • Approach: Curriculum-based hard sample mining uses downstream mesh-recovery feedback to prioritize challenging samples and avoid redundant easy cases.This two-stage strategy is intended to maximize learning value and data efficiency.
  • Results: 500,000 images with detailed 3D body meshes form a large, diverse dataset for training and evaluating human mesh-recovery models.The experiments report quality, accuracy, and real-world generalization comparable to or better than models trained on 3D game-engine datasets.
  • Results: Combining PoseDreamer with synthetic datasets outperforms mixing real-world and synthetic datasets, demonstrating complementary dataset value.The authors present this result as evidence that generative-model data can replace expensive synthetic human-data acquisition.

2 Related Work

Prior human mesh-recovery datasets are constrained by scale, diversity, realism, accessibility, or production cost. PoseDreamer positions diffusion-based generation as an efficient alternative that scales to arbitrary sample counts and varied in-the-wild scenes without manually building extensive asset libraries.

  • Data Generation with Diffusion Models: Diffusion-based data-generation methods have improved synthetic classification data and can jointly generate images with annotations for segmentation or depth.Existing approaches remain limited in task scope or require fine-tuning for each specific dataset.
  • PoseDreamer: Figure 2 presents PoseDreamer samples emphasizing photorealistic appearance, precise spatial control, and diverse challenging scenarios.These properties are described as supporting robust model training.
  • Real-World HMR Datasets: Real-world datasets provide valuable 3D annotations but often restrict actors, scenarios, environments, clothing, lighting, or body-shape detail.Examples include controlled motion-capture and laboratory datasets, alongside in-the-wild data with limited sequence diversity.
  • Synthetic HMR Datasets: Rendering pipelines can produce hundreds of thousands of fully annotated images, but their realism, availability, or production cost varies substantially.BEDLAM requires substantial monetary and computational investment, while SynBody reduces cost at the expense of realism; Ultrapose is not publicly available.
  • PoseDreamer: PoseDreamer scales efficiently to arbitrary sample counts and varied in-the-wild scenes without manually collecting large libraries of textures or prompts.This distinguishes the method from approaches that depend on extensive manually assembled assets.

3 Dataset

PoseDreamer constructs a large-scale human-mesh dataset by sampling SMPL-X parameters and captions, generating controllable images, and filtering for 3D–2D consistency and challenging content. The pipeline combines improved spatial encoding, DPO-aligned control, curriculum-based selection, and multi-stage filtering to produce 500,000 training samples.

  • PoseDreamer defines the dataset as paired RGB images and corresponding ground-truth SMPL-X avatar parameters.
  • Generating labels after image synthesis is unreliable because predicted 3D meshes often misalign with observed poses and body shapes.
  • The inverted approach samples diverse SMPL-X parameters first and uses them as conditioning signals for image generation, with alignment and filtering preserving correspondence.
  • LAION supplies realistic scene diversity, while AMASS contributes dynamic poses involving activities such as dancing, martial arts, sports, and interactions.
  • Image Generation: Independent normalization of the X, Y, and Z mesh coordinates into RGB channels provides richer spatial cues than naive vertex-ID color mapping.
  • Control Model Alignment: The control model is trained on 130,000 rendered meshes and aligned using DensePoseCOCO and AGORA annotations.
  • Control Model Alignment: DPO alignment improves adherence to input 3D pose constraints, producing a 42.1% reduction in error rate.
  • Dataset Generation: Curriculum-based selection prioritizes challenging examples through a two-stage generation pipeline instead of redundant easy samples.

4 Experimental Evaluation

PoseDreamer is evaluated across human mesh recovery benchmarks, image quality, and pipeline ablations. The results show strong downstream performance, substantially improved realism, and benefits from each pipeline component.

  • Benchmarking: Evaluation spans five diverse benchmarks covering synthetic and real-world human mesh recovery scenarios.The protocol includes AGORA, UBody, EgoBody, 3DPW, and EHF, with models trained on single datasets and dataset mixtures.
  • Benchmarking: PoseDreamer achieves performance comparable to BEDLAM, despite BEDLAM being 2× larger and more expensive to design.The diffusion-generated dataset also outperforms all evaluated real-world datasets in the reported comparisons.
  • Dataset Combinations: When combined with BEDLAM, PoseDreamer outperforms all evaluated multi-dataset mixtures, including BEDLAM+COCO and a five-dataset ensemble.The reported synergy combines BEDLAM’s precise 3D supervision with PoseDreamer’s photorealistic diversity.
  • Image Quality: PoseDreamer records an Inception Score of 9.78 and FID of 1.72, versus AGORA’s 5.55 and 8.14.These values correspond to a 76% Inception Score improvement and a 69% FID reduction compared with the best-performing baseline identified in the passage.
  • Ablation Analysis: Removing any pipeline component degrades downstream mesh recovery performance.Quality filtering prevents misaligned samples, AMASS integration adds pose complexity, and hard sample mining supports learning from challenging examples.

5 Limitations and Broader Impact

PoseDreamer trades some label precision for greater flexibility, scalability, and photorealism. Its reliance on foundation models may limit representation for underrepresented groups, while realistic human imagery introduces misuse risks.

  • Limitations: Diffusion-generated samples may contain subtle label inconsistencies that rendering engines avoid.The paper frames this as a trade-off between perfect synthetic labels and realistic visual diversity.
  • Limitations: Foundation-model training data may introduce biases that limit pose diversity for underrepresented demographic groups or activities.This boundary concerns the diversity represented by the generated samples.
  • Broader Impact: Realistic human-image generation raises ethical concerns about misleading content and unauthorized representations.The paper notes that focusing on 3D mesh parameters rather than identity-specific features helps preserve privacy.

6 Conclusion

PoseDreamer presents diffusion-based generation as a scalable alternative for human mesh recovery data. It combines photorealism and scene diversity with alignment and filtering to improve the practical applicability of synthetic data.

  • Conclusion: PoseDreamer is a viable and scalable alternative to traditional rendering pipelines for human mesh recovery.The conclusion positions diffusion-based generation as a route to large annotated datasets without traditional production costs.
  • Conclusion: The pipeline combines generative-model photorealism and scene diversity with robust alignment and filtering mechanisms.This combination is presented as bridging synthetic data quality and real-world applicability.
  • Broader Significance: The approach aims to democratize access to high-quality 3D human datasets for advancing human recognition and reconstruction methods.The stated scope is reducing the computational and financial costs of traditional synthetic data generation.

A.1 Scaling

Scaling experiments show that PoseDreamer remains useful as model size and training data increase. Mixtures using PoseDreamer and BEDLAM match or exceed larger multi-dataset training configurations on most evaluated benchmarks.

  • Scaling: A 0.75M-sample PoseDreamer+BEDLAM mixture outperforms a five-dataset mixture across all benchmarks except EgoBody and AGORA.AGORA was included in the five-dataset training set.
  • Scaling: A 250K BEDLAM plus 500K PoseDreamer mixture outperforms an equal split of the two datasets.The result indicates that dataset composition and scale affect performance.
  • Scaling: A 1.5M-sample PoseDreamer+BEDLAM mixture performs comparably to a ten-dataset mixture on benchmarks outside the training set.On the in-the-wild UBody benchmark, the two-dataset model outperforms the ten-dataset model.
  • Scaling: PoseDreamer contributes performance improvements when both data volume and model size are scaled.The reported results associate increased pose and environment diversity with improved scaling behavior.

A.2 Domain-specific Generation

PoseDreamer can generate domain-specific human data by replacing part of a general pose corpus with targeted yoga-pose images. This preserves overall benchmark performance while improving results on yoga-specific evaluation.

  • Domain-specific dataset construction: 30K yoga-pose images from MOYO replace part of a 300K-image training set sampled from AMASS and LAION poses.The comparison trains one model on randomly sampled poses and another with 30K generated yoga images.
  • Evaluation: On the MPII Yoga subset, the model trained with yoga poses decisively outperforms the baseline.Figure 7 visually supports the improved consistency of predictions from the yoga-trained model with the ground-truth pose.
  • Conclusion: The results demonstrate that the pipeline can generate data for specific domains of interest and improve downstream performance in those domains.The reported benefit is strongest for the targeted MPII Yoga evaluation.

A.3 Spatial Control Ablations

The spatial-control ablations examine alternative control representations and assess generated-sample quality using pose and head-pose criteria. The final filtering thresholds select samples using OKS and head-pose error.

  • Control representations: DensePose predictions from LAION images were often inaccurate, incomplete, or poorly bounded, producing poor pixel alignment for control.The pipeline therefore considered rendered ground-truth annotations as an alternative source of image-label pairs.
  • Control representations: The control-model study compares different control-image representations using downstream model performance.Table 7 reports the comparison of representation choices for control-model training.
  • Control representations: The initial Continuous Surface Embeddings colormap provided insufficient visual variation for distinguishing head orientations and body configurations.The passage also notes that this colormap is defined only for SMPL parameters, while the goal is full-body pose control with SMPL-X.
  • Filtering: The filtering pipeline reports pose OKS and 3D head-pose MAE distributions for generated images, with low- and high-scoring examples.These statistics characterize the quality of generated samples before final dataset selection.
  • Filtering: The final dataset-selection thresholds are 0.8 for OKS score and 25 for head-pose error.The thresholds are applied to the reported pose and head-pose quality measures.

A.5 Image Quality

PoseDreamer’s generated samples more closely match real-image feature distributions, while precise annotations support additional mesh supervision. The pipeline still exhibits failures in complex poses and physically implausible scenes.

  • Annotation supervision: PoseDreamer’s precise annotations enable explicit supervision signals within the SMPL-X topology.The paper adds sparse visual markers as a shape-sensitive complement to joint supervision.
  • Annotation supervision: Visual marker supervision provides shape-sensitive signals while maintaining efficiency through a sparse set of 64 fixed mesh vertices.Markers are predicted alongside joints and supervised in both 3D and 2D spaces.
  • Image quality: PoseDreamer samples more closely follow the real LAION faces distribution than rendered datasets in DINO and CLIP feature projections.The image-quality analysis compares UMAP and t-SNE projections across PoseDreamer, real images, and other synthetic datasets.
  • Annotation supervision: PoseDreamer’s visual-marker model achieves 2D reprojection errors approaching SMPLer-X trained on 32 combined real and synthetic datasets.This result is reported as evidence that high-quality synthetic supervision can be competitive with many data sources.
  • Failure cases: Generated images can contain deformed body parts, additional limbs, or floating people in complex poses and multi-person interactions.The limitation passage also describes occasional errors in the 3D-to-2D mapping of limb joints.

C Mitigating Distribution Gap in Diffusion-Generated Data

Web-trained diffusion models can produce oversaturated imagery with an artificial aesthetic, creating a distribution gap from natural images. PoseDreamer addresses this with a real-image-trained realism adapter combined with spatial control.

  • Distribution gap: LAION-trained diffusion models can exhibit aesthetic biases that produce oversaturated, heavily filtered images with an artificial “AI look.”These artifacts reflect visual styles common in retouched internet images rather than natural image distributions.
  • Realism adapter: PoseDreamer uses a specialized LoRA adapter trained on real-world images to reduce aesthetic artifacts from web-trained diffusion models.The adapter transforms biased web-aesthetic outputs toward more natural image appearance.
  • Realism adapter: The modular EasyControl design combines the aesthetic/style adapter with the spatial-control LoRA at inference time.This combination is intended to preserve spatial control while correcting appearance.
  • Results: The realism adapter substantially reduces oversaturation and artificial enhancement artifacts in generated images.Figure 11 compares outputs generated without and with the adapter.

D Control Model Alignment: Implementation Details

PoseDreamer aligns generated images with 3D mesh controls through preference-based training and presents diverse dataset samples with precise mesh correspondence.

  • D Control Model Alignment: Implementation Details: 40,000 preference pairs train control alignment by ranking four image variants per caption–3D mesh combination with OKS scores.The preferred and less-preferred samples support Direct Preference Optimization.
  • D Control Model Alignment: Implementation Details: A lower β value of 250 than Flow-DPO’s 500 is used because control models have a more constrained generation space.
  • D Control Model Alignment: Implementation Details: Figure 12 provides additional examples of PoseDreamer dataset samples.
  • D Control Model Alignment: Implementation Details: Dataset examples show diverse poses and scenes alongside high-quality generated images with precise 3D mesh correspondence.
Loading 2603.28763v1…