PaperlayerView exact revision on arXiv

Source-linked AI summary

StyleGAN-Human: A Data-Centric Odyssey of Human Generation

Jianglin Fu, Shikai Li, Yuming Jiang, Kwan-Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, Ziwei Liu

arXiv:2204.11823v1cs.CVcs.LG

TL;DR

Unconditional articulated-human generation remains difficult, while prior work emphasizes network engineering more than data engineering. This paper builds the 230K-image SHHQ dataset and studies data size, distribution, and alignment, finding that these choices yield distinct generation-quality effects and support a human-generation model zoo and editing benchmark.

  • Problem

    Photo-realistic unconditional generation of articulated humans in natural clothing remains challenging despite advances in StyleGAN-based image generation.

  • Method

    The paper collects SHHQ and experimentally investigates data size, data distribution, and data alignment for StyleGAN-based unconditional human generation.

  • Results

    The study reports empirical insights on the three data factors and demonstrates a six-model human-GAN zoo through several face-editing methods.

  • Takeaways & Limitations

    Data-centric analysis and released models provide resources for future unconditional human generation and human editing research.

  • Takeaways & Limitations

    The baseline still produces entangled face, hand, and clothing artifacts, occasional extra hands, and inferior hand quality in rare poses.

Abstract

from arXiv · show

Unconditional human image generation is an important task in vision and graphics, which enables various applications in the creative industry. Existing studies in this field mainly focus on "network engineering" such as designing new components and objective functions. This work takes a data-centric perspective and investigates multiple critical aspects in "data engineering", which we believe would complement the current practice. To facilitate a comprehensive study, we collect and annotate a large-scale human image dataset with over 230K samples capturing diverse poses and textures. Equipped with this large dataset, we rigorously investigate three essential factors in data engineering for StyleGAN-based human generation, namely data size, data distribution, and data alignment. Extensive experiments reveal several valuable observations w.r.t. these aspects: 1) Large-scale data, more than 40K images, are needed to train a high-fidelity unconditional human generation model with vanilla StyleGAN. 2) A balanced training set helps improve the generation quality with rare face poses compared to the long-tailed counterpart, whereas simply balancing the clothing texture distribution does not effectively bring an improvement. 3) Human GAN models with body centers for alignment outperform models trained using face centers or pelvis points as alignment anchors. In addition, a model zoo and human editing applications are demonstrated to facilitate future research in the community.

1. Introduction

The paper reframes unconditional articulated-human generation as a data-engineering problem alongside network engineering. It introduces SHHQ and studies data size, distribution, and alignment while releasing models and editing resources.

  • StyleGAN-based methods generate high-quality faces and simple objects, but photo-realistic articulated humans in natural clothing remain challenging.
  • SHHQ contains 230K clean full-body human images, providing the basis for data-centric experiments on unconditional human generation.
  • The study examines how data size, data distribution, and data alignment affect unconditional human generation quality.
  • The authors establish a model zoo containing StyleGAN, StyleGAN2, and alias-free StyleGAN models at 1024 × 512 and 512 × 256 resolutions.
  • An editing benchmark adapts image inversion and manipulation methods to human body models and demonstrates editing of clothes and attributes.

2. Related Work

Prior work spans data requirements, StyleGAN architectures, controllable human generation, and latent-space editing. The paper positions its contribution against limitations involving dataset suitability, paired data, and full-body complexity.

  • High-quality clothed-human GAN training requires sufficient image quality, data volume, attribute coverage, and appropriate single-person full-body content.
  • DeepFashion provides labels and garment diversity but is unsuitable for unconditional human generation because it emphasizes fashion items and contains relatively few full-body images.
  • StyleGAN and StyleGAN2 advanced unconditional image generation through style injection, redesigned normalization, multiscale processing, and regularization.
  • Existing controllable human-generation methods use conditional VAE, U-Net, or StyleGAN-related architectures for pose and appearance control.
  • Many controllable approaches require paired training data with adequate volume, while 3D approaches remain constrained by training-data quantity and quality.
  • Human image editing commonly combines inversion into latent space with latent-code manipulation using optimization, encoders, or hybrid methods.

3. Stylish-Humans-HQ Dataset

SHHQ is constructed by collecting, filtering, aligning, and annotating human images for unconditional full-body generation. Compared with public alternatives, it offers larger scale, high resolution, labeled attributes, and complete-body coverage.

  • 3.1. Data Collection and Preprocessing: Over 500K Internet images are collected across races, ages, and clothing styles before preprocessing yields 231,176 clean images.
  • 3.1. Data Collection and Preprocessing: Preprocessing removes low-resolution, off-center, incomplete-body, extreme-pose, multi-person, and unsuitable-background images while aligning body centers.
  • 3.2. Data Statistics: SHHQ provides the largest compared dataset scale, high-definition images up to 2240 × 1920, labeled clothing attributes, and a 100% full-body ratio.
  • 3.2. Data Statistics: Figure 3 compares upper- and lower-clothing textures and lengths, gender, and ethnicity between filtered DeepFashion and SHHQ on a base-10 logarithmic y-axis.
  • 3.2. Data Statistics: Compared with filtered DeepFashion, SHHQ has a female-to-male ratio of 1.49 and increases each garment-attribute category by an average of 24.4 times.

4. Systematic Investigation

The study examines how data size, distribution, and alignment affect StyleGAN-based unconditional human generation. Experiments show that larger datasets, balanced face-orientation distributions, and full-body alignment improve specific aspects of generation quality, while texture balancing alone is insufficient.

  • Data size: 40K images marks the point beyond which additional training data yields slower and less significant quality improvements.Below 40K, image quality improves close to linearly; ADA reduces FID for datasets smaller than 40K but cannot fully compensate for insufficient data.
  • Data distribution: Uniform face-yaw sampling reduces FID for rare rotated faces while preserving acceptable quality for dominant face orientations.In the long-tailed setting, FID increases with yaw angle, whereas the uniform setting has a more gradual increase and visually better right-most samples.
  • Data distribution: Balancing clothing-texture frequencies does not consistently improve texture quality, although the uniform setting lowers FID for the rarest texture bin.The long-tailed setting performs better for the abundant plain-texture bin, while the uniform setting performs better for the final bin, likely because plaid samples are scarce in the long-tailed data.
  • Data alignment: Full-body center alignment improves generated-image quality over face-center and pelvis alignment.The compared strategies use a face bounding-box center, pelvis keypoint, or the midpoint of extreme segmentation-mask coordinates.

5. Model Zoo and Editing Benchmark

The paper releases pretrained human StyleGAN models and evaluates their use for inversion, attribute editing, style mixing, and face–body fusion. These experiments demonstrate both latent-space interpretability and practical editing potential, while revealing that human-body disentanglement remains challenging.

  • Model Zoo: The model zoo provides StyleGAN, StyleGAN2, and alias-free StyleGAN human generators at 1024 × 512 and 512 × 256 resolutions.The models are intended to complement existing facial model zoos for human generation and editing.
  • Editing Benchmark: Low latent layers transfer pose while preserving source skin color, garment color, and identity; middle layers transfer clothing type and appearance.Style-mixing associates low, middle, and high layers with coarse, middle-scale, and fine spatial resolutions.
  • Editing Benchmark: PTI can invert real full-body human images, enabling subsequent editing with pretrained human generators.The benchmark applies PTI before manipulating real images.
  • Editing Benchmark: StyleSpace changes only targeted regions more cleanly, whereas InterFaceGAN produces more natural and photo-realistic edited regions than StyleSpace and SeFa.The comparison concerns garment-length and pose-related human attribute manipulation.
  • Editing Benchmark: Existing editing results demonstrate the model zoo’s potential for human-centric tasks, but disentangling human-body attributes remains more difficult than in faces, objects, and scenes.The paper identifies human-body latent spaces as more complicated and calls for further attention to attribute disentanglement.
  • Editing Benchmark: InsetGAN combines faces from FFHQ with bodies from the baseline model by iteratively optimizing latent codes, producing diverse fused human images.The reported examples include six human postures with different male and female faces.

6. Future Work

The paper identifies future directions for using its human models and dataset in human generation, editing, neural rendering, and multimodal generation. It particularly highlights disentanglement, 3D consistency, artifact reduction, and cross-modal human manipulation as open opportunities.

  • Human Generation / Editing: Large datasets and pretrained models support human generation, editing, virtual try-on, and motion transfer, while disentangled human-body editing remains challenging.The released model zoo is positioned to expedite research in these areas.
  • Neural Rendering: Future neural-rendering work can use the human models to improve 3D consistency and mitigate artifacts in full-body human generation.The paper encourages adapting neural-rendering approaches toward multi-view-consistent human generation.
  • Multi-modal Generation: Full-body human models could support text-to-human generation and text-driven manipulation of human attributes.These are presented as examples of multimodal research directions.

7. Conclusion

The paper studies unconditional human GAN training from a data-centric perspective using SHHQ and establishes a six-model human-GAN zoo. Editing evaluations demonstrate the zoo’s potential for human-centric applications.

  • 7. Conclusion: The work analyzes data size, data distribution, and data alignment as three critical issues in unconditional human-based GAN training.The analysis uses the 230K-image SHHQ dataset and StyleGAN-based models.
  • 7. Conclusion: The authors create a model zoo containing six human-GAN models and demonstrate its effectiveness with state-of-the-art face-editing methods.The conclusion frames the editing demonstrations as evidence of the zoo’s potential.
  • 7. Conclusion: InsetGAN results show that the model zoo can support combining independently generated faces and human bodies.The figure presents six generated bodies and six FFHQ faces fused through optimization.

A. SHHQ: StyleGAN-Human Datasets

SHHQ is a large, clean, high-quality dataset of diverse clothed human images collected for unconditional human generation. Its samples vary in clothing appearance, ethnicity, and pose and are shown unprocessed.

  • A. SHHQ: StyleGAN-Human Datasets: SHHQ contains 230K high-quality full-body human images varying in clothing appearance, ethnicity, and pose.The dataset supports the paper’s data-centric study of unconditional human generation.
  • A. SHHQ: StyleGAN-Human Datasets: The collected images are unprocessed, and qualitative comparisons are used to demonstrate the dataset’s clean, high-quality data.The paper also shows training samples and generated images from a baseline model trained on SHHQ.

B. Experiment Results

The data-size experiments compare FID and IS at two resolutions and show that increasing training data improves generation quality.

  • Increasing training data improves IS scores and reduces FID scores.The comparison covers experiments at 512 × 256 and 1024 × 512 resolutions.
  • Figure 12 includes IS scores with dotted lines representing experiments using ADA strategies.
  • The data-size results are reported for experiments S0–S5.

C. Training Scheme

The training scheme adapts StyleGAN2-ADA for unconditional human generation using a 1:2 image width-to-height ratio and training from scratch.

  • The experiments use NVIDIA’s official PyTorch StyleGAN2-ADA codebase with the StyleGAN2 architecture.
  • Human inputs use a width-to-height ratio of 1:2, with the script resolution changed accordingly.
  • All experiments are trained from scratch because no pretrained human-image model is available.

D. Limitations

Unconditional human generation remains difficult because full-body semantics are more complex than facial semantics, producing artifacts in faces, hands, and clothing details.

  • Full-body human semantics are substantially more complicated than those of a single face.
  • Baseline failures include entangled face, texture, and accessory features, as well as people with three hands.
  • Rare poses can produce inferior hand quality, while face and texture quality may also require local refinement.
  • Suggested improvements include augmenting extreme poses, changing data distribution, using fine-grained discriminators, and applying local refinement.

E.2. Human Editing

The paper demonstrates human editing and style-mixing applications using StyleGAN-based models, covering multiscale attributes, orientation, and clothing lengths.

  • The figures include raw training data, diverse-resolution datasets, baseline samples, failure cases, cropped face poses, and texture patches.
  • Coarse-, middle-, and fine-resolution style-mixing transfers different high-level attributes of clothed humans.
  • Orientation editing rotates a human from the front view to the back view in W space.
  • StyleSpace-based editing changes sleeve length and bottom length.
Loading 2204.11823v1…