PaperlayerView exact revision on arXiv

Source-linked AI summary

EVA3D: Compositional 3D Human Generation from 2D Image Collections

Fangzhou Hong, Zhaoxi Chen, Yushi Lan, Liang Pan, Ziwei Liu

arXiv:2210.04888v1cs.CV

TL;DR

Generating articulated 3D humans from 2D collections remains challenging because human poses and appearances are diverse, and image collections can have imbalanced poses and viewing angles. EVA3D learns unconditional 3D humans from sparse 2D collections using a 16-part compositional human NeRF with a human-body prior and pose-guided sampling. EVA3D achieves state-of-the-art generation results on four large-scale 2D human datasets at 512 × 256.

  • Problem

    Generating articulated 3D humans from 2D collections remains challenging because human poses and appearances are diverse, and image collections can have imbalanced poses and viewing angles.

    Source: §1 · p. 2

  • Method

    EVA3D learns unconditional 3D humans from sparse 2D collections using a 16-part compositional human NeRF with a human-body prior and pose-guided sampling.

    Source: §1 · p. 2, §3.4 · p. 6

  • Results

    EVA3D achieves state-of-the-art generation results on four large-scale 2D human datasets at 512 × 256.

    Source: §5 · p. 10

  • Takeaways & Limitations

    The authors demonstrate latent interpolation and GAN inversion as applications that pave the way for further exploration in 3D human GANs.

    Source: §1 · p. 3, §4.5 · p. 9

    Estimated SMPL parameters are inaccurate, creating a distribution shift from real poses that may compromise generation results.

    Source: §5 · p. 10

Abstract

from arXiv · show

Inverse graphics aims to recover 3D models from 2D observations. Utilizing differentiable rendering, recent 3D-aware generative models have shown impressive results of rigid object generation using 2D images. However, it remains challenging to generate articulated objects, like human bodies, due to their complexity and diversity in poses and appearances. In this work, we propose, EVA3D, an unconditional 3D human generative model learned from 2D image collections only. EVA3D can sample 3D humans with detailed geometry and render high-quality images (up to 512x256) without bells and whistles (e.g. super resolution). At the core of EVA3D is a compositional human NeRF representation, which divides the human body into local parts. Each part is represented by an individual volume. This compositional representation enables 1) inherent human priors, 2) adaptive allocation of network parameters, 3) efficient training and rendering. Moreover, to accommodate for the characteristics of sparse 2D human image collections (e.g. imbalanced pose distribution), we propose a pose-guided sampling strategy for better GAN learning. Extensive experiments validate that EVA3D achieves state-of-the-art 3D human generation performance regarding both geometry and texture quality. Notably, EVA3D demonstrates great potential and scalability to "inverse-graphics" diverse human bodies with a clean framework.

1 INTRODUCTION

Generating articulated 3D humans from 2D images is challenging because bodies vary in pose and appearance, while available image collections can have imbalanced poses and views. EVA3D addresses these challenges with a compositional human NeRF and training strategies for sparse 2D collections.

1 INTRODUCTION

  • Human generation from 2D collections remains difficult because articulated bodies combine complex poses with diverse appearances, unlike rigid objects modeled by recent 3D-aware generators.A useful representation must control pose and shape, and efficiency matters because prior human representations failed to reach high-resolution generation.Source: §1 · p. 2
  • Fashion datasets offer more varied people and clothing than AIST, but their limited poses and front-view imbalance can hinder novel-pose and novel-view synthesis.The authors propose training strategies to address this imbalance.Source: §1 · p. 2
  • EVA3D renders at 512×256 without super-resolution, using a 16-part compositional human NeRF that allocates parameters locally and avoids sampling empty space.The authors report that existing methods train at a native resolution of 1282.Source: §1 · p. 2
  • EVA3D uses SMPL skinning weights to transform between canonical and observation spaces, and a geometry template with predicted offsets to support geometry learning.The authors report robustness to dataset pose distributions and better generalizability to novel poses.Source: §1 · p. 2
  • Experiments cover two fashion datasets and comparisons with prior work on UBCFashion and AIST; the authors also demonstrate interpolation and GAN inversion applications.They describe these applications as paving the way for further exploration in 3D human GANs.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans 3D-aware generation methods, human generation from 2D collections, and representations for 3D humans. The cited approaches include voxel, mesh, NeRF, neural-renderer, and parametric or compositional representations.

3D-Aware GAN

  • 3D-aware generators have used voxels, meshes, and NeRFs; some use 2D decoders for super-resolution, while other work seeks higher raw rendering resolution for geometry and 3D consistency.EVA3D is positioned as an efficient human representation for high-resolution training.Source: §2 · p. 3

Human Generation

  • For human generation, prior 2D-image-trained 3D GANs include CNN-based neural renderers without guaranteed 3D consistency, low-resolution human NeRFs, and super-resolution approaches that still produce low-quality results.Other cited work generates geometry from 3D datasets or avatars from text inputs.Source: §2 · p. 3

3D Human Representations

  • Human representations range from parametric models with robust control but less realism to implicit functions, human NeRFs, and compositional models that use submodules for body parts.The passage also notes work on UV maps for appearance modeling, modal-invariant representations, and a multimodal 4D dataset.Source: §2 · p. 3

3 METHODOLOGY

EVA3D combines a compositional human NeRF with SMPL-guided rendering and a GAN trained from 2D human image collections. Its training uses template-based SDF prediction and pose-guided sampling to address sparse, imbalanced pose data.

3.1 PREREQUISITES

  • SMPL defines body shape and pose, while NeRF represents a scene through emitted radiance and volume density for rendering rays into RGB values.EVA3D uses SMPL's linear blend skinning to transform between canonical and observation spaces.Source: §3.1 · pp. 3–4

3.2 COMPOSITIONAL HUMAN NERF REPRESENTATION

  • The compositional representation assigns local bounding boxes to body-part subnetworks, blending outputs where boxes overlap and querying only networks containing each point.Filtering rays with predefined boxes avoids background computation and supports high-resolution training.Source: §3.2 · p. 4
  • For each target shape, pose, and camera, EVA3D keeps rays intersecting posed boxes, samples points, and maps them to canonical space using inverse LBS.The mapping also inverts shape and pose blend shapes and uses nearby posed SMPL vertices.Source: §3.2 · p. 5

3.3 3D HUMAN GAN FRAMEWORK

  • The GAN renders latent-conditioned humans using shape, pose, and camera distributions estimated from 2D collections, then adversarially trains against real images.The estimated distributions reflect the image collection rather than a motion dataset.Source: §3.3 · p. 5

3.4 TRAINING

  • EVA3D predicts an SDF offset from SMPL template geometry, using the template as a human prior for learning from imbalanced 2D poses.The offset is added to the template SDF to obtain the point's SDF value.Source: §3.4 · p. 6
  • Pose-guided sampling weights head-angle bins with a front-centered Gaussian; the resulting distribution increases side-view exposure and helps geometry generation.The sampling strategy gives front views greater weight because facial areas contain more information than other head regions.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments compare EVA3D with existing methods and examine its design through ablations, then demonstrate latent interpolation and image inversion. EVA3D leads all reported metrics across four datasets, while the ablations show a trade-off between visual quality and geometry.

4.1 EXPERIMENTAL SETUP

  • Evaluation spans sparse image collections DeepFashion and SHHQ, video datasets UBCFashion and AIST, and image-quality, pose, and geometry metrics.The baselines are ENARF-GAN, EG3D, and StyleSDF; PCKh@0.5 is evaluated only for methods that control generated poses.Source: §4.1 · p. 7

4.3 QUANTITATIVE EVALUATIONS

  • EVA3D leads all metrics in four datasets; it also achieves the best FID/KID under all settings, higher PCKh@0.5 than ENARF-GAN, and the lowest depth error.The comparison covers state-of-the-art methods.Source: Table 1 · p. 8, §4.3 · p. 8

4.2 QUALITATIVE EVALUATIONS

  • EVA3D generates high-quality, multi-view-consistent renderings and controllable poses and shapes; baselines show distorted, billboard-like, or broken geometry across the comparisons.The qualitative comparison includes DeepFashion and SHHQ; results for UBCFashion and AIST are in the supplementary material.Source: §4.2 · p. 8

4.4 ABLATION STUDIES

  • Pose-guided Gaussian sampling balances image quality and geometry: increasing σθ raises FID while lowering depth error, and the final experiments use σθ = 15°.Uniform sampling has high FID, while the original distribution gives the best visual quality but the worst geometry.Source: Table 3 · p. 9, §4.4 · p. 9, Figure 6 · p. 9
  • Compositional design enables training at 512×256 rather than the baseline’s 256×128; adding delta-SDF prediction improves FID and depth error, while pose-guided sampling further lowers depth error.The pose-guided variant slightly increases FID.Source: Table 2 · p. 9, §4.4 · p. 9

4.5 APPLICATIONS

  • Linear interpolation between latent codes produces smooth transitions; PTI inversion yields reasonable novel views, though its geometry misses details visible in RGB renderings.The geometry mismatch may be caused by PTI’s second-stage generator fine-tuning.Source: §4.5 · p. 9, Figure 7 · p. 9

5 DISCUSSION

EVA3D reports state-of-the-art generation on four large-scale 2D human datasets at 512 × 256. The authors also identify rendering, pose-estimation, clothing, and ethical limitations.

5 DISCUSSION

  • EVA3D achieves state-of-the-art generation results on four large-scale 2D human datasets at 512 × 256.The authors describe the results as high-resolution.Source: §5 · p. 10
  • The authors report circular rendering artifacts, potentially inaccurate SMPL estimates that may compromise results, and difficulty modeling loose clothes.They suggest refining SMPL estimates and using separate modules for loose clothes as future directions.Source: §5 · p. 10
  • Generated humans may be misused for misleading content or fake videos of real people, and may reflect biases in training-data distributions.The authors state that they demonstrate EVA3D's impartiality in Fig. 1.Source: ETHICS STATEMENT · p. 10
  • The authors state that reproducibility is ensured by the method description and supplementary implementation details, and that code will be released upon acceptance.Source: REPRODUCIBILITY STATEMENT · p. 10

A APPENDIX

The appendix provides supplementary dataset and implementation information, additional visual results and comparisons, and a demo video.

A APPENDIX

  • The supplement provides more information about the training datasets.Source: A APPENDIX · p. 15
  • The supplement provides additional implementation details.Source: A APPENDIX · p. 15
  • The supplement includes more visual results and comparisons, along with a demo video.Source: A APPENDIX · p. 15

A.1 DATASETS

The paper trains and evaluates on four datasets with differing sample counts, resolutions, pose sources, and clothing characteristics.

A.1 DATASETS

  • DeepFashion contributes 8,036 full-body images without dresses, resized to 512 × 256, with SMPL and camera parameters estimated using SMPLify-X.The paper aligns the bodies as proposed by Jiang et al. (2022c).Source: §A.1 · p. 15
  • SHHQ contributes 120,865 images at 512 × 256; SMPLify-X estimates perform better than SPIN estimates, and head directions are heavily imbalanced.The proposed pose-guided sampling distribution is shown alongside the head-direction distribution in Fig. 8a.Source: §A.1 · p. 15
  • UBCFashion contributes 192,179 cropped 512 × 256 frames from 500 video sequences, most featuring dresses, with SMPL sequences estimated using VIBE.Despite most models spinning in front of the camera, head directions remain heavily imbalanced.Source: §A.1 · p. 15
  • AIST contributes 72,000 samples from a multiview dancing dataset with rich poses and accurate SMPL estimates, resized to 256 × 256.The paper uses processing scripts provided by ENARF-GAN (Noguchi et al., 2022).Source: §A.1 · p. 15

A.2.1 NETWORK ARCHITECTURE

EVA3D uses separate subnetworks for body parts, with architecture and capacity choices varying across parts.

A.2.1 NETWORK ARCHITECTURE

  • Figure 8 presents head-angle distributions for SHHQ and UBCFashion.Source: Figure 8 · p. 15
  • The human body is divided into 16 parts, each assigned a subnetwork developed based on StyleSDF.Source: §A.2.1 · p. 15
  • Each subnetwork alternates stacked MLP and FiLM SIREN activation layers, then uses separate branches to estimate SDF and RGB values.Source: §A.2.1 · p. 15
  • The number of network layers varies empirically by body part, with the specific counts shown in Fig. 9a.Source: §A.2.1 · p. 15

A.2.2 TRAINING SETTINGS

The training setup specifies optimizer and sampling hyperparameters, an empirically scheduled R1 penalty, and small augmentations to address SMPL estimation errors. It also reports the model’s training and inference resources.

Hyperparameters

  • The generator and discriminator use Adam with learning rates 2 × 10−5 and 2 × 10−4, respectively; loss weights are λoff = 1.5 and λeik = 0.5.The setup samples 28 query points per ray and uses σθ = 15◦ for pose-guided sampling.Source: §A.2.2 · p. 15

Architecture

  • The body is divided into 16 parts, with the corresponding subnetwork layer counts shown alongside the architecture of each part’s subnetwork.Figure 9 illustrates the part definitions and per-part subnetworks.Source: Figure 9 · p. 16

R1 Scheduler

  • R1 regularization decreases empirically from 300 to 18.5, with its value halved every 50,000 iterations.The scheduler balances the risk of discriminator overfitting under low R1 against reduced generation quality under high R1.Source: §A.2.2 · pp. 15–16

Augmentation

  • Small random panning, scaling, and rotation augment real and fake samples to compensate for inaccurate SMPL estimates.The augmentations are applied before samples are sent to the discriminator.Source: §A.2.2 · p. 16

Runtime Analysis

  • Training takes five days on eight NVIDIA V100 GPUs with batch size 8, and inference runs at approximately 5 FPS on one V100.The passage reports test-time speed on a single NVIDIA V100 GPU.Source: §A.2.2 · p. 16

A.3 MORE QUALITATIVE RESULTS

Qualitative comparisons show EVA3D producing detailed human renderings and meshes across datasets, while ablations illustrate the effects of compositional representation, human priors, and pose-guided sampling.

Visual Comparison on UBCFashion & AIST

  • EVA3D gives the best visuals on UBCFashion and the best RGB renderings plus detailed meshes on AIST, where StyleSDF fails and EG3D misses structure.On UBCFashion, EG3D and StyleSDF render reasonably but their meshes lack detail; EVA3D preserves details such as brims on AIST.Source: §A.3 · p. 16, Figure 10 · p. 17

Qualitative Evaluations on Ablation Studies

  • The 512 × 256 composite model enables high-resolution training, while adding a template and Delta SDF improves geometry; pose-guided sampling yields plausible geometry, unlike uniform sampling’s head artifacts.The baseline trained at 256 × 128 is blurry and has broken knees; without a human prior, the composite model produces unreasonable upper-body wrinkles.Source: §A.3 · p. 16, Figure 11 · p. 18

More Qualitative Results of EVA3D

Loading 2210.04888v1…