PaperlayerView exact revision on arXiv

Source-linked AI summary

HumanNorm: Learning Normal Diffusion Model for High-quality and Realistic 3D Human Generation

Xin Huang, Ruizhi Shao, Qi Zhang, Hongwen Zhang, Ying Feng, Yebin Liu, Qing Wang

arXiv:2310.01406v2cs.CV

TL;DR

Text-to-image diffusion models lack understanding of 3D geometry and view direction, limiting text-to-3D human generation. HumanNorm combines normal- and depth-guided geometry generation with normal-aligned texture generation. The paper reports intricate geometry and realistic appearances, outperforming existing methods, while noting rigging and shading limitations.

  • Problem

    Text-to-image diffusion models lack geometry and view-direction understanding, a limitation associated with Janus artifacts, low-quality geometry, and fake 3D details.

    Source: §3.2 · pp. 3–4

  • Method

    HumanNorm learns normal-adapted and depth-adapted diffusion models for geometry and a normal-aligned diffusion model for texture, with progressive geometry generation and multi-step SDS.

    Source: §1 · p. 2

  • Results

    HumanNorm generates 3D humans with intricate geometric details and realistic appearances, outperforming existing methods.

    Source: §6 · p. 8

  • Takeaways & Limitations

    HumanNorm models support text-based texture and geometry editing, pose editing, and animation using full-body motion sequences.

    Source: §5.5 · p. 8

    Generated humans require a rigged skeleton for animation, and generated textures may exhibit undesired shading.

    Source: §6 · p. 9

Abstract

from arXiv · show

Recent text-to-3D methods employing diffusion models have made significant advancements in 3D human generation. However, these approaches face challenges due to the limitations of text-to-image diffusion models, which lack an understanding of 3D structures. Consequently, these methods struggle to achieve high-quality human generation, resulting in smooth geometry and cartoon-like appearances. In this paper, we propose HumanNorm, a novel approach for high-quality and realistic 3D human generation. The main idea is to enhance the model's 2D perception of 3D geometry by learning a normal-adapted diffusion model and a normal-aligned diffusion model. The normal-adapted diffusion model can generate high-fidelity normal maps corresponding to user prompts with view-dependent and body-aware text. The normal-aligned diffusion model learns to generate color images aligned with the normal maps, thereby transforming physical geometry details into realistic appearance. Leveraging the proposed normal diffusion model, we devise a progressive geometry generation strategy and a multi-step Score Distillation Sampling (SDS) loss to enhance the performance of 3D human generation. Comprehensive experiments substantiate HumanNorm's ability to generate 3D humans with intricate geometry and realistic appearances. HumanNorm outperforms existing text-to-3D methods in both geometry and texture quality. The project page of HumanNorm is https://humannorm.github.io/.

1. Introduction

HumanNorm addresses the limited 3D-geometry and view-direction understanding of text-to-image diffusion models, which can yield Janus artifacts, smooth geometry, and fake 3D details. It combines normal- and depth-guided geometry generation with geometry-aligned texture generation.

Motivation

  • Text-to-image diffusion models emphasize natural RGB images and have limited understanding of geometry and view direction, contributing to Janus artifacts, smooth geometry, and fake 3D details.Existing human methods using body models still show suboptimal geometry, fake 3D details, and over-saturated texture.Source: §1 · p. 2

Geometry generation

  • HumanNorm trains a normal-adapted diffusion model on multi-view human-scan normal maps and prompts with view-dependent and body-aware text to generate high-fidelity surface normals.It also learns a depth-adapted diffusion model because normal maps lack depth information; normal and depth SDS losses guide geometry generation.Source: §1 · p. 2
  • A progressive geometry strategy is designed to reduce geometric noise and enhance geometry quality.Source: §1 · p. 2

Texture generation

  • A normal-aligned diffusion model uses normal maps as conditions to align texture with geometry, while multi-step SDS recovers images over multiple diffusion steps for more natural texture.The contributions also describe the model as transforming physical geometry details into realistic appearances and the multi-step loss as mitigating over-saturation.Source: §1 · p. 2

2. Related work

The related work reviews text-to-3D generation and human-specific approaches, positioning HumanNorm as a response to persistent geometry and texture-quality challenges.

Text-to-3D content generation

  • Text-to-3D methods have used CLIP supervision, text-to-image diffusion with SDS, latent diffusion, geometry–texture decomposition, and alternative losses to generate 3D content.Examples include DreamFusion [31], Magic3D [21], Fantasia3D [5], ProlificDreamer [48], and IT3D [6].Source: §2 · p. 2

Text-to-3D human generation

  • Figure 2 illustrates normal-adapted and depth-adapted diffusion outputs; view-dependent text controls view direction, while body-aware text controls the generated body part.Source: Figure 2 · p. 3

Text-to-3D content generation

  • Multi-view diffusion and Gaussian splatting methods are also reviewed, while HumanNorm is presented as using normal-adapted diffusion to address Janus artifacts and unreasonable body proportions.The paper attributes improved geometry and texture quality to normal diffusion and multi-step SDS.Source: §2 · p. 3

Text-to-3D human generation

  • Human-generation approaches include GAN-based representations, SMPL-based priors, pose-conditioned NeRFs, and diffusion models for personalized humans.Despite these varied approaches, the cited methods still show fake 3D details, over-saturation, and smooth geometry; SMPL can also challenge generation of intricate clothing.Source: §2 · p. 3

3. Preliminary

The preliminary section describes diffusion-guided 3D generation and identifies the mismatch between text-to-image diffusion priors and the geometry and texture distributions needed for 3D generation.

3.1. Diffusion-guided 3D Generation Framework

  • The framework aligns images rendered from a 3D representation across sampled cameras with the text-conditioned image distribution, using differentiable rendering and optimization losses.Fantasia3D disentangles geometry and appearance; its geometry stage aligns rendered normal maps with natural-image distributions, and its texture stage optimizes appearance through Eq. (1).Source: §3.1 · p. 3

3.2. Bottleneck of Diffusion-guided 3D Generation

  • Text-to-image diffusion models represent natural RGB images but lack view-direction and geometry understanding, leading to Janus artifacts and low-quality geometry.Aligning normal maps directly with natural-image distributions can distort geometry, while texture optimization without geometric guidance can produce fake 3D details.Source: §3.2 · pp. 3–4

4. Method

HumanNorm separates text-to-3D human generation into geometry and texture stages, using normal-conditioned diffusion to align generated geometry and appearance.

4. Method

4.1. Normal Diffusion Model

  • HumanNorm adapts text-to-image diffusion to model prompt-conditioned normal maps for geometry, then conditions image generation on normals so texture aligns with geometry.Source: §4.1 · p. 4

4.2.1 Normal-adapted Diffusion Model

  • Camera-coordinate normal maps and view-dependent, body-aware prompts train the normal-adapted model; normal SDS optimizes geometry, while depth SDS can reduce distortion and artifacts.Source: §4.2.1 · p. 5

4.2.2 Progressive Geometry Generation

  • Progressive positional encoding suppresses high frequencies early and restores them during training; progressive SDF loss penalizes deviation from earlier SDF values to mitigate deformations and unreasonable proportions.Source: §4.2.2 · p. 5

4.3.1 Normal-aligned Diffusion Model

  • With geometry fixed, normal maps condition image diffusion through ControlNet-style guidance, aligning texture with physical details; this model underlies multi-step SDS for photorealistic texture.Source: §4.3.1 · pp. 5–6

4.3.2 Multi-step SDS Loss

  • Vanilla SDS initializes texture, then multi-step SDS refines it to recover RGB-image distributions and improve optimization stability; perceptual loss keeps render style consistent with generated images.Source: §4.3.2 · p. 6

5. Experiment

HumanNorm is evaluated qualitatively, quantitatively, through ablations, and in editing and animation applications. The reported comparisons favor its geometry and texture quality, while the ablations describe the contributions of its diffusion models and losses.

5.1. Implementation Details

  • Generation takes about two hours per prompt on one NVIDIA RTX 3090 GPU, with 15K geometry iterations and 10K texture iterations.Final rendered images and videos have 1024 × 1024 resolution.Source: §5.1 · p. 6

5.2. Qualitative Evaluation

  • Against text-to-3D content and human methods, HumanNorm produces more accurate geometry, more detailed wrinkles and facial features, and more realistic textures.The comparison notes that DreamFusion often misses feet, while methods using SMPL-X and imGHUM improve geometry but show less detail and over-saturation.Source: §5.2 · p. 7, Figure 6 · p. 7

5.3. Quantitative Evaluation

  • HumanNorm achieves lower FID and a superior CLIP score than compared methods, indicating closer alignment with generated images and prompts.The quantitative evaluation compares 120 rendered or generated images for each of 30 prompts; Table 1 reports the comparisons.Source: §5.3 · p. 7, Table 1 · p. 7

5.4. Ablation Studies

  • Replacing normal- and depth-adapted diffusion with text-to-image diffusion yields weaker facial geometry, ear holes, and smoother clothing wrinkles; removing normal-aligned diffusion blurs texture details.Using only SDS produces over-saturated color, whereas multi-step SDS yields more natural color.Source: §5.4 · p. 8, Figure 7 · p. 8
  • Normal-only SDS can leave ear artifacts because normal maps lack depth information; adding depth SDS reduces those artifacts.The depth map exposes artifacts that are not salient in the normal map.Source: §5.4 · p. 8, Figure 8 · p. 8

5.5. Applications

  • HumanNorm supports prompt-based edits to texture, geometry, and pose, and animates generated meshes using full-body motion sequences.The animated models have about 400K faces and 2K-resolution texture maps.Source: §5.5 · p. 8, Figure 9 · p. 8

6. Conclusion

HumanNorm combines normal diffusion with progressive geometry generation and multi-step SDS to generate detailed, realistic 3D humans. The authors note that animation requires a rigged skeleton and that textures may exhibit undesired shading.

6. Conclusion

  • HumanNorm uses normal diffusion to improve 2D diffusion models for 3D human generation, alongside progressive geometry generation and multi-step SDS.The conclusion reports intricate geometric details and realistic appearances, outperforming existing methods.Source: §6 · p. 8
  • The authors report that HumanNorm generates 3D humans with intricate geometry and realistic appearances, outperforming existing methods.The conclusion attributes addressing over-saturation to multi-step SDS and robust geometry generation to the progressive strategy.Source: §6 · p. 8
  • Animation requires a rigged human skeleton, and generated textures may exhibit undesired shading.Future work plans to incorporate SMPL-X for direct animation and PBR for material estimation and relighting.Source: §6 · p. 9

3D Human Generation

The paper describes SDF and DMTET as 3D representations and uses DMTET in both geometry and texture generation.

3D Human Generation

  • An SDF represents an object surface implicitly: f(x) = 0 is the surface, f(x) < 0 the interior, and f(x) > 0 the exterior.The paper describes SDFs as represented through neural networks such as MLPs.Source: A. 3D representations · p. 11
  • DMTET combines a dense tetrahedral partition with neural-network functions that encode vertex properties such as SDF, deformation, and color.The representation can be converted differentiably into a mesh for multi-view rendering.Source: A. 3D representations · p. 11
  • HumanNorm uses DMTET as its 3D representation in both geometry generation and texture generation.Source: A. 3D representations · p. 11

B. Implementation Details

The implementation trains diffusion models on rendered human data and progressively adjusts geometry optimization, then refines textures with multi-step diffusion and perceptual loss.

B. Implementation Details

  • The dataset contains 2,952 human models from THuman2.0, Twindom, and CustomHumans, rendered into 120 sets of depth, normal, and color maps per model.The body is divided into head, upper body, lower body, and full body; prompts include view-dependent and body-aware text.Source: B. Implementation Details · p. 11
  • Stable Diffusion V1.5 is fine-tuned for 15K iterations on normal and depth pairs, then for 30K iterations on normal-image pairs for normal-aligned generation.The normal maps condition the latter model.Source: B. Implementation Details · p. 11
  • Progressive positional encoding begins with 16 of 32 dimensions enabled, adding two dimensions every 500 iterations until all are active after 4,000 iterations.The procedure gradually enables higher-frequency SDF features.Source: B. Implementation Details · p. 11
  • After 3,000 iterations, progressive SDF loss compares network predictions with a coarse mesh at 100,000 sampled points per iteration.Its loss weight is 1,500 and it is computed only after the 3,000-iteration mark.Source: B. Implementation Details · pp. 11–12
  • Geometry resolution increases from 128^3 to 256^3 at 3,000 iterations and 512^3 at 6,000, supporting more detailed geometry as training proceeds.The authors state that lower early resolution stabilizes geometry generation, while later increases allow details such as hair and clothing folds.Source: B. Implementation Details · p. 12
  • Texture generation uses SDS for 2,000 coarse iterations, then multi-step SDS and perceptual loss for 8,000 fine-level iterations.Multi-step diffusion iterates t/25 + 1 times for timestep t; denoised images are cached and updated every 10 iterations.Source: B. Implementation Details · p. 12
  • Geometry uses normal and depth SDS weights of 1.0 each; texture uses a perceptual-loss weight of 1.0.The geometry and texture stages use different guidance scales and progressively changing noise-timestep ranges.Source: B. Implementation Details · p. 12
  • For full-body generation, camera sampling shifts from mostly full-body views in the first 10,000 iterations to mostly part views in the next 5,000.The first phase assigns 0.7 probability to full body; the next assigns 0.1, with 0.3 each for head, upper body, and lower body.Source: B. Implementation Details · pp. 12–13

C. User Study

The user study compares HumanNorm with five state-of-the-art methods across 30 prompts using volunteer judgments of texture, geometry, and prompt alignment. HumanNorm surpasses the compared text-to-3D content and human methods, particularly for geometry and texture quality.

C. User Study

  • Fifty volunteers evaluated color and normal-map videos across 30 prompts, judging texture quality, geometric quality, and prompt alignment.The participants included 40 computer vision and graphics students and 10 members of the general public.Source: C. User Study · p. 13
  • HumanNorm surpasses the compared text-to-3D content and human methods, particularly in geometry and texture quality.The study collected 1,500 pairwise comparisons.Source: C. User Study · p. 13

D. More Comparisons

Qualitative comparisons report that HumanNorm produces more detailed geometry and realistic appearances across head-only, upper-body, and full-body generation, with textures aligned to geometry.

D. More Comparisons

  • Against four text-to-3D content methods, HumanNorm produces head-only and upper-body humans with more detailed geometry and realistic appearances.The comparison notes that Fantasia3D can produce textures that are not aligned with geometry, unlike HumanNorm.Source: D. More Comparisons · p. 13
  • In full-body comparisons with DreamHuman and TADA, HumanNorm yields more natural appearances and geometric detail than baselines with over-saturated textures and smooth geometry.Source: D. More Comparisons · p. 13
  • Compared with AvatarVerse, HumanNorm produces more lifelike appearances, while AvatarVerse results are described as over-saturated.Source: D. More Comparisons · p. 13

E. More Ablation Studies

The ablations associate progressive SDF loss with constrained body growth and progressive positional encoding with reduced geometry noise. The paper also describes prompt-based editing of generated humans' texture and geometry.

E. More Ablation Studies

  • Progressive SDF loss constrains incorrect human-body growth and avoids unreasonable body shapes.Without it, the generated human exhibits a distorted body shape.Source: E. More Ablation Studies · p. 13
  • Progressive positional encoding reduces high-frequency learning early in training, yielding stable geometry without surface noise.Fixed-frequency hash encoding produces extensive surface noise attributed to high-frequency content learned during initial training.Source: E. More Ablation Studies · p. 14

F. Applications

  • Changing the input prompt allows edits to both texture and geometry, including clothing color and style, hairstyle, and adding a hat while maintaining Messi's identity.The paper notes that geometry editing is more challenging than texture editing, but reports precise geometry control and detailed edits.Source: F. Applications · p. 14

G. Ethics statement

The ethics statement notes that generated human models could be misused to deceive viewers and calls for continued attention to gender, race, and cultural diversity.

G. Ethics statement

  • Generated models could be misused to deceive viewers, a risk the paper says is also present in other generative-model methodologies.Source: G. Ethics statement · p. 14
  • The paper emphasizes giving precedence to gender, race, and cultural diversity in current and future generative-model research.Source: G. Ethics statement · p. 14

Related papers

Loading 2310.01406v2…