DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance

1 INTRODUCTION

• DreamFace targets prompt-driven facial assets that represent desired human traits while remaining compatible with CG production pipelines.Prior approaches produced realistic 2D faces or 3D-aware renderings, but could lack facial detail, animation, or physically based textures and were difficult to integrate into production workflows.Source: §1 · p. 2
• The authors present the system as a way for novices to create and customize text-guided facial assets with desired shapes, physically based textures, and animation capabilities.They also describe the resulting assets as compatible with existing graphics production pipelines.Source: §1 · pp. 2–3
• Its geometry pipeline selects a coarse face from a unified-topology candidate pool, then uses generic-LDM Score Distillation Sampling to refine displacements and normal maps.The coarse-to-fine process is followed by appearance generation using a generic LDM and a texture LDM, with SDS optimization in latent and image spaces.Source: §1 · pp. 2–3
• DreamFace uses a cross-identity hypernetwork to learn a universal expression prior and support personalized, video-based facial animation.A neural tracker encodes footage into the learned expression space; the neutral assets also support blendshape animation.Source: §1 · p. 3

Face modeling and generation

• Explicit meshes offer production compatibility, while implicit face representations can achieve detail but are harder to integrate into CG pipelines; DreamFace uses an ICT-FaceKit mesh with displacement and tangent-space normals.Earlier explicit methods can produce oversmoothed geometry, motivating detail refinement on the compatible mesh.Source: §2 · p. 3

Texture generation

• Texture-generation methods face limits in realism, capture-data availability, cross-dataset consistency, and GAN-based text conditioning; DreamFace proposes diffusion with prompt tuning to bridge uneven training data.The paper describes self-supervised UV methods as short of ultra-realistic rendering and supervised methods as requiring difficult-to-assemble datasets.Source: §2 · p. 3

Text to 3D generation

• Prior text-to-3D approaches use CLIP or diffusion with rendered representations, while DreamFace adds a learned texture prior to dual-path SDS to address insufficient 3D quality.The related-work discussion notes that some approaches suffer from two-face Janus problems.Source: §2 · p. 4

Facial Animation

• Facial animation work ranges from generic blendshapes and tracking to personalized retargeting; DreamFace adopts a cross-identity hypernetwork and image expression encoder for its generated assets.The approach is inspired by the Universal Prior Model for personalized expressions from unseen identities.Source: §2 · p. 4

3 OVERVIEW

• DreamFace separates text-prompt generation into geometry, physically based texture, and animation modules.The overview describes generated assets as resembling prompted shape and appearance and compatible with graphics production.Source: §3 · p. 4, Figure 3 · p. 5
• Geometry generation selects a CLIP-matched coarse face, then applies generic-LDM SDS to rendered views to refine vertex displacement and normal-map details.The overview caption specifies random view and lighting conditions for SDS.Source: Figure 3 · p. 5, §3 · p. 5
• Appearance generation combines a generic LDM for diversity with a UV-space texture LDM, using two-stage SDS in latent and image spaces before mapping to physically based assets.The pipeline also uses super-resolution to produce 4K textures.Source: §3 · p. 5
• Neutral assets support existing facial trackers and blendshapes, while a cross-identity hypernetwork and neural video tracker enable personalized, video-based animation.The hypernetwork learns a universal expression prior and transforms neutral geometry images into expressive meshes.Source: §3 · p. 5

4 GEOMETRY GENERATION

• The coarse-to-fine geometry process starts from ICT-FaceKit, whose topology has 14,062 vertices and 28,068 faces, and selects a prompt-matched candidate from its shape space.The candidate pool is randomly sampled; the described shape space uses 100 basis components and candidates are sampled from a multivariate normal distribution.Source: §4 · pp. 5–6, §4.1 · p. 6

4.1 Coarse geometry generation

• CLIP ranks candidate meshes using rendered front and three-quarter views under varied lighting, and the highest-scoring candidate becomes the coarse head mesh.The described comparison uses 30 images per candidate and a relative matching score based on image and prompt embeddings.Source: §4.1 · p. 6

4.2 Detail carving

• Detail carving adds vertex displacements and tangent-space normal maps, with generic-LDM SDS gradients from rendered faces guiding the prompt-conditioned refinement.The displacement corrects the coarse mesh, while the normal map adds rendered detail such as deep wrinkles; camera poses span an arc from 45° left to 45° right, with lighting sampled over the frontal hemisphere.Source: §4.2 · p. 6
• The detail-carving objective combines SDS with shape, geometry, and normal-map regularization, producing geometry described as prompt-matched and suitable for subsequent appearance generation.The regularizers address mesh smoothness and normal-map gradients and divergence.Source: §4.2 · pp. 6–7

4.3 Hair selection

• Hair selection uses CLIP to choose among 16 artist-created hairstyles, then selects a predefined color by rendering the hair on the generated head.The selected hairstyle and color are intended to match the input prompt.Source: §4.3 · p. 7

5 PHYSICALLY-BASED TEXTURE DIFFUSION

• A generic LDM and a texture LDM jointly optimize appearance, balancing diverse prompt generation with UV texture specifications and consistency with the predicted geometry.Two-stage SDS operates in latent and image spaces; the latent stage provides compact priors for fine-grained synthesis.Source: §5 · p. 7

5.1 Learning diffusion model in texture space

• Fine-tuning a pretrained LDM on a diverse UV texture dataset supervises conformity to texture specifications while retaining generating ability.Diffusion occurs in latent space to reduce the computational cost of operating at texture resolution.Source: §5.1 · p. 7

5.1.1 Data Collection.

• The texture dataset combines captured, public, and commercial sources, with expert unification and annotation to address differing UV layouts, artistic edits, and lighting conditions.It includes diffuse, specularity, and normal maps, paired with annotated text prompts.Source: §5.1.1 · p. 8

5.1.2 Prompt Tuning.

• Learned continuous domain prompts distinguish desired diffuse maps from maps with unwanted lighting, allowing training on both domains while targeting the desired domain.Non-face region masks condition the LDM to avoid unwanted elements in generated maps.Source: §5.1.2 · p. 8, §5.1.1 · p. 8

5.2 Two-stage Dual-path Appearance Optimization

• Two-stage dual-path SDS uses a generic LDM for diversity and a texture LDM for UV specifications, first optimizing latent texture and then refining it in image space.The image-space stage applies detailed normal maps and random lighting to enhance appearance details while disentangling lighting from diffuse maps.Source: §5.2 · p. 9

5.2.1 Latent space SDS.

• In latent-space SDS, the generic path operates on rendered latent codes while the texture path operates on texture latent codes, and their losses are combined to optimize the texture.The method decreases the shared denoising time step from its maximum to zero rather than sampling it uniformly.Source: §5.2.1 · pp. 9–10

5.2.2 Image space SDS.

• Image-space SDS decodes the texture latent into a diffuse map, renders it with geometry, a detailed normal map, and random lighting, then refines the latent code.The resulting code is decoded into a high-quality diffuse map that retains the texture-space prior while adding detail.Source: §5.2.2 · p. 10

5.3 Physically-based textures generation

• The physically based texture pipeline infers specularity and normal maps from the diffuse texture using decoders trained on 370 tuples of captured maps.The maps are enhanced and upscaled to 4096 × 4096 resolution for rendering.Source: §5.3 · p. 10

Blendshape animation

• Consistent topology supports traditional blendshape-based animation using expression parameters obtained from images or videos.DreamFace also targets personalized expressions that preserve each generated asset’s distinctive characteristics.Source: §6 · pp. 10–11

Personalized expression generation

• The cross-identity geometry hypernetwork generates person-specific expressions and serves as a universal prior, unlike methods trained from extensive captures for one identity.It learns from geometries spanning multiple identities and expressions.Source: §6 · pp. 11–12
• A unified expression code combines with a specific identity’s neutral geometry to generate its corresponding expressive geometry.The generator is conditioned on neutral geometry, and training uses geometries with varied identities and expressions.Source: §6 · pp. 11–12

Video-driven animation

• An image expression encoder extracts unified expression codes from RGB images, while the trained geometry generator is frozen during encoder training.Training uses supervision from real images and randomly rendered images; the encoder can extract codes from in-the-wild videos.Source: §6 · p. 12

Dataset

• The geometry training dataset contains static scans of varied identities and expressions plus dynamic performance sequences.It comprises 38,400 registered meshes and 614,400 images from 300 identities across genders, ages, and ethnicities.Source: §6 · pp. 12–13

Experiments

• The experimental evaluation covers DreamFace’s modules qualitatively and quantitatively, compares with state-of-the-art methods, and includes a user study.The section also presents implementation details and a gallery of generated assets.Source: §7 · p. 13

Texture editing

• Prompts support global texture edits such as aging and makeup, while masks or sketches enable tattoos, beards, and birthmarks.The examples use the trained texture LDM.Source: Figure 11 · p. 14

Video-driven animation

• Video-driven animation examples pair input RGB images with personalized driven results, with personalized expressions produced from a single image.Source: Figure 12 · p. 14

Models and training

• The generic LDM is Stable Diffusion v1.5 from RunwayML, and the CLIP checkpoint is clip-vit-large-patch14 from OpenAI.The texture LDM is made by fine-tuning a duplicate of the generic LDM for 150 epochs, taking about 12 hours on two Nvidia A6000 GPUs.Source: §7.1 · p. 14

Generation runtime

• About 5 minutes on a single Nvidia A6000 GPU suffices to generate a high-quality facial asset with the two-stage SDS design.The pipeline samples one million geometry candidates, performs 300 detail-carving steps, and uses 200 latent-space and 200 image-space optimization steps.Source: §7.1 · p. 14

Texture LDM evaluation

• The texture LDM variants with the two described designs produce the best-quality textures and the lowest KID in the desired texture domain.The figure’s qualitative comparison highlights lighting and unwanted-element removal; its lower table reports KID.Source: Figure 13 · p. 15

Generated assets

• DreamFace generates realistic, detailed characters from simple prompts, including fashion icons and fictional or exotic characters, with photo-realistic rendering in modern CG pipelines.Generated assets can capture described characteristics and retain high recognition for fictional characters.Source: §7.2 · p. 15

Texture editing

• Prompts can edit existing textures, while mixing the texture LDM with a generic LDM generates patterns from noise or sketches in masked areas that blend with the original diffuse maps.Prompt-guided denoising supports effects such as wrinkles and makeup.Source: §7.2 · p. 15

Geometry detail

• Detail carving captures character-specific geometry, including Scarlett Johansson’s mouth and a wide Na’vi-style nose bridge.The figure compares coarse and detailed geometry in close-up and full views.Source: Figure 14 · p. 15

Appearance generation

• Appearance latent-code generation with SDS and the texture LDM produces diffuse maps with precise components in consistent UV space.The figure compares two variations and their rendered results.Source: Figure 15 · p. 15

Animation

• From an input RGB image, the animation pipeline generates personalized expressions and restores detailed performance for a generated facial asset.The comparison with DECA highlights fine-grained expression detail, including a left sneer.Source: §7.2 · p. 16, Figure 17 · p. 16

Comparison with prior methods

• Compared with previous prompt-controlled methods, DreamFace preserves more appearance detail while retaining fine-grained geometry, and achieves the best text-matching score and least running time.The comparison includes generated examples and quantitative results.Source: Figure 18 · p. 16

7.3.1 Evaluation of texture LDM.

• The full texture-LDM design produces high-quality textures, while variants without prompt tuning or masking retain unwanted elements or inconsistent skin tones.The evaluation generated 1,000 samples per setting and used KID; the full design combines prompt tuning and masking.Source: §7.3.1 · pp. 16–17

7.3.7 User study.

• In the user study, the upper table reports specific-character resemblance, with 72.3% for celebrity cases and 71.6% for out-of-distribution cases.Source: §7.3.1 · p. 17, Figure 19 · p. 17

7.3.2 Detail carving.

• Detail carving captures individual characteristics and produces distinct facial features across species compared with coarse geometry.The passage compares geometry with and without carving.Source: §7.3.2 · p. 17

7.3.3 Ablation study of texture diffusion.

• The full texture-diffusion pipeline meets the physically based rendering standard, unlike variants with noisy maps, UV-space drift, or lighting.The compared variants omit latent-space SDS and the texture LDM, or omit the texture LDM.Source: §7.3.3 · p. 17

7.3.4 Texture augmentation.

• Texture augmentation provides high-resolution diffuse, specularity, and normal maps described as essential for photorealistic rendering.The module follows generation of a satisfactory texture latent code.Source: §7.3.4 · p. 17

7.3.5 Comparison of animation generation.

• Compared with DECA, DreamFace provides personalized expressions and more accurately restores detailed expressions from video.The comparison includes DECA, transferred DECA expression parameters on the generated asset, and DreamFace’s enhanced animation scheme.Source: §7.3.5 · p. 17

7.3.6 Comparison with state-of-the-art methods.

• DreamFace outperforms Text2Mesh, AvatarCLIP, and DreamFusion in both CLIP score and running time.The comparison generated 10 characters per method using the prompt “the realistic face of SOMEONE.”Source: §7.3.6 · p. 17

• DreamFace cannot currently generate full facial components such as eyes and mouth interiors, and diffusion-model inversion has not been fully explored.The authors also identify prompt-based control of lively expressions and nuanced facial motion as an unresolved challenge.Source: §7.4 · p. 18
• Using pretrained CLIP and Stable Diffusion makes DreamFace subject to biases in their training data, and its outputs may be misused for fake videos or impersonation.The authors note that these biases are not unique to DreamFace and are inherent in methods relying on these pretrained models.Source: §7.4 · p. 18

• DreamFace progressively generates personalized 3D faces from text prompts as assets compatible with CG engines.The conclusion describes novice customization of shapes and textures, effects such as aging and virtual makeup, and animation using in-the-wild video footage.Source: §8 · p. 18
• The system combines coarse-to-fine geometry, generic and texture latent diffusion models, and two-stage latent- and image-space optimization for physically based textures.The conclusion says the neutral assets support blendshape animation and the accompanying neural scheme provides personalized fine-grained animation from video input.Source: §8 · p. 18