PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashAvatar: High-fidelity Head Avatar with Efficient Gaussian Embedding

Jun Xiang, Xuan Gao, Yudong Guo, Juyong Zhang

arXiv:2312.02214v2cs.CVcs.GR

TL;DR

High-fidelity head avatars that reconstruct efficiently and render in real time are a target for interactive multimodal digital humans. FlashAvatar combines mesh-attached Gaussians, learned spatial offsets, and UV sampling. It reports photo-realistic rendering with personalized details at 300FPS, while noting reliance on accurate surface-embedded initialization and tracked expression codes.

  • Problem

    Prior dynamic Gaussian canonical-plus-deformation methods do not robustly model head avatars with complex expressions, while high-fidelity interactive digital humans also require efficient reconstruction and rendering.

    Source: §1 · pp. 1–2

  • Method

    FlashAvatar embeds a Gaussian field in a parametric face mesh, uses uniform UV sampling, and learns spatial offsets for non-surface regions and subtle facial details.

    Source: §1 · p. 2

  • Results

    FlashAvatar reconstructs an avatar from monocular video in minutes and animates it at 300FPS with photo-realistic rendering and full personalized details.

    Source: §6 · p. 8

  • Takeaways & Limitations

    At 300FPS on a consumer-grade GPU, FlashAvatar leaves more room for other real-time multimodal tasks such as speech processing, text understanding, and crossmodal translation.

    Source: §6 · p. 8

    Large tracking errors, especially global pose errors, may cause detail loss or image misalignment, and tracked expression codes cannot model dynamically changing hair with heavy non-rigid deformation.

    Source: §6 · p. 8

Abstract

from arXiv · show

We propose FlashAvatar, a novel and lightweight 3D animatable avatar representation that could reconstruct a digital avatar from a short monocular video sequence in minutes and render high-fidelity photo-realistic images at 300FPS on a consumer-grade GPU. To achieve this, we maintain a uniform 3D Gaussian field embedded in the surface of a parametric face model and learn extra spatial offset to model non-surface regions and subtle facial details. While full use of geometric priors can capture high-frequency facial details and preserve exaggerated expressions, proper initialization can help reduce the number of Gaussians, thus enabling super-fast rendering speed. Extensive experimental results demonstrate that FlashAvatar outperforms existing works regarding visual quality and personalized details and is almost an order of magnitude faster in rendering speed. Project page: https://ustc3dv.github.io/FlashAvatar/

1. Introduction

FlashAvatar targets efficient reconstruction and real-time rendering of high-fidelity head avatars, addressing trade-offs in prior mesh, implicit, and voxel-based approaches. It combines mesh-attached Gaussians with learned offsets and uniform UV sampling to model facial details while reducing Gaussian count and supporting fast rendering.

Motivation

  • Prior 3DMM-based methods face limits from coarse geometry and fixed topology, while neural implicit and voxel approaches retain computation or volume-rendering speed constraints.The introduction notes that volume rendering’s excessive sampling and alpha composition still limits inference speed.Source: §1 · p. 1
  • The canonical-plus-deformation strategy used by prior dynamic Gaussian methods does not robustly model complex head expressions, even when conditioned on expression code.The authors report this finding from their experiments.Source: §1 · p. 2

Method

  • FlashAvatar attaches Gaussians to a parametric face mesh for main appearance and expression, then predicts spatial offsets for non-surface features and subtle facial dynamics.The mesh moves the attached Gaussians with expression changes, reducing the need to learn large deformations.Source: §1 · p. 2

Initialization

  • Flexible UV sampling creates a more uniform Gaussian position distribution than uneven mesh-vertex or face sampling, improving reconstruction and allowing density control.Figure 2 samples only the head region, including the neck, using fewer vertices than FLAME’s 5023.Source: §1 · p. 2, Figure 2 · p. 2

Contributions

  • 10K-level Gaussians and 300FPS at 512 × 512 accompany the approach, which recovers fine facial details, thin structures, and subtle expressions.The reported speed is enabled by uniform, flexible UV sampling and mesh-based initialization.Source: §1 · p. 2

2. Related Work

Related work spans mesh-based and implicit head models, dynamic Gaussian scene representations, and techniques for accelerating radiance fields. FlashAvatar uses mesh-guided Gaussian initialization and density control to represent dynamic head avatars efficiently.

2.1. Digital Head Model

  • Mesh-based head models offer explicit geometry, while implicit models use neural functions; prior approaches also face limits in detail, temporal consistency, point count, or efficiency.The paper positions FlashAvatar as the first mesh-guided Gaussian field for head-avatar modeling.Source: §2.1 · p. 2

2.2. Scene representations with 3D-GS

  • Dynamic3DGS reconstructs frames individually, whereas Deformable3DGS and 4D-GS use canonical Gaussians with timestamp-conditioned deformation; FlashAvatar instead attaches Gaussians to the head mesh and learns dynamic offsets.The cited dynamic Gaussian methods are [29], [51], and [48], respectively.Source: §2.2 · p. 3

2.3. Radiance field acceleration

  • FlashAvatar applies 3D-GS’s tile-based rasterizer and uses rational Gaussian initialization and density control to compress the representation and reach 300FPS.The passage contrasts voxel acceleration methods’ cache requirements and INGP’s visual-quality gap with state-of-the-art NeRF methods [2].Source: §2.3 · p. 3

3. Background

The background introduces 3D Gaussians as scene primitives and describes their projection and color compositing for rendering. It motivates surface-concentrated initialization and a fixed Gaussian count for head avatars.

3D Gaussian Splatting

  • 3D-GS uses 3D Gaussians as geometric primitives, with covariance Σ centered at point μ; anisotropic splatting and fast back-propagation make them expressive and easy to optimize.Each Gaussian’s projected density uses a 2D Gaussian multiplied by opacity.Source: §3 · p. 3

Rendering

  • For rendering, Gaussians are projected into 2D using the viewing transformation and the Jacobian of an affine approximation, then sorted and blended to calculate pixel color.The color blend accumulates each Gaussian’s color weighted by opacity and the transmittance of preceding Gaussians.Source: §3 · p. 3

Initialization and density

  • The method assumes ideal avatar Gaussians concentrate on the head surface, motivating initial attachment to the FLAME mesh.A fixed Gaussian count is favored over 3D-GS’s splitting scheme because head-avatar complexity lies within a specific range; uncontrolled densification increases memory use and slows rendering.Source: §3 · p. 3

4. Methods

FlashAvatar embeds a uniform Gaussian field in a tracked FLAME mesh and learns expression-conditioned spatial offsets for non-surface regions and subtle facial details. Its pipeline combines this representation with a 300 FPS rendering target.

4. Methods

  • The pipeline uses tracked FLAME meshes and expression codes to drive a Gaussian representation that renders at 300 FPS.Gaussians are embedded in the dynamic mesh and splatted to render the image at a given pose.Source: §4 · p. 3, Figure 3 · p. 4

4.1. Surface-embedded Gaussian Initialization

  • UV sampling creates a uniform canonical Gaussian field, enables density control, and broadens and closes the mouth region for interior-mouth modeling.Rasterization maps UV pixels to mesh positions; fixed barycentric coordinates recover Gaussian positions as expressions change.Source: §4.1 · p. 4

4.2. Gaussian Offset

  • An MLP takes canonical-mesh positions and expression codes to predict spatial residuals that adjust Gaussian position, rotation, and scale.Dynamic scaling alongside position and rotation helps model fixed-size parts such as teeth.Source: §4.2 · p. 4

4.3. Training Scheme

  • Training combines Huber photometric loss with extra mouth-region weighting and LPIPS perceptual loss, whose structural regularization helps stabilize training.The mouth-weighted term uses a mask; LPIPS uses VGG and is weighted 0 for the first 15,000 steps, then 0.05.Source: §4.3 · p. 5

4.4. Implementation Details

  • The implementation samples 13,453 Gaussians at UV resolution 128 and trains Gaussian attributes and the offset network with Adam.Each epoch randomly samples 2,000 training frames; the offset-network learning rate is 1e-4.Source: §4.4 · pp. 5–6

5. Experiments

Across the reported comparisons, FlashAvatar outperforms other methods in pixel-wise error and perceptual quality, while producing fine details and subtle expressions. The experiments also examine its reconstruction speed and the effects of Gaussian density, initialization, pruning, and dynamic offsets.

5.2. Comparison with Representative Methods

  • FlashAvatar outperforms other methods in both pixel-wise error metrics and perceptual quality, and its qualitative results recover fine details, thin structures, and subtle expressions with about 10K Gaussians.The comparison uses average errors on tested videos and reports MSE, L1 distance, PSNR, SSIM, and LPIPS.Source: Table 1 · p. 6, §5.2 · pp. 6–7

5.1. Dataset

  • Processed videos are 1–3 minutes long, with the last 500 frames used for testing; videos are cropped, subsampled to 25 FPS, and resized to 5122 resolution.Foreground segmentation uses RVM, and an off-the-shelf face-parsing framework parses the mouth region.Source: §5.1 · p. 6

5.3. Comparison with C + D strategy

  • The canonical-plus-deformation strategy struggles with complex expressions and facial details at low Gaussian counts, while mesh guidance with extra offsets preserves exaggerated expressions and fine details.Random initialization produces artifacts, and mesh-surface initialization removes most but not all of them; subtle expressions remain less well captured than with FlashAvatar.Source: §5.3 · p. 7

5.4. Training Efficiency

  • FlashAvatar renders above 300 FPS and reconstructs a photo-realistic avatar with fine hair strands and textures within a couple of minutes on one RTX 3090.The coarse head appearance can be recovered in several seconds.Source: §5.4 · p. 7

5.5. Ablation Studies

  • Increasing Gaussian sampling density improves quality but slows rendering; the authors use UV resolution 128 and recommend choosing density for the application.Table 2 reports the influence of Gaussian density at UV resolution 128.Source: §5.5 · p. 7, Table 2 · p. 7
  • UV-space initialization distributes Gaussians more uniformly than vertex initialization, yielding more photo-realistic results with fewer Gaussians.Vertex-attached Gaussians do not converge to satisfactory results.Source: §5.5 · p. 7, Figure 10 · p. 8
  • Further local pruning or densification yields only slight gains in rendering quality and speed under the authors’ settings, although region-adaptive Gaussian placement could improve results.The authors suggest distributing Gaussians according to regional complexity and semantic correspondence.Source: §5.5 · p. 7
  • A static offset can reconstruct static regions such as hair but does not model facial changes well; expression-conditioned dynamic offsets produce more faithful visuals.The authors attribute the static-offset shortcoming to coarse FLAME geometry and facial-expression complexity.Source: §5.5 · p. 8, Figure 11 · p. 8

6. Conclusion and Discussion

FlashAvatar combines a Gaussian radiance field with a parametric face model to reconstruct and animate detailed avatars efficiently. The authors report fast, high-quality rendering, while noting limitations from tracking errors and dynamically changing hair.

6. Conclusion and Discussion

  • FlashAvatar combines a non-neural Gaussian radiance field with an explicit parametric face model to reconstruct an avatar from monocular video in minutes.The representation learns Gaussian offsets for non-surface features and small facial dynamics.Source: §6 · p. 8
  • The method still relies on good surface-embedded Gaussian initialization; large tracking errors, especially global pose errors, may cause lost details or image misalignment.Expression-code conditioning also prevents modeling dynamically changing hair with heavy non-rigid deformation.Source: §6 · p. 8
  • 300FPS on a consumer-grade GPU with SOTA rendering quality is reported for FlashAvatar, leaving more room for other real-time multimodal processes.The paper identifies speech processing, text understanding, and crossmodal translation as examples of such processes.Source: §6 · p. 8

A.1. Additional Ablations

The additional ablations show that closing the FLAME mouth cavity helps represent the interior mouth, while perceptual-loss supervision preserves personalized facial attributes and improves photo-realism.

A.1. Additional Ablations

  • Adding faces to close the FLAME mouth cavity helps model the interior mouth, including teeth.Figure 12 illustrates this design choice.Source: §A.1 · p. 12, Figure 12 · p. 12
  • Without cavity closure, nearby lip Gaussians can make the upper and lower teeth stick together, producing blurry results in challenging cases.Figure 12 depicts the mouth-cavity ablation.Source: §A.1 · p. 12, Figure 12 · p. 12
  • Perceptual-loss supervision helps maintain personalized facial attributes and boosts photo-realism.Figure 13 compares results with and without perceptual loss.Source: §A.1 · p. 12, Figure 13 · p. 12

A.2. Additional Results

The additional results identify reliance on good surface-embedded Gaussian initialization as a limitation: large tracking errors cannot be handled reliably.

A.2. Additional Results

  • FlashAvatar still relies on a good surface-embedded Gaussian initialization.Source: §A.2 · p. 12
  • The method cannot handle large tracking errors.Source: §A.2 · p. 12
  • The stated limitation concerns the method’s dependence on its initialization and its handling of tracking errors.Source: §A.2 · p. 12

B.1. Network Architecture

The appendix points to the offset-network architecture, describes Gaussian sampling within a FLAME mask, and discusses the work’s broader impact and misuse risks.

B.1. Network Architecture

B.2. FLAME Masks

  • For head-and-neck Gaussian sampling, a FLAME mask excludes the mesh boundary.Figure 16 identifies the excluded boundary region.Source: §B.2 · p. 12, Figure 16 · p. 12

C. Broader Impact

  • The work presents fast, detailed avatar reconstruction as a step toward practical multimodal digital-human applications, while warning of DeepFake misuse.Source: C. Broader Impact · p. 12
  • The authors discourage generating fake images or videos to spread false information or damage reputations, and acknowledge they may be unable to prevent nefarious use.Source: C. Broader Impact · p. 13
Loading 2312.02214v2…