PaperlayerView exact revision on arXiv

Source-linked AI summary

MonoGaussianAvatar: Monocular Gaussian Point-based Head Avatar

Yufan Chen, Lizhen Wang, Qijing Li, Hongjiang Xiao, Shengping Zhang, Hongxun Yao, Yebin Liu

arXiv:2312.04558v1cs.CV

TL;DR

Head avatars reconstructed from monocular video need to represent detailed geometry and appearance while supporting expression and pose changes, but existing Gaussian-splatting approaches face head-avatar-specific challenges. MonoGaussianAvatar combines explicit 3D Gaussian points with a deformation field, and the authors report that it outperforms other state-of-the-art 3D head-avatar approaches. Its current limitations include eyeglass-lens reflections and reliance on 3DMM priors for expressions.

  • Problem

    Existing Gaussian-splatting methods face head-avatar challenges in parameter initialization, point insertion and deletion, and adapting Gaussian parameters to deformed expressions.

    Source: §1 · p. 2

  • Method

    MonoGaussianAvatar represents head geometry and appearance with 3D Gaussian points and uses a Gaussian deformation field to adjust their parameters for animation.

    Source: §1 · p. 2, §3.2 · p. 5

  • Results

    The authors report that MonoGaussianAvatar outperforms other state-of-the-art 3D head-avatar approaches.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The method maintains rigid structures such as teeth without holes even under substantial movement.

    Source: §5 · p. 8

    The method cannot model eyeglass-lens reflections and relies on 3DMM blendshapes and skinning weights, limiting its handling of exceptionally challenging expressions beyond predefined priors.

    Source: §5 · p. 8

Abstract

from arXiv · show

The ability to animate photo-realistic head avatars reconstructed from monocular portrait video sequences represents a crucial step in bridging the gap between the virtual and real worlds. Recent advancements in head avatar techniques, including explicit 3D morphable meshes (3DMM), point clouds, and neural implicit representation have been exploited for this ongoing research. However, 3DMM-based methods are constrained by their fixed topologies, point-based approaches suffer from a heavy training burden due to the extensive quantity of points involved, and the last ones suffer from limitations in deformation flexibility and rendering efficiency. In response to these challenges, we propose MonoGaussianAvatar (Monocular Gaussian Point-based Head Avatar), a novel approach that harnesses 3D Gaussian point representation coupled with a Gaussian deformation field to learn explicit head avatars from monocular portrait videos. We define our head avatars with Gaussian points characterized by adaptable shapes, enabling flexible topology. These points exhibit movement with a Gaussian deformation field in alignment with the target pose and expression of a person, facilitating efficient deformation. Additionally, the Gaussian points have controllable shape, size, color, and opacity combined with Gaussian splatting, allowing for efficient training and rendering. Experiments demonstrate the superior performance of our method, which achieves state-of-the-art results among previous methods.

1. Introduction

Monocular head avatars must capture detailed facial geometry and appearance while supporting dynamic expressions, but existing representations face constraints in topology, training burden, or deformation. MonoGaussianAvatar uses flexible 3D Gaussian points and a Gaussian deformation field, with tailored initialization and point management.

1. Introduction

  • 3DMMs have fixed topologies, neural implicit methods can be inefficient to train and render, and PointAvatar's fixed rendering radius increases point requirements and can leave holes during substantial movement.The introduction also notes that rigid parts such as teeth may deform like the lips in PointAvatar.Source: §1 · p. 2
  • MonoGaussianAvatar represents head geometry and appearance with 3D Gaussian points and a continuous deformation field for animation.Their scale and rotation flexibility is presented as addressing holes and inaccuracies in rigid regions, while matching PointAvatar rendering quality with an equivalent number of points.Source: §1 · p. 2
  • The method adapts Gaussian initialization and point insertion and deletion for head avatars, then deforms Gaussian parameters to fit changing expressions.The deformation field is described as preventing information from multiple frames from being smoothed into a single frame during point-image alignment.Source: §1 · p. 2
  • The authors report superior results to state-of-the-art methods on structure similarity, image similarity, and Peak Signal-to-Noise Ratio.The stated contributions also include reconstructing detailed geometry and appearance from monocular videos and preserving accessories such as glasses in novel poses.Source: §1 · p. 2

2. Related Work

Prior head-avatar work spans mesh, implicit, point-based, and Gaussian representations, each with different strengths and constraints. The related work describes Gaussian points as combining explicit representation with adjustable scale and rotation, and Figure 2 outlines the proposed method's processing pipeline.

2. Related Work

  • 3DMM-based avatar methods represent basic facial geometry and expressions but remain limited by fixed topologies when modeling components such as hair.Neural Head Avatar predicts vertex offsets and textures, while ROME uses a neural texture map.Source: §2 · p. 2
  • Neural implicit methods include SDF- and NeRF-based approaches, but training and rendering efficiency remain challenges.The related work describes SDFs as supporting accurate geometry and several NeRF-based variants as targeting faster training or improved rendered-image quality.Source: §2 · pp. 2–3
  • Mesh methods support efficient rasterization but can miss fine structures, while PointAvatar offers flexible topology and controllable animation but can produce holes during substantial movement.PointAvatar's fixed rendering radius requires a balance between rendering quality and point count.Source: §2 · p. 3
  • Traditional full-head avatar methods use dense multiview videos to reconstruct geometry and high-fidelity animation, but often require heavy computation.Source: §2 · p. 3
  • 3D Gaussian points retain explicit representation while adding scale and rotation flexibility that can help model rigid facial regions such as teeth.The method pipeline initializes Gaussian parameters, transforms mean positions from canonical to deformed space, renders the deformed points, and manages point insertion and deletion.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

MonoGaussianAvatar reconstructs a head avatar from monocular portrait video using Gaussian points whose parameters and positions are adapted for the subject’s expressions and poses. Its pipeline combines point updates, deformation, Gaussian splatting, and image-based training losses.

3. Method

  • The method represents each subject with Gaussian points and uses a Gaussian deformation field to animate their parameters and positions with target expression and pose.The Gaussian parameter prediction network supplies opacity, rotation, scale, and color, while FLAME expression and pose parameters determine deformed positions.Source: §3 · pp. 3–4

3.1. Gaussian Setting

  • The Gaussian setting uses a two-stage initialization strategy and a point insertion and deletion approach tailored to avatar reconstruction.Source: §3.1 · p. 4

3.1.1 Gaussian Initialization

  • Gaussian points in initialized space are parameterized by mean position, rotation, scale, opacity, and color, then mapped to canonical space with a learnable offset.The method begins with 400 randomly sampled points on a sphere and uses an MLP to predict the parameters from each mean position.Source: §3.1.1 · p. 4

3.1.2 Point Insertion and Deletion

  • A coarse-to-fine point insertion and deletion strategy updates point density and sampling radius during training, pruning deformed-space points with opacity below 0.1.The process starts with a sparse cloud for coarse head shape and adds dense points later for fine detail; Figure 3 describes rapid convergence and refinement at the maximum point setting.Source: §3.1.2 · pp. 4–5, Figure 3 · p. 4

3.2. Gaussian Deformation

  • A person-specific topology-consistent template uses deformation blendshapes and skinning weights to transform canonical-space points according to target FLAME expression and pose.The deformation field also predicts offsets for Gaussian parameters because leaving those parameters unchanged during deformation produces blur and inconsistent geometry in novel poses.Source: §3.2 · p. 5

3.3. Gaussian Splatting Rendering

  • Gaussian splatting renders the deformed points using their positions, rotations, scales, opacities, and colors, blending depth-ordered points to form each pixel.For novel views, the points are projected from 3D to 2D using a camera-plane transformation.Source: §3.3 · p. 5

3.4. Training Objectives

  • Training uses RGB supervision and VGG feature loss, alongside FLAME-related losses and a D-SSIM term in the total objective.RGB loss aligns rendered and ground-truth per-pixel colors, while Gaussian splatting permits whole-image perceptual losses during training.Source: §3.4 · pp. 5–6

4. Experiments

Experiments compare MonoGaussianAvatar with three state-of-the-art facial reenactment methods on DSLR and smartphone videos and ablate three design choices.

4.1. Comparison with SOTA Methods

  • MonoGaussianAvatar achieves the best L1, LPIPS, SSIM, and PSNR metrics on both DSLR and smartphone videos, with smartphone LPIPS similar to PointAvatar.The other smartphone metrics are significantly higher than PointAvatar’s, possibly because movement in that test sequence is limited.Source: Table 1 · p. 6, §4.1 · p. 7
  • Under substantial movement, the method maintains rigid teeth and avoids holes, while point allocation recovers hair geometry and detailed skin.The teeth comparison includes PointAvatar; hair allocation is contrasted with IMavatar, and detailed skin with NerFace.Source: §4.1 · p. 7, Figure 5 · p. 7

4.2. Ablation

  • Gaussian parameter deformation stabilizes accessories in novel poses and improves appearance; without it, structures vary and become blurred.The ablation describes stable convergence with the field incorporated during training.Source: §4.2 · p. 7, Figure 6 · p. 7
  • Opacity-based pruning (<0.1) and scheduled upsampling improve geometry in canonical and deformed spaces over PointAvatar’s strategy, enabling finer facial detail.The method prunes low-opacity points each epoch and upsamples toward a predetermined count at specific epochs.Source: §4.2 · p. 8, Figure 7 · p. 8
  • Randomly initializing 400 Gaussian points on a sphere performs better and converges more consistently than initializing 100000 points on the FLAME template.The authors suggest that the many well-initialized points may be identified as visual points, making point deletion ineffective.Source: §4.2 · p. 8, Figure 8 · p. 8

5. Discussion and Conclusion

MonoGaussianAvatar is reported to outperform other state-of-the-art 3D head-avatar approaches, while retaining rigid structures such as teeth without holes under substantial movement. The authors also identify limitations in eyeglass-lens reflections and handling expressions beyond predefined 3DMM priors, and caution about potential deepfake misuse.

5. Discussion and Conclusion

  • The method cannot model eyeglass-lens reflections and is constrained on exceptionally challenging expressions outside predefined 3DMM blendshape and skinning-weight priors.The authors suggest physical refraction modeling and color disentangling as possible directions for the lens-reflection limitation.Source: §5 · p. 8
  • The authors caution that lifelike personalized head characters reconstructed from monocular video could be misused to create deepfakes.Source: §5 · p. 8
  • The method outperforms other state-of-the-art 3D head-avatar approaches and maintains rigid structures such as teeth without holes under substantial movement.The authors describe this as an overall experimental finding.Source: §5 · p. 8

6. Additional Results

The additional results report efficient training and rendering relative to state-of-the-art methods, as well as improved appearance detail at higher rendering resolution. They also show faster training convergence than an implicit-based method.

6.1. Different rendering resolution

  • 1024 × 1024 rendering exhibits better appearance detail than 512 × 512 rendering in the same case.The main-paper results are rendered at 512 × 512.Source: §6.1 · p. 11, Figure 9 · p. 11

6.2. Training and Rendering Efficiency

  • The method attains superior training and per-image rendering efficiency compared with existing state-of-the-art methods.Table 2 compares training and rendering time per image for the same case.Source: Table 2 · p. 11, §6.2 · p. 11
  • Training convergence is significantly faster than for the implicit-based method, with convergence shown for two cases.Source: Figure 10 · p. 11, §6.2 · p. 11

7. Implementation Details

The implementation describes point management, network architecture, optimization, evaluation, preprocessing, and data ethics. It specifies the point-growth schedule, network roles, training settings, shared comparison preprocessing, and consent conditions for collected data.

7. Implementation Details

  • The approach was implemented in PyTorch using an NVIDIA GTX 3090.Source: §7 · p. 11

7.1. Point Insertion and Deletion

  • Point management starts with 400 sphere points, prunes opacity below 0.1, and progressively upsamples to 100,000 while shrinking radii before fixing the sampling radius at 0.004 after epoch 100.The designated quantity and radius schedules are updated at staged intervals; rendering radius integrates Gaussian-point scales.Source: §7.1 · pp. 11–12

7.2. Network Architecture

  • The initialized MLP predicts Gaussian parameters from mean position, while deformation MLPs map mean position and remaining parameters toward deformed space.The MLP layers use weight normalization; the initialized network uses Softplus, and the deformation network uses Softplus on the left and ReLU on the right.Source: §7.2 · p. 12, Figure 11 · p. 12, Figure 12 · p. 12

7.3. Training Details

  • Training uses Adam at a 1e−4 learning rate, with stated RGB, D-SSIM, FLAME, and VGG loss weights and staged learning-rate and FLAME-regularization decay.The learning-rate decay occurs at epochs 80 and 100; FLAME-regularization decay occurs at epochs 20, 30, 50, and 70.Source: §7.3 · p. 12

7.4. Evaluation Details

  • Pre-tracked FLAME expression and pose parameters are fine-tuned during training and evaluation, while evaluation uses only RGB loss.Source: §7.4 · p. 12

7.5. Data Preprocessing

  • All methods use the same preprocessing pipeline, camera, and FLAME parameters to reduce differences from face-tracking schemes; captured subjects' images are cropped and resized to 512 × 512 and 1024 × 1024.Source: §7.5 · p. 12

7.6. Ethics

  • Three subjects were captured by the authors, who obtained written consent; their data will be made public only if the subjects grant publication permission.The experiments also use data from three subjects obtained from other datasets.Source: §7.6 · p. 12

Related papers

Loading 2312.04558v1…