Generalizable and Animatable Gaussian Head Avatar

1 Introduction

• Some NeRF-based methods require identity-specific training and optimization, while some methods cannot render in real time during inference; some 3D Gaussian methods render in real time but still require identity-specific training and fail to generalize to unseen identities.Earlier 2D-based methods struggle to maintain multi-view consistency of expressions and identities when head poses change significantly [Yin et al., 2022, Ren et al., 2021].Source: §1 · pp. 1–2
• Dual-lifting predicts forward and backward lifting distances from one image, forming an almost closed Gaussian point distribution for 3D reconstruction.3DMM priors constrain lifting distances to support correct 3D lifting and capture source-image details.Source: §1 · p. 2
• GAGAvatar is presented as the first generalizable 3D Gaussian head-avatar framework with single-forward reconstruction and real-time reenactment.The framework reconstructs an animatable avatar from a single image.Source: §1 · p. 2
• The expression-control method combines 3DMM priors and 3D Gaussians to transfer expression information while avoiding redundant computations.It uses global image features, learnable 3DMM vertex features, and vertex positions to construct expression Gaussians.Source: §1 · p. 2

2.1 2D-based Talking Head Synthesis

• 2D methods inject expression and pose features or deform source images with warp fields, but large pose and expression variations can cause distortions and identity changes.Some 2D methods also lack viewpoint control and free-viewpoint rendering; audio-driven approaches may not explicitly control facial expressions and poses [Guo et al., 2021, Tang et al., 2022, Zhang et al., 2023b].Source: §2.1 · pp. 2–3

2.2 3D-based Head Avatar Reconstruction

• Many 3D avatar approaches require identity-specific videos or test-time optimization, while inversion-based methods can fail to preserve source identity.The need for thousands of personal-image frames in some methods also raises privacy concerns.Source: §2.2 · p. 3
• NeRF methods often lack real-time speed, and existing 3D Gaussian methods require identity-specific video training rather than generalizing to unseen identities.GAGAvatar is positioned as a one-shot Gaussian method that generalizes to unseen identities and renders in real time.Source: §2.2 · p. 3

3 Method

• The pipeline combines reconstruction and expression branches, splats their Gaussians for a coarse image, then uses a neural renderer to produce a fine result.The source and driving images provide features for reconstruction and expression control, respectively.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Dual-lifting and Reconstruction Branch

• Dual-lifting predicts pixel offsets from a feature plane to form separate forward and backward Gaussian sets, reducing lifting ambiguity and supporting large viewpoint changes.The predicted sets form an almost closed point distribution that can be rendered from any viewpoint.Source: §3.1 · pp. 4–5

3.2 Expression Branch

• The expression branch predicts Gaussian parameters from source-image global features and learnable 3DMM-vertex features, using the vertices as positions for expression control.During reenactment, camera pose and expression-Gaussian positions are modified without redundant calculations.Source: §3.2 · p. 5

3.3 Neural Renderer

• The method predicts 32-dimensional Gaussian features and uses a neural renderer to decode the coarse splat into RGB, producing fine images and resolving conflicts between Gaussian sets.The dual-lifting method generates 175,232 points; the renderer is trained from scratch without pretrained initialization.Source: §3.3 · p. 5

3.4 Training Strategy and Loss Functions

• Training combines coarse- and fine-image reconstruction and perceptual losses with nearest-3DMM-vertex lifting constraints; experiments report improved 3D structure and large-viewpoint performance.Constraining only a subset of lifting points leaves areas such as hair and accessories learnable.Source: §3.4 · pp. 5–6

4.1 Experiment Setting

• Training uses 586,382 VFHQ frames; evaluation uses 5,000 VFHQ frames from 100 videos and 19 HDTF clips, against ten listed baselines.Each VFHQ video's first frame is the source, with remaining frames used for driving and targets.Source: §4.1 · pp. 6–7, §4.2 · p. 7

4.2 Main Results

• On VFHQ and HDTF, GAGAvatar outperforms prior methods in reconstruction and synthesis quality and expression-control accuracy, while cross-reenactment identity consistency is slightly worse than some methods.The comparison is reported in Tables 1 and 2.Source: Table 1 · p. 7, Table 2 · p. 7, §4.2 · p. 8
• Qualitative comparisons show detailed avatars, subtle eye and mouth movements, and maintained identity consistency and image quality under large head rotations.The paper also reports faster high-quality reconstruction and rendering than the baseline methods.Source: §4.2 · pp. 7–8
• 67 FPS on an A100 makes GAGAvatar the paper's first real-time method for animatable one-shot head-avatar reconstruction.The reenactment timing excludes driving-parameter calculation and averages 100 frames.Source: §4.2 · p. 8, Table 3 · p. 8

4.3 Ablation Studies

• Dual-lifting significantly improves reconstruction quality over single-plane lifting, which can create ambiguities by lifting points both forward and backward.Lifting occurs once per identity, so subsequent expression driving does not require recalculation or affect reenactment performance.Source: §4.3 · p. 8, Table 4 · p. 9
• Removing lifting distance loss degrades reconstruction, especially at facial edges; dual-lifting also produces more reasonable, less-flat structure with points away from the input view.The lifting-point visualization shows the 3D structure from front and top views.Source: §4.3 · p. 9, Figure 5 · p. 9
• Removing global identity features lowers identity similarity and reenactment quality, while neural rendering improves the detail and expression of otherwise reasonable sparse-Gaussian results.The neural renderer refines coarse images and features without increasing output resolution.Source: §4.3 · p. 9
• Extreme expressions and occlusions remain tractable, though blur and difficult lighting reduce detail; neural rendering also resolves RGB conflicts between static and dynamic Gaussian sets.The authors attribute conflict resolution to 32-D Gaussian features that carry more information than RGB values.Source: §4.3 · p. 9, Figure 6 · p. 10, Figure 7 · p. 10

• The authors report that GAGAvatar outperforms state-of-the-art baselines in head-avatar reconstruction quality and reenactment accuracy, with significant improvements in rendering speed.Source: §5 · p. 10
• The authors warn that the method could be misused to generate fake videos and spread false information, and they oppose such misuse.Source: §5 · p. 10
• The method may generate less detail for unseen areas, while its 3DMM-based expression branch cannot control areas outside the model, such as hair and tongue.Source: §5 · p. 10

• DINOv2 Base processes 3 × 518 × 518 images and produces 296×296 local feature maps plus 768-dimensional global features.Source: §A.1 · p. 16
• Four non-downsampling ResBlock groups predict 41 Gaussian-parameter dimensions per pixel, covering color, opacity, scale, rotation, and lifting distance.Source: §A.1 · p. 16
• An MLP combines 5023 FLAME point features of 256 dimensions each with global features to predict 40-dimensional expression Gaussian parameters; StyleUNet maps 32 × 512 × 512 images to 3 × 512 × 512.Source: §A.1 · p. 16

• The VFHQ split uses 15,204 training clips and 100 test videos, sampling 50 frames per test clip for 5,000 test frames; HDTF uses 1,900 test frames from 19 videos.Source: §A.2 · p. 16
• VFHQ training clips are sampled by duration at 25, 50, or 75 frames for clips under 2 seconds, 2–3 seconds, or over 3 seconds, respectively.Source: §A.2 · p. 16
• Frames are background-removed and resized to 512 × 512, with 3DMM parameters automatically extracted and refined; the authors note the labels are noisy but support building a large dataset.Source: §A.2 · p. 16

A.3 More Evaluation Details

• For fair comparison, outputs from the evaluated methods are landmark-cropped to a common head region and resized to 512×512 after using each method’s official preprocessing and implementation.Source: §A.3 · p. 16
• Portrait4D and Portrait4D-v2 are described as orthogonal to this work because their data-generation and learning contributions could also benefit the method.Source: §A.3 · p. 16

B Preliminaries of 3DMM

• FLAME serves as an expression signal and geometry prior, representing head shape through identity shape variation, jaw and neck pose deformation, and expression blend-shapes.Source: B Preliminaries of 3DMM · pp. 16–17
• The displayed FLAME equation combines a template mesh with shape, pose, and expression blend-shape components.Source: B Preliminaries of 3DMM · pp. 16–17
• Figure 8 presents in-the-wild examples in the order of input image, driving image, and driving plus novel-view results.Source: Figure 8 · p. 17

C Per-part Rendering and 3D Lifting of Our Method

• The Gaussians contain 32-D features, and the figure displays only their first three dimensions as RGB without the neural rendering module.The caption cautions that RGB values alone should not be used to judge each branch’s importance.Source: Figure 9 · p. 18
• The dual-lifting point cloud captures in-the-wild details including glasses and hair.Source: Figure 10 · p. 18, C Per-part Rendering and 3D Lifting of… · p. 18
• Dual-lifting Gaussians reconstruct the head’s base structure and facial details, while the expression Gaussians are rendered separately.The per-part rendering is consistent with the intended roles of the two dual-lifting components.Source: C Per-part Rendering and 3D Lifting of… · pp. 17–18

D More Qualitative Results

• Qualitative comparisons show self-identity and cross-identity reenactment results on VFHQ and HDTF.Figure 11 presents self-identity results and Figure 13 presents cross-identity results; their captions identify the datasets and row groupings.Source: D More Qualitative Results · p. 18, Figure 11 · p. 19, Figure 13 · p. 20
• The method maintains identity and 3D consistency as the viewing angle changes, including in multi-view results.The figures include reenactment comparisons and multi-view results on VFHQ, as well as in-the-wild examples.Source: D More Qualitative Results · p. 18, Figure 12 · p. 19, Figure 14 · p. 21
• Supplementary video results are described as timing-stable despite no special processing.Source: D More Qualitative Results · p. 18

E More In-Depth Ethical Discussion

• The authors identify deepfakes, privacy violations, and false information as ethical risks of the framework.Source: E More In-Depth Ethical Discussion · p. 20
• Planned safeguards include visible and invisible watermarking and a strict license prohibiting commercial synthesis of real individuals without explicit consent.The authors say invisible marks are intended to help track video sources, including after re-editing.Source: E More In-Depth Ethical Discussion · pp. 20, 23

F Limitations and Future Work

• Unseen regions in the source image may lack detail in novel views, producing statistically average expectations for features absent from the input.Examples include completing the other half of a face from a side view or generating an open mouth from a closed-mouth input.Source: F Limitations and Future Work · p. 23
• The expression branch, based on 3DMM and learned from VFHQ video data, may miss extreme facial movements and parts not modeled by 3DMM.Examples include asymmetric eye states, the tongue, and hair.Source: F Limitations and Future Work · p. 23
• Future directions include learning expression embeddings from images, generating data to reduce data and tracking-accuracy requirements, gathering more expressive data, and extending to full-body avatars.Source: F Limitations and Future Work · p. 23