Source-linked AI summary
Multi-view Consistent 3D Gaussian Head Avatars 'without' Multi-view Generation
Aviral Chharia, Fernando De la Torre
TL;DR
Existing 3D Gaussian head-avatar methods often require multi-view data, 3D capture, or intermediate view synthesis, limiting minimal-resource generation. MVCHead directly regresses multi-view-consistent 3D Gaussians from 2D images alone and achieves state-of-the-art perceptual quality with stronger texture and geometric consistency.
Problem
3D head-avatar generation needs multi-view consistency but existing methods depend on multi-view datasets, 3D captures, or intermediate 2D view synthesis.
Method
MVCHead is a single-shot state space model using hierarchical coarse-to-fine Gaussian refinement, HiBiSS scanning, and an SE(3) Multi-view Critic to enforce consistency directly in 3D.
Results
MVCHead achieves state-of-the-art perceptual quality, surpasses prior methods in texture and geometric consistency, and maintains comparable shape consistency.
Takeaways & Limitations
FaceGS-10K provides a large-scale dataset of ready-to-use 3D Gaussian heads for training, benchmarking, and evaluation of 3D-aware head models.
Takeaways & Limitations
The method is supervised only by randomly sampled 2D images, which provide appearance distributions but no ground-truth cross-view correspondences.
Abstract
from arXiv · showhide
High-fidelity 3D Gaussian head avatar generation is critical for applications such as AR/VR, telepresence, and digital humans. Existing methods depend on multi-view datasets, 3D captures, or intermediate 2D view synthesis. In contrast, we learn both conditional and unconditional 3D head models from randomly sampled 2D images alone, without using multi-view data, 3D supervision, or intermediate view generation. We introduce MVCHead, a single-shot state space model that enforces multi-view consistency (MVC) directly in the 3D representation while regressing 3D Gaussians under these constraints. At its core, we propose a Hierarchical State Space (HiSS) block that progressively refines Gaussians from coarse to fine, while capturing long-range dependencies. Within each HiSS block, we modify Mamba's standard unidirectional scan with the proposed Hierarchical Bi-directional State Scan (HiBiSS) that aligns recurrence with the axes along which multi-view inconsistencies are strongest. Finally, we design an SE(3) Multi-view Critic that judges whether a set of self-renders arises from a single underlying 3D configuration, rewarding cross-view pixel alignment without observing real multi-view pairs. MVCHead achieves state-of-the-art perceptual quality, surpasses prior methods in both texture and geometric consistency, and maintains comparable shape consistency. To demonstrate scalability, we release FaceGS-10K, the first large-scale dataset of ready-to-use 3D Gaussian head assets for training and evaluation of 3D head models. Project Page and code: https://humansensinglab.github.io/MVCHead/
1. Introduction
MVCHead targets scalable 3D Gaussian head avatar generation from 2D images alone, avoiding costly multi-view capture, intermediate view synthesis, and explicit 3D data. It directly enforces multi-view consistency through a state-space architecture and learned critic, achieving superior perceptual, texture, and geometric consistency while releasing FaceGS-10K.
- Motivation: High-fidelity 3D Gaussian head avatars support AR/VR, telepresence, digital characters, and large-scale film and game content creation.The paper emphasizes non-identifiable avatars for privacy and rapid content creation.
- Existing paradigms: Existing approaches rely on expensive studio captures, intermediate view synthesis, or feed-forward generation without explicit multi-view supervision.These paradigms differ in supervision, data requirements, scalability, and how they address multi-view consistency.
- MVCHead: MVCHead directly generates 3D Gaussian heads from latent codes in one forward pass using hierarchical state-space blocks and an SE(3) Multi-view Critic.HiSS progressively refines Gaussians, HiBiSS aligns recurrence with view-drift axes, and the critic rewards cross-view pixel alignment.
- Results: MVCHead surpasses prior methods in perceptual quality, texture consistency, and geometric consistency while maintaining comparable shape consistency.The paper also reports state-of-the-art performance across the evaluated multi-view consistency axes.
- Resources: FaceGS-10K provides a large-scale collection of ready-to-use 3D Gaussian head assets for training, benchmarking, and evaluation.The dataset is presented as a scalability resource for 3D-aware head models.
2. Related Works
Prior work ranges from high-quality but costly multi-view optimization to feed-forward generators that avoid per-subject optimization. MVCHead addresses the remaining challenge of enforcing multi-view consistency without paired multi-view supervision, while using state-space modeling for 3D head generation.
- Multi-view optimization-based methods: Multi-view optimization methods reconstruct detailed heads from dense studio-captured sequences and largely guarantee multi-view consistency, but their capture requirements limit scalability.Datasets such as RenderMe-360 and NeRSemble provide dense multi-view coverage.
- Open challenge: Enforcing multi-view consistency without paired multi-view supervision remains a key bottleneck for scalable 3D Gaussian head generation.This limitation is especially relevant to methods operating without real multi-view pairs.
- State Space Models: State-space models capture long-range dependencies, and MVCHead uses their propagation along inconsistency axes for 3D head avatar generation.The paper positions MVCHead as the first state-space model for 3D head avatars.
3. MVCHead
MVCHead generates 3D Gaussian heads from 2D images alone and enforces multi-view consistency through hierarchical state-space refinement, axis-aligned bidirectional scans, and an SE(3)-aware critic. Its end-to-end pipeline combines geometric consistency, texture realism, and spatial regularization without explicit 3D supervision.
- MVCHead: MVCHead learns a latent-to-3D-head mapping from 2D images alone, without expensive studio captures or intermediate view synthesis.The output is a set of anisotropic 3D Gaussians rendered differentiably under camera poses.
- MVCHead: HiSS blocks progressively refine Gaussian representations from coarse to fine, using coarser Gaussians as anchors for finer-level regression.The final avatar aggregates the progressively refined Gaussian primitives in a single splatting pass.
- HiBiSS: HiBiSS applies four directional state-space scans across rows and columns to target horizontal yaw drift and vertical pitch drift.The scans propagate shared appearance and geometric cues before upsampling and attribute regression.
- SE(3) Multi-view Critic: The SE(3) Multi-view Critic scores rendered image sets with their camera poses, assigning higher scores when views from one avatar are mutually consistent.Positive sets use different poses of the same avatar, while negative sets combine rendered views from different avatars.
- Training objective: The training objective combines the SE(3) Multi-view Critic, camera-conditioned adversarial texture matching, and Gaussian spatial regularization.The joint objective trains the Gaussian decoder, critic, and texture discriminator using 2D-image supervision.
4. Experiments and Results
MVCHead is evaluated under matched FFHQ and FFHQ-C protocols for perceptual realism, multi-view consistency, component effectiveness, and dataset scalability. It achieves strong visual quality and consistency results, while FaceGS-10K extends the work with ready-to-use Gaussian head assets.
- Experimental Setup: MVCHead is trained independently on FFHQ and FFHQ-C and benchmarked against state-of-the-art generative 3D head models under the established protocol.Training used Adam for 10M steps on 4 NVIDIA H100 GPUs over 3 days.
- Perceptual Realism: MVCHead achieves state-of-the-art perceptual realism across standard and wide-view evaluation, producing coherent renders across synthetic identities and viewpoints.FID evaluates 50K renders in Inception-V3 feature space, while FID3D samples camera poses across a wider viewpoint range.
- Multi-view Consistency: MVCHead improves global shape consistency while maintaining comparable local depth accuracy to the compared method.Chamfer Distance captures global shape discrepancies, whereas masked depth error measures local silhouette and fine-structure accuracy.
- Multi-view Consistency: MVCHead exhibits strong texture consistency across corresponding self-renders, preserving fine patterns such as eyebrows, lip color, blemishes, and hair edges.The evaluation uses cPSNR, cSSIM, and cLPIPS between renders from independently fitted 3DGS representations.
- Multi-view Consistency: MVCHead achieves lower MEt3R than the state of the art, indicating stronger geometric consistency under large pose changes.MEt3R is computed over many random camera pairs sampled uniformly along yaw and pitch around a canonical rig.
- Ablation Study: Ablations show that adversarial loss prevents training collapse, MVC loss improves FID and MEt3R, the state-space component improves MVC, and HiBiSS outperforms unidirectional scanning.These results support the roles of realism supervision, the SE(3) Multi-view Critic, state-space modeling, and axis-aligned bidirectional recurrence.
5. Conclusion and Future Work
MVCHead addresses multi-view consistency for 3D Gaussian head avatars in a minimal-resource setting, using a single forward pass without studio data or intermediate view synthesis. It achieves state-of-the-art performance while exposing limitations in viewing coverage and geometric supervision.
- MVCHead is the first state space model for 3D Gaussian heads designed for multi-view consistency in the minimal-resource setting.
- The model generates high-fidelity, multi-view consistent 3D head avatars in a single forward pass and achieves SOTA performance on five of six metrics.
- HiSS and HiBiSS align state-space recurrence with principal axes of drift, while the SE(3) Multi-view Critic enhances consistency without studio data or intermediate view synthesis.
- Limitations: MVCHead is trained only on front and side views, so it cannot generate full 360° avatars.
- Limitations: Its geometric priors are learned entirely from 2D supervision, motivating explicit structural constraints and harder critic negatives as future directions.