Source-linked AI summary
Photorealistic Novel View Synthesis of Human Faces using Next-Scale Transformers
Federico Stella, Fei Jiang, Zhongshi Jiang, Zohar Barzelay, Emanuel Garbin, Amin Jourabloo, Liuhao Ge
TL;DR
Photorealistic human face view synthesis must preserve identity and geometric coherence at high resolution across multiple target cameras. The paper adapts next-scale autoregressive transformers for simultaneous multi-view generation and staged high-resolution training, reporting strong perceptual and cross-view results while identifying limitations in expression, accessory, and background generalization.
Problem
High-resolution human view synthesis across multiple cameras remains challenging because identity, fine appearance, and geometric coherence must be preserved.
Method
The method extends next-scale autoregressive transformers with multi-view conditioning and staged training, then couples generated views with transformer-based pixel-aligned 3D Gaussian lifting.
Results
The method achieves state-of-the-art results on PSGS and generalizes effectively to the Ava-256 dataset, with quantitative and qualitative improvements over previous methods.
Takeaways & Limitations
Next-scale autoregression provides an effective backbone for photorealistic, view-consistent human face synthesis with simultaneous multi-view outputs.
Takeaways & Limitations
Training data contains only neutral expressions and no accessories, limiting generalization to unseen expressions and accessories; background color bleeding also affects views.
Abstract
from arXiv · showhide
Photorealistic novel view synthesis of people remains challenging at high spatial resolutions and across multiple target cameras, where preserving identity, fine appearance details, and geometric coherence is critical. We build on the next-scale autoregressive paradigm and adapt it for human-centric view synthesis by enabling higher image resolutions, multi-view outputs and stronger cross-view consistency in a single forward pass. We train on a synthetic dataset of human faces spanning diverse identities and apparel. Contrary to diffusion models, this paradigm does not need 2D pre-training and, thanks to its next-scale architecture, it benefits from lower-resolution, general-purpose pre-trainings, with the full-sized purpose-specific images being used only in the last training stages. This enables our architecture to converge with a smaller amount of purpose-specific training data, allowing us to use a smaller but more realistic training dataset. The resulting model produces sharp and realistic views, with the option to synthesize multiple novel viewpoints simultaneously for improved agreement across views. Empirically, we observe gains in perceptual fidelity and cross-view coherence on human subjects, demonstrating that next-scale autoregression is an effective backbone for scalable, multi-output human view synthesis. We also couple our pipeline with an existing transformer-based model for pixel-aligned 3D gaussian lifting from multi-view facial inputs, resulting in accurate and photorealistic 3D models of human faces.
1 Introduction
The paper extends next-scale autoregressive transformers to photorealistic, high-resolution human face view synthesis with multiple simultaneous inputs and outputs. It targets identity, appearance, and geometric consistency across viewpoints while reducing reliance on purpose-specific data and compute.
- The method addresses high-resolution human face synthesis across wide viewpoint changes while preserving identity, geometry, shading, and appearance consistency.
- It extends next-scale autoregressive transformers to support 512×512 images and multiple simultaneous input and output views.
- Cross-view attention operates in a single forward pass by allowing tokens at each scale to attend across all output views and previous scales.
- The training strategy enables high-resolution convergence without retraining from scratch, lowering required compute and purpose-specific data.
- The pipeline couples multi-view synthesis with transformer-based pixel-aligned 3D Gaussian lifting for downstream human-face reconstruction.
- The approach is also applied to human body reconstruction without architectural changes as a potential future direction.
2 Related works
Prior work spans diffusion-based and autoregressive generation, parametric and deep-learning face reconstruction, and fast volumetric approaches. These methods commonly trade off image quality, viewpoint coverage, consistency, speed, or data requirements.
- Diffusion models generate high-quality images, while traditional autoregressive approaches have generally lagged in image quality and diversity.
- Next-scale transformers replace raster-scan token generation with scale-based generation to improve image modeling and speed.
- Traditional 3D morphable models represent face shape and appearance parametrically, whereas GAN-based methods often struggle with non-frontal consistency.
- Face reconstruction methods include approaches that converge with relatively small 3D face-scan datasets and improve consistency using multiple conditioning views.
- VoluMe uses a fast 2D U-Net to generate a volumetric face representation from one image, making it particularly suitable for near-frontal views.
3 Method
The method encodes input views at multiple scales, conditions a next-scale transformer on camera information, and generates all target views scale by scale. A staged training strategy adapts object-focused pretraining to high-resolution, multi-view human-face synthesis.
- Pipeline: The pipeline generates six canonical views before lifting them into a full 3D head, and it does not require input camera poses.
- Image encoding: The VQ-VAE encoder maps images to discrete multi-scale latent feature maps, while the decoder progressively reconstructs RGB images from those scales.
- Architecture: The architecture encodes inputs into multi-scale features, combines camera-pose conditioning with global embeddings, and decodes transformer outputs into images for Gaussian lifting.
- Next-scale transformer: Next-scale generation predicts progressively higher-resolution scales, conditioning each scale on previously generated scales.
- Next-scale transformer: All output views are generated at each scale before the model advances, enabling cross-view attention and correlations across multiple detail levels.
- Training strategy: Training adapts a pre-trained ArchonView model to 512×512 resolution by enlarging existing scales rather than adding more scales.
- Training strategy: The later stages specialize models on about 2.9K PSGS training subjects and then train simultaneous prediction of six canonical views.
4 Experiments
Experiments evaluate the proposed human-face view-synthesis pipeline on PSGS and AVA-256, including canonical views, turntable views, full image-to-3D pipelines, and component ablations. The method generally outperforms baselines, while results are less reliable on AVA-256 and generalization remains limited by training-data coverage.
- Datasets: PSGS contains 3.2K subjects, including 288 validation subjects, and provides 32 turntable views plus 32 perturbed camera and head-pose variants per subject.The dataset is smaller but more realistic than existing synthetic datasets and supports loosely posed training inputs.
- Metrics: The evaluation uses PSNR, SSIM, LPIPS, DreamSim, and ArcFace to measure image quality, perceived similarity, feature-space distance, mid-level perceptual distance, and identity shift.ArcFace assigns 0 when face detection disagrees between prediction and ground truth, and 1 when neither image contains a detected face.
- Novel-view synthesis results: On PSGS six-view synthesis, the proposed method outperforms baselines across most metrics, except for slightly better ArcFace performance from Splatter Image (PSGS).Splatter Image produces blurry back and side views and slight side-view misalignments, while FaceLift-NVS produces detailed but artificial-looking results.
- Full image-to-3D results: On PSGS 32-view turntable evaluation, the proposed method again outperforms baselines across the metrics and can render arbitrary target views without an additional step.The single-input single-output model provides more detailed non-frontal results than Splatter Image, whereas the full pipeline is recommended when cross-view consistency is needed.
- Cross-dataset evaluation: AVA-256 results are less reliable because camera-pose definitions require manual alignment and released experimental code is unavailable, despite automatic background removal and alignment.With the same training data, the proposed method fully outperforms FaceLift; FaceLift shows improved identity preservation when using its larger training dataset.
- Ablation studies: Ablations show that adapting ArchonView to 512×512 while preserving the number of scales is crucial, and omitting classifier-free guidance improves PSNR from 19.99 to 20.28 and ArcFace from 0.4044 to 0.4296.Larger models struggle to converge at the new resolution, motivating an initial resolution-adaptation training stage.
- Limitations: The model struggles with unseen expressions and accessories because the training data contains only neutral expressions and no accessories.All methods also suffer from background color bleeding, especially when brighter backgrounds worsen hair-color contamination.
5 Conclusions
The method achieves state-of-the-art novel-view synthesis results on PSGS and generalizes effectively to Ava-256, while remaining applicable beyond faces. Future work targets more diverse training data and further VAR specialization.
- The method achieves state-of-the-art results on PSGS in both quantitative metrics and qualitative visual quality.
- The approach also performs effectively on Ava-256, indicating generalization across datasets and scenarios.
- The pipeline can extend beyond human faces to domains such as full-body reconstruction.
- Future work includes exploring more diverse training data and developments in the next-scale VAR framework.
A.1 Training details
Training proceeds in staged phases for single-input, multi-output, and multi-input models, using substantial A100 compute and separately fine-tuning the VQ-VAE.
- The first training stage lasts 50 hours on 64 NVIDIA A100 GPUs.
- The single-input model is fine-tuned for 18 hours, then specialized for multi-output generation for 2 hours on the same hardware.
- The multi-input model is fine-tuned for 14 hours on 128 A100 GPUs and specialized for multi-output generation for 2 hours.
- The VQ-VAE is fine-tuned for 30 epochs on 8 A100 GPUs using a codebook size V = 4096.
A.2 Multiple-output SOS token embedding
The study compares separate and shared multi-view SOS-token designs and selects separate tokens despite negligible PSNR differences because they are the most intuitive and straightforward.
- Three d12 SOS-token designs are evaluated at 256x256: separate output-view tokens, shared tokens with concatenated targets, and shared tokens without view information.
- 19.99, 19.85 and 19.85 PSNR are obtained for the three designs respectively, showing little to no impact from the choice.
- The separate-token design is selected because it is the most intuitive and straightforward.
A.3 A different scenario: human bodies
The method transfers to human body reconstruction without architectural changes, producing canonical and arbitrary target views and supporting direct turntable rendering alongside 3D Gaussian lifting.
- The single-input pipeline is trained on PSGS for human body reconstruction using the same process as for human faces, including VQ-VAE fine-tuning.
- The single-input single-output model can render any target view because it is not restricted to the six canonical views.
- Turntable views can be rendered directly by the method without the lifting step, demonstrating flexibility for body reconstruction.
- Qualitative validation results compare canonical views produced by the method with ground truth and input views.
A.4 Object NVS training
The models are initialized with ArchonView weights and adapted to 512×512 resolution during the first training stage. The SS3D dataset provides example object views for this training setting.
- ArchonView weights initialize the models before adaptation to 512×512 resolution.The resolution adaptation occurs during the first training stage.
- The SS3D dataset is used in the training process after the models are adapted to higher resolution.The passage identifies SS3D as the employed dataset and references example views.
- Figure A.3 shows example object views from the SS3D dataset.
A.5 Multi-scale VQ-VAE fine-tuning
Fine-tuning the VQ-VAE at the new resolution and on the new dataset improves reconstruction quality at low cost. The effect is evaluated by autoencoding six canonical validation views.
- Fine-tuning the VQ-VAE on the new resolution and dataset produces small but clear reconstruction-quality improvements.The passage characterizes fine-tuning as inexpensive and complementary to the main ablation study.
- The results are obtained by autoencoding the six canonical views of the PSGS validation set.
A.6 Additional qualitatives
Additional qualitative results cover human body reconstruction, canonical and turntable views, object views, VQ-VAE reconstructions, and results on PSGS and AVA-256.
- Human body reconstruction: Our method produces final PSGS human-body views after GS-LRM 3D lifting, compared with ground truth.
- Human body reconstruction: The method can render turntable views directly without lifting because it is not restricted to canonical views.
- Object views: Figure A.3 shows example object views from the SS3D dataset.
- VQ-VAE fine-tuning: Figure A.4 compares base and fine-tuned VQ-VAE reconstructions at 256 and 512 resolutions against ground truth, showing small improvements after fine-tuning.
- Additional datasets: Figure A.5 presents qualitative PSGS validation results for the six canonical views used by each model’s novel-view-synthesis component.
- Additional datasets: Figure A.6 provides additional qualitative results on AVA-256 for subject 20220310–1128–ZSC414.