Real3D-Portrait: One-shot Realistic 3D Talking Portrait Synthesis

1 INTRODUCTION

• Prior 2D methods can distort at large head movements, while many NeRF methods require expensive individual training for unseen identities.Existing one-shot 3D approaches also struggle to achieve accurate reconstruction and animation simultaneously.Source: §1 · pp. 1–2
• A large Image-to-Plane model distills 3D prior knowledge from a trained 3D face generative model to reconstruct an input image’s 3D representation in one forward pass.A motion adapter uses PNCC to predict a residual motion diff-plane that edits the reconstructed representation.Source: §1 · p. 2
• The Head-Torso-Background Super-Resolution model separately handles head, torso, and background, including torso movement and switchable background rendering.Its components include head-image super-resolution, a warping-based torso branch, and a background branch.Source: §1 · p. 2
• A generic audio-to-motion model maps audio to PNCC, generalizes to unseen identities without adaptation, and supports eye-blink and mouth-amplitude control.Together, the framework supports both audio- and video-driven scenarios.Source: §1 · p. 2
• Real3D-Portrait outperforms existing one-shot talking-face systems and achieves performance comparable to state-of-the-art person-specific methods.Ablation studies are reported to support the effectiveness of each component.Source: §1 · p. 2

2D/3D Face Animation

• The paper frames one-shot 3D talking-face generation around reconstructing an input image’s 3D face and rendering it to match motion or audio.Its inference pipeline takes one source image with a video or audio driving condition to synthesize a talking avatar with torso and background.Source: §2 · p. 2, Figure 1 · p. 3

3D Face Representation

• 3D Morphable Models provide a geometry prior but have low-fidelity meshes and omit regions such as hair, hats, and eyeglasses; NeRFs and tri-planes support more detailed 3D face representations.GAN inversion can be slow and temporally jittery, while predictor-based image-to-plane mapping is more efficient and stable at inference.Source: §2 · pp. 2–3

2D/3D Face Animation

• 2D warping methods can preserve image fidelity but face artifacts and distortions at large head poses without a strict 3D constraint.Earlier GAN-based 2D animation methods are described as unstable in training and poor in visual quality.Source: §2 · p. 3
• NeRF-based talking-face methods combine image fidelity with geometry constraints, but many require tens of hours of training per identity (Guo et al., 2021; Tang et al., 2022; Ye et al., 2023a).One-shot tri-plane approaches either animate in 2D before iterative 3D GAN inversion (Ma et al., 2023; Li, 2023; Trevithick et al., 2023) or predict and morph a 3D representation, whose generalizability is limited by few large-view video frames (Li et al., 2023a;b).Source: §2 · p. 3

3.1 IMAGE-TO-PLANE MODEL FOR 3D FACE RECONSTRUCTION

• The I2P model maps a source image to a canonical tri-plane, using fused ViT and VGG branches and pre-training on multi-view images synthesized by EG3D.The ViT branch handles pixel-to-world canonicalization; the VGG branch captures high-frequency appearance features.Source: §3.1 · p. 4

3.2 MOTION ADAPTER FOR 3D FACE ANIMATION

• The motion adapter uses source and driving PNCCs to predict a residual diff-plane that edits the source tri-plane for the target expression.PNCCs encode facial expression using a 3DMM; the adapter concatenates source and driving PNCCs because the I2P tri-plane represents the source expression.Source: §3.2 · p. 5
• Training pairs randomly selected video frames as source and target, while a Laplacian loss regularizes adjacent motion diff-planes to reduce temporal jitter.The training stage learns the motion adapter and fine-tunes the I2P and volume-renderer/super-resolution modules using head-region images.Source: §3.2 · pp. 5–6

3.3 HEAD-TORSO-BACKGROUND SUPER-RESOLUTION MODEL

• HTB-SR models head, torso, and background separately, using keypoint-conditioned torso warping and an inpainted background before occlusion-aware feature fusion.The background is preprocessed with KNN inpainting; the authors note neural inpainting could produce a more realistic background.Source: §3.3 · p. 6

3.4 GENERIC AUDIO-TO-MOTION MODEL

• The A2M model uses a flow-enhanced VAE to predict 3DMM expression codes from audio, with landmark supervision, temporal smoothing, and eye-blink and mouth-amplitude controls.At inference, audio or a reference video supplies driving expression, and no further training is required for a new identity.Source: §3.4 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation compares video- and audio-driven baselines using CelebV-HQ for motion-adapter and HTB-SR training, VoxCeleb2 for A2M, and specified benchmark settings.The setup also describes baseline methods and their one-shot or person-specific status.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 QUANTITATIVE EVALUATION

• Real3D-Portrait leads video-driven comparisons in image quality, identity preservation, and motion accuracy, and beats one-shot audio baselines while approaching person-specific RAD-NeRF.For audio-driven generation, it has better lip synchronization than RAD-NeRF and comparable image quality and identity preservation.Source: §4.2 · p. 8, Table 1 · p. 8, Table 2 · p. 8

4.3 QUALITATIVE EVALUATION

• Qualitative demonstrations show PNCC animation, natural torso movement at large head poses, switchable backgrounds, and synchronized lip motion from the generic audio-to-motion model.Demo videos and additional visualizations are provided for comparison.Source: §4.3 · p. 8
• In user studies, Real3D-Portrait rates above one-shot methods for identity preservation and visual quality, near RAD-NeRF on those aspects, and above RAD-NeRF for lip synchronization.Among video-driven methods using ground-truth motion, it also has the best lip-synchronization rating.Source: §4.3 · pp. 8–9, Table 3 · p. 9

4.4 ABLATION STUDIES

• Pretraining improves I2P identity similarity, image quality, and expression accuracy; fine-tuning, the 87M model scale, and Laplacian loss also support reported outcomes.The paper reports better image quality at 87M than 50M parameters, with no obvious performance difference between the default and 200M models.Source: §4.4 · p. 9, Table 4 · p. 9
• HTB-SR ablations report worse visual quality with unsupervised keypoints or without background inpainting, while alpha-blending fusion supports identity preservation and image quality.Replacing alpha-blending with channel-wise concatenation also produces artifacts.Source: §4.4 · p. 9, Table 4 · p. 9

• The framework combines a pretrained large Image-to-Plane model and PNCC-conditioned motion adapter for one-shot 3D avatar reconstruction and animation.HTB-SR supports natural torso movement and switchable backgrounds; a generic audio-to-motion model supports video- and audio-driven applications.Source: §5 · p. 9

• The authors identify deepfake-related misuse as an ethical concern and plan license restrictions, visible watermarks, and invisible maker-identifying watermarks.They state that synthesized videos should be used only for educational or other legal purposes.Source: §7 · p. 10

A COMPARSION BETWEEN DIFFERENT METHODS

• Pretraining I2P on an image multi-view dataset and fine-tuning it with a motion adapter distinguish Real3D-Portrait from HiDe-NeRF and GOS-Avatar.The framework targets accurate 3D reconstruction and stable face animation.Source: A COMPARSION BETWEEN DIFFERENT METHODS · p. 14
• The motion adapter morphs the canonical tri-plane between expressions, unlike HiDe-NeRF’s deformation field or GOS-Avatar’s expression neutralization step.The authors say the neutralization step requires additional supervision and may lose source-image information.Source: A COMPARSION BETWEEN DIFFERENT METHODS · p. 14
• Real3D-Portrait separately models head, torso, and background, and adds a generic audio-to-motion model, whereas the baselines omit background modeling and support video-driven use only.The authors associate the separate segment modeling with realistic torso movement and overall video naturalness.Source: A COMPARSION BETWEEN DIFFERENT METHODS · p. 14

B.1 PRETRAINING IMAGE-TO-PLANE MODEL

• The I2P model is trained by reconstructing a canonical tri-plane from a reference image and rendering it from another camera; training pairs use EG3D-generated identities and views.The camera distance is varied from 2.4 to 5.0 rather than being fixed at 2.7.Source: §B.1 · pp. 14–15, Figure 4 · p. 15

B.2 OBTAINING PNCC FROM 3DMM COEFFICIENTS

• PNCC provides an appearance-agnostic facial-geometry condition, built from 3DMM identity and expression coefficients; driving expressions come from video or the audio-to-motion model.The paper obtains the coefficients from an image using 3DMM fitting.Source: §B.2 · pp. 15–16

B.3 WARPING-BASED TORSO BRANCH

• The torso branch uses camera-transformed 3DMM facial keypoints to estimate a deformation field and warp source-torso features.The authors motivate 2D warping by treating torso dynamics as nearly joint translation with the head.Source: §B.3 · pp. 16–17, Figure 6 · p. 16

B.4 BACKGROUND BRANCH WITH KNN-BASED INPAINTING

• The background branch inpaints source-person pixels with nearest background colors, extracts background features, and allows background switching at inference.A super-resolution branch integrates the rendered head with torso and background features into a 512×512 image.Source: Figure 7 · p. 16, §B.2 · p. 16, §B.4 · p. 17

B.5 OBTAINING THE HEAD AND TORSO OCCLUSION MASK

• Head and torso occlusion masks support alpha-blending-style fusion; the head mask comes from NeRF volume rendering, while the torso estimator predicts its mask alongside dense motion.The appendix identifies the head and torso masks as inputs to the segment fusion.Source: §B.5 · p. 17

B.6 DETAILED STRUCTURE OF AUDIO-TO-MOTION MODEL

• The audio-to-motion model combines a VAE with a flow-based prior and predicts a sequence of 64-dimensional 3DMM expression codes from audio.The predicted codes can be converted to PNCC using the Z-Buffer algorithm.Source: §B.6 · p. 17, Figure 8 · p. 18

C.1 MODEL CONFIGURATION

• The appendix points to Table 6 for the model configuration and its hyperparameter settings.The supplied table caption identifies it as “Model Configuration.”Source: §C.1 · p. 18, Table 6 · p. 18

C.2 TRAINING DETAILS.

• Training uses 8 NVIDIA A100 GPUs; I2P, the motion adapter, HTB-SR, and A2M train for 250,000, 200,000, 200,000, and 100,000 steps, respectively.The reported training times are about 72, 60, 30, and 16 hours, respectively.Source: §C.2 · p. 19

D.1 EVALUATION DETAILS

• Evaluation includes 100 same-identity and 100 cross-identity video pairs, plus 100 audio-driven test videos formed from 10 out-of-domain images and 10 audios.For audio-driven tests, head-pose sequences are taken from randomly selected CelebV-HQ videos whose first-frame pose is nearest to the source-image pose.Source: §D.1 · p. 19

D.2 USER STUDY SETTING

• Twenty participants rate each video on a 1–5 Likert scale for identity preservation, visual quality, and semantic-level audio-lip synchronization.The audio-lip synchronization rating explicitly disregards visual quality.Source: §D.2 · p. 19

D.3 ADDITIONAL QUALITATIVE RESULTS

• The PNCC animation illustration traces an input head image and driving PNCC through 128×128 volume rendering, a depth view, and a 512×512 image processed by the training-time naive SR module.The depth image is included to visualize the modeled avatar’s 3D geometry.Source: §D.3 · p. 19, Figure 10 · p. 20
• The video-driven comparison reports large-pose degradation for Face-vid2vid, identity loss for OTAvatar, and texture jitter for HiDe-NeRF, while Real3D-Portrait produces identity-preserving, realistic results.The figure caption recommends the demo video for clearer comparison.Source: §D.3 · p. 19, Figure 9 · p. 20
• The warping-based torso branch generates realistic torso segments under large and critical head poses.Source: §D.3 · p. 20
• The background branch supports switching the background during inference.Source: §D.3 · p. 20
• The generic audio-to-motion model produces audio-lip synchronization in the audio-driven scenario.Source: §D.3 · p. 20

D.4 ADDITIONAL ABLATION STUDIES

• Without I2P pretraining, image quality degrades and facial animation becomes inaccurate; the pretrained model yields high fidelity, identity preservation, and accurate motion control.Source: §D.4 · pp. 20–21
• The method demonstrates realistic torso segments across different head poses, switchable backgrounds, and accurate lip motion from its generic audio-to-motion model.Source: Figure 11 · p. 21, Figure 12 · p. 21, Figure 13 · p. 21
• Without alpha-blending fusion, head-torso boundaries appear blurry and hair can show hollow artifacts; the proposed fusion addresses both.A face mask limits background information within the head region and spatially restrains feature fusion at the boundary.Source: §D.4 · pp. 21–22

D.5 ADDITIONAL QUANTITATIVE COMPARISON WITH RECENT 2D BASELINES

• Real3D-Portrait achieves the best CSIM, FID, AED, and APD performance among the compared 2D baselines.The comparison is reported in Table 7.Source: Table 7 · p. 22, §D.5 · p. 22

E LIMITATIONS AND FUTURE WORK

• Without large-posed training images, the method fails on large head poses such as side views.Source: E LIMITATIONS AND FUTURE WORK · p. 23
• Image quality may improve with more high-fidelity training data and more carefully designed networks; a few-shot in-context method is proposed as desirable for identity preservation and visual quality.Source: E LIMITATIONS AND FUTURE WORK · p. 23
• During large pose motion, KNN-based inpainted backgrounds can leak unnaturalness; the authors propose a neural inpainting system or end-to-end learning within HTB-SR.Source: E LIMITATIONS AND FUTURE WORK · p. 23