AudioLift: Audio-Driven Single Image to Full-Head Gaussian Avatar via Multi-view Video Generation

• The task is challenging because a single portrait leaves unseen head geometry ambiguous, while audio-to-motion generation must preserve temporal coherence.Source: §1 · p. 2
• Parametric 3DMM and FLAME motion spaces struggle with subtle dynamics, extreme emotions, and movements of hair or accessories.Source: §1 · p. 2
• AudioLift generates a full-head 3D Gaussian talking avatar from a single portrait and audio, including vivid and extreme expressions.Source: §1 · pp. 1–2
• It jointly generates four audio-driven oblique views in a 2 × 2 grid with a frontal view, using self-attention to promote cross-view consistency.Source: §1 · p. 2
• A Gaussian reconstructor uses view-aware corruption and active-volume regularization to handle inconsistent views, while conservative rear-prior distillation completes unseen rear regions.Source: §1 · p. 2
• Experiments on benchmark datasets and in-the-wild examples show improved identity preservation, expression quality, and novel-view fidelity over existing methods.Source: §1 · p. 2

Audio-driven 2D talking heads

• Recent audio-driven video diffusion methods jointly model appearance and motion, including hair and headwear, but their single-view outputs do not provide explicit 3D novel-view rendering (Tian et al., 2024; Cui et al., 2025; 2024; Lin et al., 2025a; Gan et al., 2025; Tu et al., 2025; Cai et al., 2026).AudioLift adapts an audio-driven video model to generate synchronized views and reconstructs a 3D Gaussian avatar.Source: §2 · p. 3

3D talking avatars and parametric head templates

• Prior one-shot 3D avatar methods often rely on audio-driven 3DMM motion, while FLAME-based deformation methods require per-person optimization; AudioLift avoids both constraints (Ye et al., 2024; Kim et al., 2024; Gong et al., 2025; Xu et al., 2025; Taubner et al., 2025b;a; Jia et al., 2026).AudioLift reconstructs a full-head Gaussian avatar from generated audio-driven multi-view video without parametric motion or per-person optimization.Source: §2 · p. 3

Multi-view generation and reconstruction

• Prior multi-view video generation methods use video backbones to avoid temporal flicker but require reference video, pose, or 3DMM controls; AudioLift generates synchronized views from only a portrait and audio (Taubner et al., 2025a; Cheng et al., 2026; Jin et al., 2026; Lyu et al., 2026b).Earlier consistency strategies share self-attention across view tokens or add dedicated cross-view layers (Shi et al., 2024; Gao et al., 2024; Kant et al., 2024; Huang et al., 2024; Liu et al., 2024; Gao et al., 2025; Bai et al., 2025).Source: §2 · p. 3

3.1 FRAMEWORK OVERVIEW

• A shared Gaussian reconstruction lifts five calibrated views at each timestep into a 3D head avatar, enabling rendering from arbitrary viewpoints and times.The Gaussian sequence benefits from the generated videos’ temporal coherence while its explicit 3D representation maintains multi-view consistency.Source: §3.1 · p. 3, Figure 2 · p. 4

3.2 STAGE I: AUDIO-DRIVEN MULTI-VIEW VIDEO GENERATION

• Stage I generates four oblique views in a 2×2 canvas plus a frontal stream, using shared self-attention and same-time audio cross-attention to exchange information and synchronize motion.The five views are interleaved in one transformer sequence; speech features from a frozen Wav2Vec2 encoder provide time-matched audio context.Source: §3.2 · pp. 4–5

3.3 STAGE II: INCONSISTENCY-ROBUST MULTI-VIEW GAUSSIAN LIFTING

• Stage II applies a feed-forward, ray-conditioned Gaussian reconstructor independently to each frame, producing an ordered sequence of 3D scenes that represents the 4D avatar.Its pixel-aligned decoder predicts Gaussians parameterized by center, anisotropic scale, rotation, opacity, and color.Source: §3.1 · p. 3, §3.3 · p. 5
• To tolerate generated-view disagreement, training corrupts oblique views with grid distortion or camera jitter while keeping the identity- and expression-informative frontal view and reconstruction targets clean.Because generated expressions may differ from ground truth, the reconstructor is trained on recorded multi-view videos with these structured input corruptions.Source: §3.3 · p. 5
• Active-volume regularization makes opacity-carrying Gaussians outside a canonical head volume transparent and moves active outliers toward its boundary.This addresses depth ambiguity that can otherwise leave visible exterior sheets in novel views.Source: §3.3 · pp. 5–6
• For unseen rear views, low-weight photometric and perceptual distillation from a frozen FaceLift teacher encourages plausible completion without requiring subject-exact rear supervision.The method caches teacher renders from cameras sampled across three rear yaw ranges because NeRSemble captures only the frontal arc.Source: §3.3 · p. 6

4.1 EXPERIMENTAL SETUPS

• Evaluation uses 42 RenderMe-360 clips from 21 identities and 21 HDTF portrait-audio pairs, with 81-frame, 25-fps quantitative clips and arbitrary-length inference also supported.Training uses 2,641 NeRSemble sequences from 328 identities; Stage II uses 648 clips and 10,368 synchronized multi-view scenes.Source: §4.1 · p. 6
• Comparisons use released-checkpoint baselines on RenderMe-360 and HDTF, with frontal and novel-view results on RenderMe-360 but frontal-only results on HDTF; appearance and identity are evaluated separately.Appearance metrics are PSNR, SSIM, LPIPS, and FID; identity uses reference-image ArcFace cosine similarity. Frontal appearance metrics use a common 512 × 512 white-background scoring canvas, and novel-view metrics use the same canvas; implementation uses 8 NVIDIA RTX 6000 Ada GPUs.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• Across viewpoints, AudioLift maintains coherent head and face-neck structure, reconstructs a plausible head back, preserves accessories, and reproduces wider mouth openings than the compared methods.The comparisons show head-neck separation and implausible rear views for Real3D-Portrait (Ye et al., 2024), side-view artifacts and inconsistent expressions for SDTalk (Jia et al., 2026), and distorted non-frontal facial structures for NeRFFaceSpeech (Kim et al., 2024).Source: §4.2 · p. 7, Figure 4 · p. 7, Figure 5 · p. 8
• On RenderMe-360, AudioLift records FID 44.53 vs. 77.05–108.08 for baselines and achieves the best novel-view PSNR, SSIM, and LPIPS.The novel-view evaluation tests generalization from the frontal reference to non-frontal viewpoints.Source: §4.2 · pp. 6–7, Table 1 · p. 7
• On HDTF, AudioLift leads in PSNR, SSIM, LPIPS, and FID, while its CSIM-ref of 0.666 is close to the best baseline’s 0.671.HDTF comparisons are frontal-only because the dataset lacks synchronized multi-camera ground truth.Source: §4.2 · p. 7, Table 2 · p. 8

4.3 ABLATION STUDIES

• Adding a fifth frontal view sharpens frontal reconstructions and improves all three reported metrics over using the four generated views alone.The comparison keeps the generated four-view inputs and reconstruction checkpoint fixed; Figure 6(a) shows fewer artifacts and closer resemblance to ground truth.Source: §4.3 · p. 8, Figure 6 · p. 9, Table 3 · p. 9
• View-aware corruption reconciles inconsistent views, active-volume regularization suppresses floating Gaussians, and rear-prior distillation improves the back-of-head appearance.Direct fine-tuning can fit observed cameras while producing incorrect side and rear geometry; the three components target distinct failure modes.Source: §4.3 · pp. 8–9, Figure 6 · p. 9

5 CONCLUSION

• Experiments on RenderMe-360 and HDTF, plus in-the-wild examples, show expressive facial motion, consistent novel-view rendering, and improved geometric completeness across 360°.The framework accounts for inconsistencies among generated views during full-head Gaussian avatar reconstruction.Source: §5 · p. 9

AI USE STATEMENT

• Generative AI assisted implementation, translation, evaluation discussions, manuscript preparation, formatting, and literature discovery; authors reviewed AI-assisted work and retained responsibility for the final content.AI tools did not originate the core method or generate research datasets; authors made final experimental decisions and scientific judgments.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The paper warns that single-portrait, audio-driven avatars could enable impersonation, fabricated speech, and misleading media, and recommends permissions, synthetic-content labeling, and avoiding harmful uses.The authors say experiments use existing research datasets, document sources and licenses for additional in-the-wild assets in the Appendix, and caution that generated examples are not authentic recordings or statements.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The paper points to Sections 3 and Appendices B–F for architecture, implementation, initialization, training, data, evaluation, and ablation details intended to support reproduction.Appendix E specifies model initialization, training schedules, optimizer settings, random seeds, and inference configurations; Appendix F documents dataset construction, identity splits, preprocessing, and baseline generation.Source: REPRODUCIBILITY STATEMENT · p. 10

A ADDITIONAL IN-THE-WILD RESULTS

• The in-the-wild appendix presents qualitative results on paintings, anime, stylized art, and photographs driven by English, Japanese, and Swahili speech.Frames in Sections A.1 and A.2 are selected by phoneme, choosing the clearest target-sound instance among frames where the audio is most certain.Source: A ADDITIONAL IN-THE-WILD RESULTS · p. 15
• Table 4 reports an auxiliary mouth-opening measurement using automatically selected phoneme times across all 16 clips in the curated set.The appendix presents this measurement alongside the selected frames.Source: A ADDITIONAL IN-THE-WILD RESULTS · p. 15

A.1 COMPARISON WITH PRIOR METHODS

• Across all 16 in-the-wild clips, AudioLift opens wide on open vowels and fully closes on /m/, /b/, and /p/, while baselines maintain nearly constant narrow apertures.Table 4 measures the mean opening at open vowels minus the mean opening at those bilabials; the supplied passage gives no numerical values.Source: Table 4 · p. 15, §A.1 · p. 15

A.2 TEMPORAL CONSISTENCY

• Long-window strips track each clip over time, allowing identity, accessories, and background drift to be checked beyond a single instant.Figure 16 shows a Van Gogh self-portrait, a wide-brimmed hat and face-paint portrait, and an anime portrait.Source: §A.2 · p. 20, Figure 16 · p. 20

A.3 LONG-VIDEO INFERENCE

• AudioLift maintains quality and identity on 25-second singing and 22.6-second speech inputs, well beyond the training window.Figure 17 presents the two long-audio examples.Source: §A.3 · p. 20, Figure 17 · p. 20

A.4 MULTI-VIEW CONSISTENCY

• Across five azimuths and open- and closed-mouth instants, identity, accessories, and mouth shape remain consistent as the camera moves.Examples include paintings, an anime portrait, and a portrait with a wide-brimmed hat and face paint.Source: §A.4 · p. 20, Figure 18 · p. 21, Figure 19 · p. 21

A.5 SOURCES OF THE IN-THE-WILD PAIRS

• The in-the-wild appendix uses 16 image–audio pairs selected by visual inspection from 150 candidates, not a random sample; none comes from training or evaluation datasets.The 16 references comprise eight FFHQ photographs and eight Wikimedia Commons images; backgrounds are removed and references are composited and framed to 832 × 1216.Source: §A.5 · p. 23

B EVALUATION PROTOCOLS AND METRIC IMPLEMENTATION

• The appendix specifies protocols for frontal PSNR, SSIM, LPIPS, and FID; seven-view CSIM-ref; paired novel-view reconstruction; and two quantitative ablations.External comparisons and ablations use different datasets, frame ranges, and PSNR masks.Source: B EVALUATION PROTOCOLS AND METRIC… · p. 24

B.1 EVALUATION COHORTS

• HDTF evaluation uses the same 21 subjects across methods, with 3.24-second audio windows and 81 video frames selected from reference-verified, single-speaker segments.Two clips use a fallback that may cross a pause; the reference portrait and audio are shared across methods.Source: §B.1 · p. 25

B.2 SHARED FRONTAL EVALUATION PROTOCOL

• Frontal comparisons exclude the reference-conditioned frame 0 and score frames 1–80 on a shared 512 × 512 canvas, yielding 3,360 RenderMe-360 and 1,680 HDTF pairs per method.RenderMe-360 uses camera 25 and HDTF uses source video; metrics are averaged first within clips, then equally across identities.Source: §B.2 · pp. 25–26

B.3 FRONTAL PSNR

• Frontal PSNR is computed per paired frame after floating-point error calculation with peak RGB value 255, then averaged in dB rather than computed from clip-aggregated MSE.Source: §B.3 · p. 26

B.4 FRONTAL SSIM

• Frontal SSIM uses the same frame pairs, canvas, frames 1–80, and hierarchical averaging as PSNR, with scikit-image’s default 7 × 7 uniform window and sample covariance.RGB channels are aggregated into one scalar per frame with channel axis 2 and data range 255.Source: §B.4 · pp. 26–27

B.5 FRONTAL LPIPS

• Frontal LPIPS uses the same paired frames, 512 × 512 canvas, frames 1–80, and hierarchical averaging, with fixed AlexNet weights in evaluation mode.The evaluator uses lpips.LPIPS(net=alex, version=0.1) and returns one distance per paired frame.Source: §B.5 · p. 27

B.6 FRONTAL FID

• Frontal FID pools Inception features across frames into one value per method, rather than averaging per-frame or per-clip FIDs.On HDTF, each GT and method distribution uses 1,680 feature vectors from the 21 evaluated subjects; reported values are point estimates without confidence intervals.Source: §B.6 · p. 27

B.7 REFERENCE-IMAGE IDENTITY SIMILARITY (CSIM-REF)

• At 320 × 320 detector input, 6,508 NeRFFaceSpeech frames missed at 640 × 640 are scored, while other methods’ final CSIM-ref scores change by at most 0.003.The smaller input recovers intended detections without changing which face is scored in the reported validation; multi-view CSIM-ref averages valid frame scores across seven yaw views.Source: §B.7 · pp. 27–28

B.8 PAIRED NOVEL-VIEW PSNR, SSIM, AND LPIPS

• Novel-view PSNR, SSIM, and LPIPS compare frame-0 renderings against simultaneous GT from five registered RenderMe-360 target cameras, producing 210 image pairs per method.Scores are averaged across cameras, sessions, and identities; HDTF is excluded because it lacks synchronized target cameras, and novel-view FID is not reported.Source: §B.8 · p. 28

Evaluation data

• Both Table 3 ablations use eight NeRSemble identities disjoint from training, with predictions paired to ground truth from the same identity, time index, and target camera.Each identity contributes one clip, and internal-ablation images use a 416 × 608 canvas rather than the external benchmarks’ 512 × 512 canvas.Source: §B.9 · p. 28

Fifth frontal view

• The fifth-frontal-view ablation compares reconstructions from the same final five-view generator and frozen FaceLift reconstructor, with versus without its generated frontal output.It evaluates frames 0–80 from frontal camera 222200037, yielding 648 image pairs per condition. PSNR uses GT foreground pixels, while SSIM and LPIPS use the full image.Source: §B.9 · p. 29

View-aware corruption

• The view-aware-corruption ablation compares released FaceLift, its checkpoint after 3,240 adaptation updates, and a further 3,240 updates with view-aware corruption.All checkpoints receive the same five Stage-I-generated views and are evaluated across 16 calibrated training-pool cameras on frames 1–80, for 10,240 image pairs per row; no synthetic corruption is added at evaluation.Source: §B.9 · p. 29

Metric implementation and aggregation

• Scores are averaged first within each identity and then equally across the eight identities, with PSNR computed per image before averaging.PSNR uses float64 RGB squared errors; SSIM uses channel axis 2, data range 255, and a uniform 7 × 7 window; AlexNet LPIPS v0.1 uses RGB mapped to [−1, 1]. Aggregation uses unrounded values, with table values rounded to two decimals for PSNR and four for SSIM and LPIPS.Source: §B.9 · p. 29

B.10 GENERALIZATION TO HELD-OUT CAMERAS

• The 12-camera variant holds out cameras 222200046, 222200040, 222200049, and 222200039 from source-view sampling and real-target supervision during training.The experiment uses eight training-disjoint identities and five fixed Stage-I-generated input views; each checkpoint reconstructs one Gaussian cloud per time index for rendering at all four held-out cameras against simultaneous GT observations.Source: §B.10 · p. 29
• Table 7 separates anchor-frame scores from frames 1–80, with 32 image pairs per checkpoint for frame 0 and 2,560 for frames 1–80.Scores use full 416×608 RGB images on white, calibrated cameras and the same renderer, without post-hoc Gaussian filtering or GT masking; the frontal camera is excluded. Temporal scores also reflect generated-versus-recorded performance differences.Source: §B.10 · p. 29, Table 7 · p. 30
• The full 12-camera model improves all three mean metrics over released FaceLift in both temporal splits; it has the highest PSNR, while the corruption-trained intermediate checkpoint has slightly better SSIM and LPIPS.The corruption-trained checkpoint also improves all three metrics over the preceding fine-tuned checkpoint.Source: §B.10 · pp. 29–30, Table 7 · p. 30

B.11 MOUTH-OPENING CONTRAST

• Mouth opening is the central inner-lip distance divided by mouth-corner distance, measured on 512 × 512 frontal frames using InsightFace buffalo 68-point landmarks and the largest detected face.The contrast is mean normalized opening for /a/-type open vowels minus the mean for /m/, /b/, and /p/, pooling valid tokens across 16 clips rather than averaging clip means.Source: §B.11 · p. 30
• The diagnostic greedily decodes audio phonemes with wav2vec2-lv-60-espeak-cv-ft, retains tokens with peak posterior at least 0.5, and samples the nearest 25-fps frame to each token-run midpoint.Frames for /b/ and /p/ are shifted 40 ms earlier; missing detections and out-of-range frames are excluded.Source: §B.11 · p. 30
• For each method, the reported mouth-opening contrast is maximized over one shared integer offset across all clips within [−6, 6] frames, not separately aligned per clip.At the selected offsets, all methods have 82 open-vowel and 81 bilabial observations, with none missing; the measure is an auxiliary diagnostic of audio-conditioned opening–closing contrast.Source: §B.11 · p. 30

C STAGE-I IMPLEMENTATION DETAILS

• Stage I uses StableAvatar’s audio-conditioned Wan2.1-1.3B diffusion backbone, with hidden width D = 1536 and 30 transformer blocks.The context index t = 0 is separate from generated-video frame indices.Source: C STAGE-I IMPLEMENTATION DETAILS · p. 31

C.1 CLEAN REFERENCE-FRAME CONDITIONING

• Frame-wise reference conditioning preserves sharper facial details than channel-wise conditioning, especially for degraded or off-frontal portraits.Both strategies produce plausible multi-view outputs for standard frontal portraits; the reference is a fixed shared identity anchor, not an output frame.Source: §C.1 · p. 31, Figure 22 · p. 31

C.2 TOKEN LAYOUT, POSITIONAL COORDINATES, AND DECODING

• Canvas and frontal tokens are grouped by time, assigned distinct spatial RoPE coordinates, and exchange information through global self-attention without an explicit view embedding.The shared context position aligns the clean reference latent with a zero placeholder in the frontal stream.Source: §C.2 · pp. 31–32
• The shared output head splits each time group into canvas and frontal patches, removes context predictions, and decodes the streams separately into four oblique views and one frontal view.The streams use matching denoising schedules, and the canvas video is divided into four cells.Source: §C.2 · p. 32

C.3 AUDIO, IMAGE, AND TEXT CONDITIONING

• Frozen Wav2Vec2-Base features are projected to width 1536 and windowed into 21 audio contexts, each with 15 positions in the standard 81-frame example.The 3.24-second, 25-fps example yields 161 speech features; out-of-range indices are zero-padded on the right and remain unmasked in attention.Source: §C.3 · p. 32
• Two frozen audio-adapter blocks refine each projected audio window using the corresponding noisy canvas patch tokens as visual keys and values.The adapter excludes the clean reference group and uses visual context embedded before the first diffusion-transformer block.Source: §C.3 · p. 32
• In each of 30 diffusion-transformer blocks, canvas and frontal tokens query their time-matched refined audio context through shared-weight audio cross-attention.The adapter runs once per forward pass, and its output is reused across all 30 blocks before being recomputed at the next denoising step.Source: §C.3 · p. 33
• Frozen image and text encoders supply portrait and prompt features through inherited StableAvatar pathways, whose attention outputs are summed with audio before projection.The image cross-attention pathway is separate from the VAE reference context.Source: §C.3 · p. 33

C.4 TRAINING SCOPE AND LOSS DETAILS

• The five-view model adds patch embedding and the output head to the four-view warm start, reaching approximately 283.7 million trainable parameters.The four-view warm start fine-tunes only transformer self-attention; the five-view model is initialized from it.Source: §C.4 · p. 33
• The VAE, text/image/audio encoders, audio adapter, cross-attention layers, and transformer feed-forward layers remain frozen in both models.Source: §C.4 · p. 33
• Both streams use independently drawn Gaussian noise but share the same sampled flow time when constructing their flow-matching inputs and targets.For each stream s, z_s,σ = (1 − σ)z_s + σϵ_s and u_s = ϵ_s − z_s.Source: §C.4 · p. 33
• The clean reference and zero frontal context are excluded from flow-matching supervision.Source: §C.4 · p. 33
• A shared random draw selects full-frame, face-focused, or lip-focused weighting with probabilities 0.4, 0.1, and 0.5, respectively.Full-frame weighting uses 1 + m_face + m_lip; the other modes use their corresponding masks.Source: §C.4 · p. 33
• Each stream uses masks resized to its latent resolution, falls back to the face mask for an absent or empty lip mask, and averages weighted squared error over the full latent tensor.Inherited flow-time weighting is applied, and the two stream losses are summed with frontal coefficient 1.0.Source: §C.4 · p. 33
• StableAvatar’s auxiliary difference loss is an unmasked mean squared error on adjacent latent channels and applies only to the canvas stream.Source: §C.4 · p. 33

D.1 TRAINING CAMERA POOLS

• The 12-camera variant holds out four NeRSemble cameras from source sampling and real-target supervision, while the 16-camera variant uses all cameras and does not establish held-out-camera generalization.The 16-camera checkpoint is retrained from the released FaceLift checkpoint on the same 648-clip subset and curriculum, without using RenderMe-360 or HDTF evaluation data.Source: §D.1 · p. 35
• Both training variants reconstruct from the same five Stage-I views; 12 and 16 refer to training-pool size, not simultaneous reconstruction inputs.Inference uses four grid views and one frontal view, and the 16-camera variant is not used to establish generalization to the four held-out NeRSemble cameras.Source: §D.1 · p. 35

D.2 VIEW-AWARE CORRUPTION

• For each five-camera source set, the most-frontal view stays clean while grid distortion and camera jitter are independently enabled with probability 0.5 on the other four views.The anchor is selected by angular proximity to frontal camera 222200037; corruption affects source inputs only, leaving reconstruction targets and target cameras unchanged.Source: §D.2 · p. 35
• Grid distortion follows LGM (Tang et al., 2024), perturbing per-scene control grids with strength 0.5 while leaving camera parameters unchanged.The grid uses 8–16 control points per axis, bounded perturbations, fixed image-boundary endpoints, piecewise-linear interpolation, and bilinear image sampling.Source: §D.2 · p. 35
• Camera jitter applies independent view-wise rotations of up to 3° about camera-up and 1.5° about camera-right, changing pose while preserving distance to the scene origin.Intrinsics and source images remain unchanged, and the same jitter settings are used for both camera-pool variants.Source: §D.2 · p. 35

D.3 ACTIVE-VOLUME REGULARIZATION

• The active-volume loss suppresses opacity outside a fixed axis-aligned head box and moves opacity-carrying outlier centers toward its boundary.The box is [−0.91, 0.91] × [−0.91, 0.91] × [−1, 1] in scene coordinates; distance is zero inside or on the box and penalizes centers, not Gaussian extent. The two global Gaussians are excluded.Source: §D.3 · pp. 35–36
• Its complementary terms penalize the exterior share of total opacity and opacity-weighted squared distance, with gradients separately targeting opacity reduction and outlier movement.The exterior term gives opacity gradients but no position gradients; the distance term detaches opacity to provide position gradients, while detached normalization avoids incentivizing increased interior opacity. The distance weight is β = 1 and overall weight λAV = 0.1 when enabled; otherwise the contribution is zero.Source: §D.3 · p. 36

Stage-I initialization and training

• Stage I initializes a four-view model from released 1.3B StableAvatar weights for 9,599 updates, then trains five-view weights for 6,800 more.The selected cooldown checkpoint is halfway through a 2,000-update linear decay, at approximately 5 × 10−5 learning rate; training uses 81-frame clips and global batch size eight for five-view runs.Source: E TRAINING SCHEDULES AND INFERENCE… · p. 37, Table 9 · p. 37

Stage-II data and training

• Stage II adapts from released FaceLift using 10,368 scenes and 51,840 five-exposure examples, supervising five source views and three additional real target views.Its two 3,240-update adaptation phases each traverse the schedule once; the full-camera model after a final 150-update refinement is used for external comparisons.Source: E TRAINING SCHEDULES AND INFERENCE… · p. 37, Table 10 · p. 37

Corruption seeds and rear targets

• Rear-view teacher targets are sampled across three yaw intervals from cached clean canonical views, and teacher supervision is weighted 0.2 in both final phases.Real and teacher rendering losses combine pixel MSE with 0.5 times inherited VGG19 perceptual loss; active-volume weight is 0.1 in two indicated refinement phases.Source: E TRAINING SCHEDULES AND INFERENCE… · pp. 37–38

Inference and long sequences

• Stage I uses 30 Euler denoising steps; long sequences use 81-frame windows with overlapping latent updates crossfaded at each step except the first.The nominal overlap is five latent frames with stride 16; Stage II reconstructs each time index independently from five calibrated views, rendering requested views from the same predicted Gaussians.Source: E TRAINING SCHEDULES AND INFERENCE… · p. 38

F DATASET CONSTRUCTION AND BASELINE GENERATION

• The NeRSemble split contains 2,641 training clips from 328 identities and 81 development clips from ten separate identities, with external datasets reserved for evaluation.Each clip uses an 81-frame, 25-fps window and corresponding 3.24-second audio; identities remain within one split, and development identities support checkpoint selection.Source: F DATASET CONSTRUCTION AND BASELINE… · p. 39
• Stage II uses captured synchronized images from a fixed 648-clip subset spanning all 328 training identities, with five source views in both camera variants.Images are foreground-matted and composited over white; the 12-camera variant excludes four cameras included in the 16-camera variant.Source: F DATASET CONSTRUCTION AND BASELINE… · p. 39

F.1 BASELINE VIDEO GENERATION

• On RenderMe-360 and HDTF, each baseline uses the same portrait and 51,840-sample speech segment and produces 81 frames at 25 fps.Frontal appearance metrics score exported videos, while identity scoring follows a separate native-video procedure.Source: §F.1 · pp. 39–40
• Real3D-Portrait uses released audio-to-motion and torso models with static source pose, periodic blinking, and its released head, torso, and background compositor.The evaluated output is an 81-frame 512 × 512 video generated from the source crop and audio using the released HuBERT/F0 processing path.Source: §F.1 · p. 39
• SDTalk uses its released FLAME-based source tracker and keeps source camera and eye measurements fixed while audio changes over the sequence.Its source is segmented and resized to 512 × 512; the model outputs 512 × 512 RGB video.Source: §F.1 · pp. 39–40
• NeRFFaceSpeech performs a fresh source-image inversion and pivotal tuning for each clip before generating audio-driven frames with its released deformation and inpainting operations.Inversion runs 1,000 updates; PTI runs up to 2,000, stopping when source AlexNet LPIPS reaches 0.06.Source: §F.1 · p. 40

F.2 CALIBRATED NOVEL-VIEW RENDERING

• For calibrated novel views, five target cameras render the same frame-0 identity and expression state, transferred with one clip-level similarity mapping based on physical calibration and face landmarks.The mapping is shared across target cameras; calibration metadata are used only for camera registration.Source: §F.2 · p. 40
• Each baseline uses a method-specific camera adapter to render calibrated target views from its source-fitted state.Real3D-Portrait adjusts source framing around an EG3D camera; SDTalk transfers pose using PnP and rasterizes its Gaussian features; NeRFFaceSpeech transfers cameras through its aligned image transform and recomputes mouth inpainting per view.Source: §F.2 · pp. 40–41