Video Diffusion Avatars: Taming Video Diffusion Models into Controllable 3D Avatars

• Learned expression spaces can exceed 3DMM fidelity but carry viewpoint bias, which DISTRACT reveals and the VDA disentangler removes for expression control.The introduction also notes that 3DMM-driven expression control loses fidelity through 2D-to-3D registration.Source: §1 · p. 2
• VDAs use the video diffusion model itself as the avatar, conditioning it on a single identity image to control expression, upper-body pose, and camera viewpoint.The approach builds no explicit 3D asset and preserves the video prior.Source: §1 · p. 2
• A proxy mesh jointly encodes camera viewpoint and upper-body pose while exposing separate control over both.Source: §1 · p. 2
• Camera triplets from public multi-view data teach multi-view consistency, with training taking 34 hours on one A100.The reported training preserves performance on challenging in-the-wild inputs.Source: §1 · p. 2
• VDAs outperform state-of-the-art single-image 3D avatars on BecomingLit, Multiface, and VFHQ-test, improving LPIPS by 29% and JOD by 66% over the strongest baseline.Source: §1 · p. 2
• The retained video prior produces shadows, reflections, refractions, and hair dynamics that are difficult to render explicitly.Source: §1 · p. 2

• 3DMMs often miss micro-expressions and mouth interiors, while learned expression latents can go beyond the linear 3DMM space but show viewpoint bias.The paper identifies cycle-consistency and end-to-end video learning among approaches for learning expression spaces.Source: §2 · p. 2
• Avatar systems commonly use NeRFs or 3DGS, while diffusion-distillation methods convert generated multi-view images into explicit 3D avatars.These methods trade the photorealistic diffusion prior for an explicit 3D asset.Source: §2 · p. 3
• Many diffusion portrait-animation methods lack explicit camera control; other VDM avatars add camera or expression controls, or distill generated views into 3D.The Figure 2 architecture injects expression separately from camera and upper-body pose, using a proxy mesh for the latter controls.Source: §2 · p. 3, Figure 2 · p. 3

3 VIDEO DIFFUSION AVATARS (VDAS)

• The avatar function takes a reference image, per-frame disentangled expression codes, and target-camera proxy mesh and skeleton signals, then synthesizes the video directly.Because no explicit 3D avatar is built, multi-view consistency must be learned rather than built into a renderer.Source: §3 · p. 3

3.1 EXPRESSION CONTROL AND THE DISTRACT BENCHMARK

• X-NeMo reaches only 18% Rank-1 on viewpoint invariance, while an MLP probe raises it to 98%, showing expression survives but is entangled with viewpoint.The DISTRACT benchmark tests viewpoint and identity invariance; Table 1 reports results on raw codes and after an MLP probe.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 LATENT EXPRESSION INJECTION

• Disentangled X-NeMo expression codes are translated into WAN-Animate’s native space, recombined with a mean camera code, then passed through face-fuser blocks while DiT blocks remain frozen.The method fine-tunes the translator, entangler, disentangler, face-token and face-fuser components, and pose encoders.Source: §3.2 · p. 4

3.3 POSE CONDITIONING

• A FLAME-head/SAM-3D-Body proxy mesh encodes camera rays in head- and pelvis-aligned frames, separating camera viewpoint from neck rotation for joint camera and pose conditioning.The method also replaces 2D skeleton detections with skeletons derived from SAM-3D-Body for novel-view control.Source: §3.3 · pp. 4–5

3.4 TRAINING WITH CAMERA TRIPLETS

• Three-camera samples pair a reference view, driving clip, and target view; differing driver and target cameras suppress viewpoint leakage in expression codes during rectified-flow training.This public-data strategy trains in 34 hours on a single GPU.Source: §3 · p. 3, §3.4 · p. 5

4 EXPERIMENTS

• WAN-Animate-14B is fine-tuned on curated NeRSemble and AVA-256 data for 34 hours on one A100 GPU.Inference uses 20 denoising steps and a proxy-mesh head from monocular pixel3DMM tracking, with body and neck from SAM-3D.Source: Implementation details · p. 5

4.1 EVALUATION SETUP

• Evaluation uses single reference images, driving videos, and target videos from different viewpoints, plus a static 3DGS fit tested at 16 rig cameras.The held-out reenactment datasets are BecomingLit and Multiface; the probe tests views outside the generated camera trajectory.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 RESULTS

• VDAs outperform every baseline on every metric across BecomingLit, Multiface, and VFHQ-test.The VFHQ-test results are reported in Table 3.Source: §4.2 · p. 6, Table 3 · p. 7
• Figure 4 presents self-reenactment on BecomingLit and Multiface, where VDAs retain sharper identity detail and transfer subtle and rare expressions.The rare expressions described are outside the FLAME space, which limits the baselines.Source: §4.2 · p. 6, Figure 4 · p. 6
• Cross-reenactment drives the reference identity with another person’s expression and renders it to the target camera.Without a ground-truth target, the evaluation uses identity similarity to the reference and expression and keypoint distances to the driver as seen from that camera.Source: Figure 5 · p. 7, §4.2 · p. 7
• Figures 7 and 8 test wide camera sweeps and an orbit to the back of the head, beyond the viewpoints of the training captures.Figure 7 isolates viewpoint consistency from expression transfer; Figure 8 shows a 360° novel-view synthesis setup.Source: Figure 7 · p. 7, Figure 8 · p. 8
• The static 3DGS probe measures whether generated views agree and remain faithful to the subject; the reported fits outperform every baseline.Figure 6 shows the fitted reconstruction re-rendered at a rig camera against ground truth.Source: §4.2 · p. 7, Figure 6 · p. 7
• The self-reenactment ablations use BecomingLit v3 (n=25), the only ablation dataset.Table 5 reports the ablation scope.Source: Table 5 · p. 8

5 LIMITATIONS AND FUTURE WORK

• VDAs take about 12 minutes to generate 77 frames at 576 × 1024 on one A100, and long sequences are prone to temporal drift.Unlike a built 3DGS avatar, they do not render in real time after construction.Source: §5 · p. 9
• A four-step autoregressive prototype generates 4.8 frames per second at 288 × 512 on one A100 and stays stable for over a thousand frames, with a small quality loss.The implementation is unoptimized and uses Self Forcing distillation.Source: §5 · p. 9
• Lighting is baked into the reference image, expression space is fixed rather than trained end-to-end, and the recipe is anchored to WAN-Animate-14B.The authors expect the recipe to transfer to other backbones.Source: §5 · p. 9

6 CONCLUSION

• VDAs turn a single photograph into a multi-view-consistent video avatar by controlling expression, pose, and camera viewpoint without building a 3D asset.The conclusion describes the generations as more faithful than both 3DGS-based and diffusion-based avatars.Source: §6 · pp. 9–10
• Learned expression spaces can capture expressions more faithfully than 3DMMs, but must be disentangled from camera viewpoint for best results.The recipe is stated to work with different expression spaces, including image-encoder spaces.Source: §6 · p. 10
• The preserved video prior supports rendering shadows, reflections, refractions, hair dynamics, and motion-deforming clothing, including beyond training subjects, effects, and viewpoints.Training uses public multi-view data and takes 34 hours on a single GPU.Source: §6 · p. 10

ETHICS STATEMENT

• Single-image avatar videos can enable impersonation and non-consensual synthetic media.The statement identifies this as a potential use of the capability.Source: ETHICS STATEMENT · p. 10
• The authors train and evaluate on multi-view datasets whose participants gave informed consent and whose authors released them for research use.The named datasets are NeRSemble, AVA-256, BecomingLit, and Multiface.Source: ETHICS STATEMENT · p. 10
• Evaluation also uses the public VFHQ benchmark.The statement cites VFHQ as a public benchmark.Source: ETHICS STATEMENT · p. 10

• The authors say they will release the full code and trained model checkpoints.The code covers preprocessing, training, inference, and evaluation.Source: REPRODUCIBILITY STATEMENT · p. 10
• Training uses the public NeRSemble and AVA-256 datasets, while evaluation uses public BecomingLit, Multiface, and VFHQ datasets.Source: REPRODUCIBILITY STATEMENT · p. 10
• The paper maps its sections to architecture and training, hyperparameters, proxy-mesh construction, disentanglement, evaluation protocols, and baseline procedures.Source: REPRODUCIBILITY STATEMENT · p. 10

• AI-generated portrait images serve only as qualitative reference inputs and are used neither for training nor quantitative evaluation.Source: AI USE STATEMENT · p. 10
• The authors used generative AI to implement specified methods, create and edit code, search literature, and turn author-written outlines into draft prose.Source: AI USE STATEMENT · p. 10
• The authors report reviewing and testing generated code, checking evaluation numbers against source files, and manually revising AI-assisted text.Source: AI USE STATEMENT · p. 10

• The paper reports face-crop PSNR and LPIPS, video-level JOD, ArcFace identity similarity, DECA-expression similarity, and normalized keypoint distance for pose.Source: A EVALUATION METRICS · p. 15
• Pixel-aligned metrics use ground-truth 3DMM alignment on multi-view benchmarks, but VFHQ-test uses facial-keypoint alignment and common head-scale cropping.Source: A EVALUATION METRICS · p. 15
• Cross-reenactment omits pixel metrics because it lacks a ground-truth target image; identity is measured against the reference and expression and pose against the driver video.Source: A EVALUATION METRICS · p. 15

B EVALUATION PROTOCOL DETAILS

• DISTRACT tests viewpoint and identity invariance, while reenactment evaluation includes 25 BecomingLit self-cases, 13 cross-cases, 22 Multiface cases, and 44 VFHQ-test clips.Source: B EVALUATION PROTOCOL DETAILS · pp. 15–16
• The 3DGS probe fits static scenes to generated frames along an elliptical camera path, then compares renders at 16 real rig cameras against captured photos.Source: B EVALUATION PROTOCOL DETAILS · p. 16

C BASELINE DETAILS

• Compared methods receive the same reference, driving video, and target calibration, but use method-specific expression estimates and conditioning; the ablations vary expression and proxy-mesh inputs.Source: Baselines · p. 16

D FACTORPORTRAIT COMPARISON DETAILS

• FactorPortrait’s comparison uses project-page self-reenactment clips without camera parameters or a separate driving video, and the authors state that the setup favors FactorPortrait.Source: D FACTORPORTRAIT COMPARISON DETAILS · p. 16

E IMPLEMENTATION DETAILS

• Training uses 4,696 81-frame clips from 640 identities and fine-tunes 861.5M WAN-Animate parameters; inference uses 20 Euler steps at 576 × 1024.Source: E IMPLEMENTATION DETAILS · p. 17

F.1 PELVIS-ANCHORED KINEMATIC FITTING

• Pelvis-anchored fitting reduces median pelvis drift from 52–79 mm to 6–12 mm and shoulder wander from 11.2 to 0.07 mm, with unchanged face residual.The face residual is 3.04 versus 3.06 mm; the shoulder pin also reduces frame-to-frame jitter by 14×.Source: §F.1 · p. 18

F.2 POSE-AWARE PROXY-MESH COLORING

• Proxy-mesh colors use ray direction and Lambertian shading in each body part’s local frame, so camera changes shift both regions while neck rotation changes head color relative to torso.Source: §F.2 · p. 18, Figure 11 · p. 19

G DISENTANGLEMENT MODULE DETAILS

• The disentangler uses cycle consistency to require recovery after swapping latents and supervised contrastive learning to group expression codes by label.Source: G DISENTANGLEMENT MODULE DETAILS · pp. 19–20

I TRAINING-FREE USE OF THE DISENTANGLED CODES

• With frozen generators, disentangled codes preserve expression during X-NeMo head rotation, while re-entangling WAN-Animate codes to the target view removes cross-view mouth artifacts.X-NeMo’s camera code moves the head rather than the viewpoint; the training-free WAN-Animate variant retains artifacts and coarse camera control.Source: I TRAINING-FREE USE OF THE… · p. 22, Figure 12 · p. 23, Figure 13 · p. 24

J EMERGENT VIEW INVARIANCE

• Cross-view training raises DISTRACT rank-1 accuracy from 0.244 to 0.792, while self-driven training reaches 0.486.The runs differ only in whether the driving expression code comes from a different camera or the reconstructed frame’s camera.Source: J EMERGENT VIEW INVARIANCE · p. 22, Figure 14 · p. 24

K STREAMING DISTILLATION PROTOTYPE

• The four-step student denoises clips about 11× faster, matching identity, expression, and keypoint accuracy at moderate image-quality loss, but remains below real time.On one A100 it generates 4.8 frames per second at 288 × 512; a 41-second, 1029-frame video takes about four minutes.Source: K STREAMING DISTILLATION PROTOTYPE · p. 25, Table 9 · p. 25, Figure 15 · p. 26

L FULL QUANTITATIVE RESULTS

• Table 10 covers eight evaluated encoders and reports that classical 3DMM regressors are consistently less viewpoint-biased than the learned encoders.The benchmark includes 500 Task 1 triplets and 126 Task 2 triplets.Source: Table 10 · p. 25, L FULL QUANTITATIVE RESULTS · p. 25

M ADDITIONAL QUALITATIVE RESULTS

• Qualitative comparisons cover BecomingLit, Multiface, and VFHQ; eight identities are shown from the larger test sets for the first two benchmarks.The BecomingLit and Multiface examples were selected for expression diversity and narrowed through manual review.Source: M ADDITIONAL QUALITATIVE RESULTS · p. 26, Figure 16 · p. 27, Figure 17 · p. 28