Imitator: Personalized Speech-driven 3D Facial Animation

1. Introduction

• Facial motions are central to communication, and known subjects’ animations need to match their idiosyncrasies to avoid appearing unnatural or implausible.The paper motivates audio-driven animation for cases where a face cannot be tracked or captured.Source: §1 · p. 2
• Imitator outperforms state-of-the-art methods in metrical evaluation and user study while matching target subjects’ motions in style and expressiveness.The user study also finds personalized facial expressions important for perceived realism.Source: §1 · p. 2
• A transformer predicts generalized motion representations, which an adaptable decoder maps to new users’ facial expressions using commodity-camera RGB video.The approach disentangles generalized viseme generation from person-specific motion decoding.Source: §1 · p. 2
• A lip-contact loss uses bilabial consonant cues to improve closure of the lips for ’m’, ’b’, and ’p’.The loss is motivated by the importance of plausible lip closures for realistic facial animation.Source: §1 · p. 2

2. Related Work

• Talking-head methods synthesize photorealistic video from audio, whereas Imitator predicts 3D facial animation for driving digital avatars.The related-work discussion also contrasts video-generation approaches with animation that does not require retraining the entire model for a new person’s motion style.Source: §2 · p. 2
• Earlier methods animated predefined facial rigs using procedural rules, while data-driven methods learned viseme patterns from data.Later approaches used pretrained speech models to represent input audio before CNN or autoregressive models mapped it to 3DMM space or directly to 3D meshes.Source: §2 · pp. 2–3
• Imitator’s audio pipeline encodes speech, predicts generalized motion features with an autoregressive viseme decoder, then decodes person-specific facial expressions.The final outputs are vertex displacements on a template mesh.Source: Figure 2 · p. 3

3. Method

• A style-agnostic transformer provides generic audio-driven motion features, while a short subject video guides identity-specific style adaptation.A bilabial-based lip contact loss further targets more plausible closures.Source: §3 · p. 3

3.1. Model Architecture

• Audio features feed an autoregressive viseme decoder; a motion decoder combines its identity-agnostic outputs with a style embedding and deformation basis to produce facial meshes.The viseme decoder avoids identity-specific motion inputs, while the motion decoder maps style-aware visemes into the motion space.Source: §3.1 · pp. 3–4

3.2. Training

• Training uses autoregressive prediction and combines vertex reconstruction, velocity, and lip-contact losses.The velocity loss is intended to improve temporal consistency; the lip-contact loss targets bilabial consonant frames, whose ground-truth faces have closed mouths.Source: §3.2 · p. 4

3.3. Style Adaptation

• For a new subject, the method first optimizes the style embedding against tracked reference faces, then refines the deformation basis using reconstruction and velocity losses.The two-stage process reuses the pretrained motion decoder; basis refinement can capture person-specific deformations such as asymmetric lip motions.Source: §3.3 · p. 5

4. Dataset

• VOCAset contains 12 actors, each with 40 sequences lasting 3–5 seconds; its test split contains two actors.The dataset provides audio and high-quality 3D face reconstructions at 60 fps, sampled at 30 fps for these experiments.Source: §4 · p. 5
• New-subject adaptation uses a short video clip and a 3DMM-based tracker to obtain per-frame 3D face shape.The external-sequence procedure assumes a monocular RGB video of about two minutes, divided into train, validation, and test sequences.Source: §4 · p. 5
• For VOCAset, transcript-audio alignment identifies bilabial consonant times, and local minima in lip distance before those times mark likely closures.Figure 3 illustrates detected closures; lip-loss weights are assigned around them.Source: §4 · p. 5, Figure 3 · p. 5
• External-video adaptation estimates subject shape from the first frame, fits per-frame FLAME blendshape parameters, and uses reconstruction and velocity losses without bilabial labels.The resulting coefficients provide the per-frame face meshes used to adapt the motion decoder.Source: §4 · p. 5

5.1. Ablation Studies

• The complete method produces expressive motion and accurate lip closures; removing style yields generic, muted expressions, while removing Llip yields improper closures.The per-vertex error maps show missing person-specific details when style adaptation is replaced with the training style.Source: Figure 5 · p. 7, §5.1 · p. 8

5. Results

• Imitator achieves the lowest lip reconstruction and lip-sync errors while performing on par with baselines on full-face metrics.The evaluation also reports that a single 5-second reference video yields significantly better lip scores than the baselines.Source: Table 1 · p. 6, §5 · p. 8
• In a 56-participant A/B study, Imitator consistently outperforms VOCA and Faceformer and achieves realism and lip-sync similar to ground truth.Participants had not seen the actors’ original speaking styles; the authors report that personalized synthesis led to more realistic-looking animations.Source: §5 · p. 8, Table 2 · p. 8
• On external sequences, Imitator adapts to each subject’s speaking style, while VOCA and Faceformer miss person-specific deformations and MeshTalk uses a different identity.MeshTalk’s pretrained model cannot be trained on VOCAset, so the comparison uses a different identity.Source: §5 · pp. 5, 8, Figure 4 · p. 6

5.1. Ablation Studies

• Using roughly 60 seconds of reference video, style-agnostic motion-decoder adaptation takes under 30 minutes, versus about 6 hours for an identity-dependent transformer.The paper also reports improved lip closures for bilabial consonants with the lip-contact loss.Source: §5.1 · p. 8
• Style-code optimization makes lip-distance amplitude and frequency resemble the reference, and motion-basis refinement further captures asymmetrical lip deformations.The figure compares the initial decoder, optimized style code, and refined motion basis against the reference lip-distance curve.Source: §5.1 · p. 8, Figure 6 · p. 8

• Style adaptation is limited to the speaking style of the subject in the reference video.The discussion does not claim support for transferring other speaking styles.Source: §6 · p. 8
• The method does not control talking style by emotion, such as sad, happy, or angry delivery.Conditioning the viseme transformer and motion decoder on an emotion flag is left for future work.Source: §6 · p. 8
• Expressiveness and facial detail depend on face-tracker quality; improved tracking is expected to yield better face shapes.Source: §6 · p. 8

• A short reference video is used to learn a personalized motion decoder driven by a generalized autoregressive transformer that maps audio to intermediate viseme features.Source: §7 · p. 9
• The studies find personalized facial animation essential for perceived realism, while the bilabial lip-closure loss further improves results.Source: §7 · p. 9
• The authors describe personalized facial animation as a stepping stone toward audio-driven digital doubles.Source: §7 · p. 9

• Quantitative metrics improve as the number of reference sequences increases, with each sequence lasting about 3–5 seconds.The study samples 1, 4, 10, or 20 sequences from the training set of VOCA test subjects.Source: §9 · p. 12, Table 3 · p. 12
• Even adaptation from a single reference sequence produces significantly better animation than the baseline methods.The authors identify this result as evidence of the impact of style on generated animation.Source: §9 · p. 12
• With more reference data, the lip-distance curve better fits the ground-truth curve on a test sequence.Figure 7 illustrates this trend for one sequence.Source: §9 · p. 12, Figure 7 · p. 12

10.1. Audio Encoder:

• The audio encoder interpolates audio features to the motion rate, processes them with 12 transformer layers, and projects them to 64-dimensional phoneme features.The encoder uses 12 attention heads.Source: §10.1 · p. 12

10.2. Auto-regressive Viseme Decoder:

• The autoregressive viseme decoder starts from a zero token and uses positional encoding, self-attention, and multi-head attention to generate context-weighted viseme representations.Its multi-head attention combines head outputs through a learned projection.Source: §10.2 · p. 12
• Audio-motion attention uses viseme queries and audio keys and values to align the modalities, with a diagonal bias favoring audio features at the current time.The resulting audio-context-injected viseme features feed motion synthesis.Source: §10.2 · pp. 12–13

10.3. Motion Decoder:

• A subject-adapted motion decoder concatenates style embeddings with viseme features and maps them to per-vertex 3D displacements.Its final deformation basis is adapted using a reference sequence.Source: §10.3 · p. 13

11. Broader Impact

• The animations may drive photorealistic digital doubles for audio-driven immersive telepresence in AR or VR.The paper also identifies character animation and entertainment as application contexts.Source: §11 · p. 13
• The authors warn that voice cloning combined with image synthesis could turn the animations into tools for identity theft, cyber mobbing, or other harmful crimes.They state that open, transparent research could raise awareness and that they will share implementation for multimedia-forensics research.Source: §11 · p. 13
• All study participants gave written consent for their video material to be used in the publication.Source: §11 · p. 13