PaperlayerView exact revision on arXiv

Source-linked AI summary

AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

Kwan Yun, Serin Yoon, Sunjin Jung, Jung Eun Yoo, Inyup Lee, Junyong Noh

arXiv:2608.16143v1cs.GRcs.CVcs.MMcs.SD

TL;DR

Existing audio-driven 3D speech animation methods require character-specific data or laborious rigging, limiting animation for arbitrary characters. AnyTalk adapts a video diffusion model with Character-specific Fine-tuning and converts generated talking-head video into blendshape animation without animation data. It also provides AnyTalkRT, a distilled real-time variant reaching 110 FPS.

  • Problem

    Audio-driven 3D speech animation commonly requires paired audio–3D data for each mesh or blendshape configuration, while manual lip-sync keyframing is time-consuming and expertise-intensive.

  • Method

    AnyTalk fine-tunes a talking-head video diffusion model on rendered target-character images with zeroed-out audio embeddings, then estimates blendshape parameters from the generated video through landmark-based optimization.

  • Results

    AnyTalk produces 3D speech animation for arbitrary characters without animation data, and AnyTalkRT achieves 110 FPS for real-time applications.

  • Takeaways & Limitations

    The method broadens audio-driven speech animation to arbitrary characters by combining character-personalized talking-head generation with 3D blendshape estimation.

  • Takeaways & Limitations

    The approach depends on predefined blendshapes and frontal-view, sparse 2D landmarks, while real-time distillation trades generation quality against inference time.

Abstract

from arXiv · show

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.

1 INTRODUCTION

AnyTalk addresses the data and manual-effort barriers that limit audio-driven 3D speech animation for arbitrary characters. It adapts talking-head video generation to a target character, then converts the generated video into blendshape-driven 3D animation without animation data.

  • Motivation: Manual lip-sync keyframing is time-consuming and expertise-intensive, while existing learned methods require paired audio–3D data for each mesh or blendshape configuration.These requirements make adoption difficult for independent developers and small studios.
  • Overview: AnyTalk uses a 2D talking-head generation model and uplifts its output into 3D animation for arbitrary characters without relying on 3D animation data for training.The pipeline first generates a speech-synchronized video, then estimates blendshape parameters through optimization.
  • Character adaptation: Character-specific Fine-tuning adapts a pretrained talking-head model to a target character using rendered character images paired with zeroed-out audio embeddings instead of video data.Freezing the temporal module while updating the spatial module preserves the model’s motion prior.
  • 3D uplift: Homography-based warping with talk-invariant landmark selection aligns rendered character images and generated video, enabling accurate blendshape estimation.The alignment supports landmark-matching optimization for the final 3D speech animation.
  • Real-time variant: AnyTalkRT distills the optimization-based system into a streamlined network for real-time applications, achieving 110 FPS inference.The real-time variant is trained with feature matching and reconstruction losses.

2 RELATED WORK

Prior work includes audio-driven 2D and 3D facial animation, diffusion-based talking-head generation, identity-specific methods, and mesh-oriented systems. AnyTalk extends character personalization to audio-driven video models without paired motion or audio data.

  • 2D talking-head generation: Audio-driven 2D facial animation uses representations including facial landmarks, 3D Morphable Model parameters, depth maps, and semantic maps.Recent diffusion and flow-based methods improve realism through larger networks and more extensive data.
  • 3D facial animation: Audio-driven 3D methods such as VOCA, FaceFormer, and CodeTalker model speech-to-motion using paired animation data, transformers, or quantized latent spaces.These methods represent different strategies for producing natural facial motion from audio.
  • Identity-specific methods: Single-identity approaches reduce per-character data requirements but still use audio-visual data for one identity.Retargeting and shared embedding methods are examples of this setting.
  • Mesh diversity: ScanTalk handles diverse mesh structures but depends on meshes manually aligned, resampled, and modified to match VOCASET properties.Its performance varies with how well an input mesh aligns with those properties.
  • Video diffusion personalization: AnyTalk is presented as the first method to personalize an audio-driven talking-head model to a target character without motion or audio data.It uses a zeroed-out audio embedding as a stop-moving signal to disentangle audio from motion.

3 METHOD

AnyTalk generates character-specific talking-head videos, then estimates blendshape parameters to convert them into 3D speech animation. Character-specific fine-tuning aligns generated videos with arbitrary target characters without speech-animation data, while landmark-based optimization transfers motion and suppresses unwanted pose variation.

  • Overview: The method has two stages: audio-driven video generation followed by blendshape-parameter optimization that produces 3D animation from generated frames.The input character must be rigged because the second stage estimates motion through blendshapes.
  • Overview: Pre-trained talking-head models can synchronize speech but exhibit domain mismatch, undesired head motion, and input inconsistency relative to rendered 3D characters.These discrepancies hinder blendshape estimation because the optimization compares rendered character images with generated video.
  • Character-specific Fine-tuning: Character-specific Fine-tuning adapts a general video model using rendered target-character images paired with zeroed audio embeddings, preserving speech-driven motion while reproducing the character appearance.Only the spatial residual network is trained; attention layers and ReferenceNet remain frozen to preserve the original audio understanding and motion prior.
  • Character-specific Fine-tuning: During inference, non-zero speech supplies motion information while wpose = 0, wexp = 1, and wlip = 2 produce dynamic lip motion with a stable head pose.This re-weighting supports the subsequent optimization by emphasizing speech-related motion and suppressing head movement.
  • Optimization: Landmark matching optimizes blendshape parameters by aligning rendered character vertices with generated-video landmarks after talk-invariant homography warping.Landmark vertices are identified once from the neutral mesh; the objective combines talk-landmark, asymmetric mouth-opening, and regularization losses, followed by Gaussian filtering to reduce temporal jitter.
  • Optimization: Large occluding components, including masks or highly stylized dental geometry, can interfere with ray-casting and require temporary removal or manual vertex assignment.This limitation affects automatic identification of talk-related landmark vertices on the neutral mesh.

4 EXPERIMENTS

Experiments evaluate AnyTalk across five heterogeneous characters, audio clips, qualitative lip synchronization, and ground-truth-free quantitative metrics. The method follows phonemes closely, preserves target-character appearance, and achieves the best reported scores on both LSE-D and LSE-C.

  • Experimental Setup: Experiments cover five characters with varied vertices, meshes, blendshape parameters, mesh types, and artistic styles.The evaluation uses 30 audio clips, producing 150 speech animations per method.
  • Qualitative Results: Seven key frames aligned with phonemes show that generated lip movements closely follow the spoken audio.The evaluation presents each phoneme or letter alongside its corresponding rendered frame.
  • Qualitative Results: Character-specific fine-tuning produces videos that follow the target character without visual mismatches, while optimization preserves phoneme-corresponding lip motion in the final animation.The two-stage visualization compares the DCsF video with the optimized 3D animation.
  • Baseline Comparison: AnyTalk more precisely follows the input sentence than ScanTalk, DiffSpeaker + NFR, and CodeTalker + NFR, which exhibit only subtle movements.The comparison uses nine frames from synthesized facial animations under identical rendering settings.
  • Quantitative Evaluation: AnyTalk achieves the best results on both LSE-D and LSE-C among the compared methods.ScanTalk achieves the second-best performance, while DiffSpeaker + NFR and CodeTalker + NFR produce similar results.

4.6 Ablation Study

Ablations show that character-specific fine-tuning, zeroed audio conditioning, frozen motion-related modules, landmark optimization, and stabilized source video each contribute to faithful 3D speech animation. Photometric optimization instead produces weaker motion and substantially increases runtime.

  • 4.6.1 Ablation Study on Video Generation: Removing CsF causes character mismatches and lip movements that fail to match the audio.Observed artifacts include squeezed pupils, double chins, and inconsistent appearances.
  • 4.6.1 Ablation Study on Video Generation: Removing Czero reduces expressiveness and mouth opening because motion signals are not correctly disentangled from the no-motion signal.The no-Czero variant produces less expressive lip motion in both video generation and final animation.
  • 4.6.1 Ablation Study on Video Generation: Not freezing the relevant modules causes insufficient mouth opening, attributed to motion-signal entanglement and overfitting in the motion module.The controlled fine-tuning study reports that restricting updates to spatial residual blocks preserves audio-driven dynamics, whereas training all layers causes catastrophic forgetting.
  • 4.6.2 Ablation Study on Final 3D Speech Animation: Removing talk-related landmark optimization creates chin artifacts because blendshapes are estimated using only a sparse mouth-opening loss.The ablation compares the full optimization process with alternatives that modify landmark filtering and losses.
  • 4.6.2 Ablation Study on Final 3D Speech Animation: Using only photometric loss yields no or subtle movement, while adding it to the original losses slightly degrades metrics and increases generation time from 3.12 s to 9.32 s per frame.The reported runtime increase is a factor of 2.96 and requires differentiable rendering at every iteration.
  • Head Pose Stabilization: Setting wpose = 0 during initial generation provides the cleanest source for 3D uplifting and yields the highest motion fidelity.The subsequent optimization suppresses head motion even when wpose is high, but stabilization during generation improves the source video.

4.7 User Study

A user study compares AnyTalk with three baseline systems on perceptual naturalness and lip synchronization. Participants significantly prefer AnyTalk in both categories and across all reported baseline comparisons.

  • Study Design: Twenty-one participants evaluated AnyTalk, ScanTalk, DiffSpeaker + NFR, and CodeTalker + NFR using randomized two-alternative forced-choice comparisons.Each participant assessed 75 side-by-side video pairs and answered questions about naturalness and lip synchronization.
  • Perceptual Results: Participants preferred AnyTalk over ScanTalk for perceptual naturalness in 413 of 525 comparisons (78.6%), with p < 0.001.Preferences were tested against a 50% chance level using exact binomial tests.
  • Perceptual Results: Participants preferred AnyTalk for lip synchronization over ScanTalk in 403 of 525 trials (76.8%), with p < 0.001.Comparisons with DiffSpeaker + NFR and CodeTalker + NFR also showed significant improvements for all metrics.

5 APPLICATIONS

The paper distills AnyTalk into AnyTalkRT for real-time arbitrary-character animation and evaluates the streamlined model against AnyTalk. It also applies the CsF and landmark-based optimization procedure to MEMO to demonstrate generality.

  • Real-time animation: AnyTalkRT is trained by distilling around 1,600 speech animations generated by AnyTalk into a streamlined network.The training uses feature matching and blendshape reconstruction losses.
  • Real-time animation: Distillation matches AnyTalkRT audio features to warped talk-related landmarks and reconstructs blendshapes, with λ set to 400.Training used 360 epochs, AdamW, batch size 128, and OneCycleLR scheduling.
  • Real-time animation: 9.09 ms per frame, or 110 frames per second, was achieved by AnyTalkRT in full-precision PyTorch inference.The streamlined inference excludes the optimization phase.
  • Generality: The same CsF and landmark-based blendshape optimization procedure was applied to MEMO, including spatial-layer fine-tuning with zeroed-out audio embeddings.The resulting MEMO adaptation is referred to as AnyTalkMEMO, while the fine-tuned model is MEMOCsF.
  • Real-time animation: The AnyTalkRT distillation pipeline uses Audio2Feat for input-audio feature extraction and Feat2BS for blendshape-parameter prediction.These components are shown in the distillation process.
  • Real-time animation: Although distillation enabled real-time performance, the lip-sync metric degraded relative to AnyTalk.This comparison is stated in the quantitative-results caption.

6 CONCLUSION

AnyTalk provides animation-data-free audio-driven 3D speech animation for arbitrary avatars through CsF and landmark-based optimization. The method supports real-time distilled inference, but its quality and applicability remain bounded by optimization cost, rigging, frontal-view dependence, and sparse landmarks.

  • 6 CONCLUSION: AnyTalk enables audio-driven 3D speech animation for arbitrary avatars without requiring animation data.The method is presented as a new direction beyond supervised, character-specific speech animation.
  • 6 CONCLUSION: Character-specific Fine-tuning with zeroed-out audio embeddings narrows the domain gap between generated videos and rendered character images.
  • 6 CONCLUSION: Landmark-based face alignment and matching uplift generated video into speech animations while prioritizing 3D geometric stability and view-independence.The paper notes that this priority can produce less pronounced lip movements than per-frame 2D-centric models.
  • Limitations and Future Work: AnyTalk’s optimization takes around 3.12 seconds per frame, creating a quality–inference-time trade-off addressed by distillation.
  • 6 CONCLUSION: AnyTalkRT depicts phoneme-following speech motion similar to AnyTalk while enabling real-time performance through distillation.
  • Limitations and Future Work: The method requires rigged input characters, depends on frontal views, and may miss subtle facial details because its 2D landmarks are sparse.Poor rigging, such as lacking a mouth-opening blendshape, can cause animation failure.

Supplementary Material for AnyTalk: Speech Animation for Arbitrary Characters

The supplementary material provides additional experimental details, extended comparisons, and analyses that complement the main AnyTalk paper.

  • Supplementary Material: The supplementary material is titled “Leveraging a Video Generation Model.”

1 CONTENT

The supplementary content organizes additional implementation details, comparisons, experiments, and analyses of viewpoint consistency, reconstruction mismatch, and naturalness.

  • Section 2: Section 2 covers the dataset, baseline implementations, additional comparisons, and extended experiments.The experiments include lip-sync sensitivity, evaluation without CsF, and temporal stability analysis.
  • Section 3: Section 3 analyzes inconsistent lip motion across viewpoints, unseen-view 3D reconstruction mismatches, and approaches for enhancing naturalness.

2 EXPERIMENTS DETAILS

Experiments evaluate AnyTalk across characters, baselines, inference settings, and lip-sync reliability. The method is compared qualitatively and quantitatively against retargeting and generic audio-driven animation approaches.

  • Experimental Setup: Five characters span distinct mesh structures, visual styles, and blendshape configurations, including both unified and separately partitioned facial meshes.Victor uses one mesh, whereas Emily, VMan, Malcolm, and Morphy divide facial components across multiple meshes.
  • Baselines: AnyTalk is evaluated against ScanTalk, DiffSpeaker, CodeTalker, and Choi et al. + Hallo using character-specific mesh handling and retargeting procedures.The comparisons include generic inference baselines, post-hoc alignment, and a non-personalized AnyTalk variant without CsF.
  • Comparison Results: 11.304 LSE-D and 3.155 LSE-C are achieved by Ours, compared with 16.571 LSE-D and 0.285 LSE-C for Choi et al. + Hallo.The comparison uses five characters and 30 audio clips, totaling 150 speech animations.
  • Metric Sensitivity: 11.65 LSE-D and 3.07 LSE-C are obtained for stylized characters, versus 11.05 LSE-D and 3.22 LSE-C for non-stylized characters.The reported performance gap between the two character groups is not statistically significant.
  • Personalization Ablation: Without CsF, AnyTalk outperforms DiffSpeaker + NFR and CodeTalker + NFR across all metrics, while CsF further improves LSE-D and maintains strong confidence.The non-personalized variant achieves the highest lip-sync confidence among the evaluated methods and remains competitive with ScanTalk in lip-sync accuracy.
  • Temporal Stability: Temporal stability is assessed by measuring average jerk of facial landmarks across consecutive generated frames for AnyTalk and AnyTalk_RT.Lower average jerk indicates smoother, less jittery motion, with landmarks extracted using XPose.

3 MULTI-VIEW GENERATION AND 3D RECONSTRUCTION ANALYSIS

The analysis tests whether multi-view generation or generic 3D reconstruction can overcome AnyTalk’s single-view dependency. Diffusion outputs vary across viewpoints, while reconstructed geometry deviates from target meshes, leaving integrated multi-view methods as future work.

  • Experimental Analysis: Supplementary experiments generate the same audio from left, frontal, and right viewpoints and reconstruct the resulting frames into 3D meshes.The viewpoints are left (−45°), front (0°), and right (+45°), using Hallo for generation and Hunyuan3D-2 for reconstruction.
  • Multi-View Generation: Generated frames preserve approximately intended head poses, but lip motion and facial expressions differ noticeably across viewpoints.Figure 4 compares input renders with generated frames at identical timestamps for the three views.
  • Multi-View Generation: Diffusion sampling produces non-deterministic outputs for identical audio, creating conflicting supervision signals that prevent sharp, geometrically consistent 3D convergence.This variability undermines multi-view optimization strategies that assume consistent motion priors across generated videos.
  • 3D Reconstruction: Hunyuan3D-2 reconstructions often deviate from source character shape and proportions, especially from left and right viewpoints.Distorted jawlines and misaligned landmarks indicate difficulty reproducing subtle target-mesh details from diffusion-generated images.
  • Implications: Naively combining diffusion generation across views or applying off-the-shelf reconstruction is insufficient to resolve the current single-view dependency.Future work is directed toward multi-view-consistent talking-head diffusion or end-to-end systems jointly optimizing view synthesis and geometric fidelity.
  • Enhancing Naturalness: Random expressions such as eye blinks can be added through blendshape weighting, while setting w = 1 applies a constant expression throughout the sequence.Random start times and durations control dynamic expressions.
Loading 2608.16143v1…