Source-linked AI summary
Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer
Ivan Mikheev, Viacheslav Vasilev, Anna Dmitrienko, Alexey Letunovskiy, Ivan Kirillov, Kirill Chernyshev, Denis Dimitrov
TL;DR
Text-to-audio-video models generate soundtracks but cannot control the speaker’s identity. This paper adds reference-conditioned voice cloning with a single zero-initialized layer, achieving the highest speaker similarity across three verification networks while enabling roughly 30× faster audio-only inference.
Problem
Text-to-audio-video models lack control over the identity of the speaker in synthesized audio, limiting personalized and person-specific audio-visual generation.
Method
The method adds reference latents and global speaker-embedding modulation through a single zero-initialized linear layer, followed by short voice-aware fine-tuning.
Results
K6A 5B achieves the highest speaker similarity across all six SECS measures and three verification networks, significantly outperforming every competing system.
Takeaways & Limitations
Audio-only evaluation preserves voice-cloning behavior while reducing inference cost by approximately 30×, and the original text-to-audio-video functionality remains intact.
Takeaways & Limitations
A word-error-rate gap to dedicated text-to-speech systems suggests that text-fidelity loss remains an open extension.
Abstract
from arXiv · showhide
Text-to-audio-video (T2AV) generation models produce a video and its soundtrack from a textual description, but offer no control over whose voice speaks in the output. We show that a base T2AV model can be turned into a voice-cloning model by adding a single zero-initialized linear layer on top of its audio backbone, fine-tuning for a comparatively short training schedule, and conditioning on a short reference recording at inference time. The reference is injected through two complementary signals: its diffusion latents are prepended to the audio stream, and a global speaker embedding modulates token of the target audio. On a benchmark of 674 speaker-text pairs spanning 30 speakers we compare against five strong voice-cloning text-to-speech baselines: our enhanced 5B model attains the highest speaker-encoder cosine similarity (SECS) across three independent verification networks (ECAPA-TDNN, WavLM-SV, Resemblyzer), statistically significantly outperforming every baseline. A side product of the architecture is that the audio path can be evaluated without the video path at inference time, yielding a ~30x speed-up over the full audio-video diffusion loop while preserving the voice-cloning behaviour.
1. Introduction
Existing T2AV models generate synchronized video and audio but cannot control speaker identity, limiting personalized audiovisual applications. The paper introduces a minimal voice-cloning extension using one zero-initialized linear layer, short fine-tuning, reference conditioning, and isolated audio inference.
- Problem: T2AV diffusion models synthesize video and sound from text but realize speaker instructions as arbitrary samples, offering no speaker-identity control.This limits personalized content creation, avatar dubbing, and rendering of specific people.
- Method: The proposed drop-in extension adds voice-cloning conditioning with one zero-initialized linear layer on top of the audio backbone.Reference latents are prepended to the audio stream, while a global speaker embedding modulates audio signals through FiLM.
- Method: A single voice-aware fine-tuning stage enables voice cloning while preserving the base model’s original audio-visual generation quality.The recipe is designed as short fine-tuning on top of the pretrained T2AV backbone.
- Evaluation: 674 samples across unique-text speaker pairs were evaluated with three speaker encoders, and the model significantly outperformed strong TTS baselines in reference speaker fidelity.The evaluation protocol measures speaker similarity using three independent encoders.
- Inference: ∼30× speed-up is achieved by running the asymmetric AV-DiT audio path in isolation while preserving the full loop’s voice-cloning behaviour.This provides a practical inference variant that avoids running the video path.
2. Method
The method augments an asymmetric audio–video diffusion transformer with two complementary reference signals: prepended audio latents and global speaker FiLM modulation. Zero-initialized conditioning preserves the base model initially, while three-way classifier-free guidance separately controls text and speaker fidelity.
- AV-DiT backbone: d_v = 1792 and d_a = 896 define separate video and audio dimensions in an AV-DiT backbone connected by crossmodal attention.Both streams use the CrossDiT architecture, and the text condition is shared between them.
- Reference conditioning: Reference audio is injected through prepended VAE latents and a global speaker embedding that modulates target audio tokens with FiLM.The speaker embedding is 1024-dimensional and comes from a frozen Qwen3-TTS speaker encoder.
- Reference conditioning: 32 fused decoder blocks let target tokens attend to fixed reference tokens through existing self-attention, maintaining consistent timbre during denoising without new attention layers.At inference, only the target portion is updated while the reference portion remains fixed.
- Initialization: Zero-initialized FiLM weights and bias set (γ, β) = (0, 0), so the augmented model exactly matches the base T2AV model on its first forward pass.Reference tokens are excluded from FiLM modulation.
- Inference guidance: Three-way classifier-free guidance trades text fidelity against speaker fidelity at inference without retraining.Independent 0.1 dropout probabilities for text and the joint reference signal provide unconditional, text-conditioned, and fully conditioned configurations.
- Training: The entire model is fine-tuned in one voice-aware stage, while 10% of training steps use independently unnoised audio and video latents to preserve the original joint distribution.AdamW uses 1×10−5 for base parameters and 5×10−5 for W_film.
3. Experiments
Experiments on a 674-sample, 30-speaker VCTK benchmark show that K6A 5B achieves the strongest speaker similarity across six SECS measures and all competing systems. Reference-aware fine-tuning also preserves or improves reference-free generation, while audio-only inference enables an approximately 30× speed-up without sacrificing voice cloning.
- Benchmark: 674 samples span 30 native-English speakers with unique 5–16-word texts, separate reference and enrollment clips, and held-out target texts.The benchmark uses native 48 kHz VCTK studio-quality audio and avoids text overlap between samples.
- Benchmark results: K6A 5B achieves the highest speaker similarity on all six SECS columns, surpassing six competing systems across three verification networks.The lead includes WavLM-SV 0.944 vs. reference, Resemblyzer 0.866, and ECAPA-TDNN 0.766.
- Benchmark results: K6A 5B and K6AV LITE prioritize faithful reference imitation, while Qwen3-TTS and IndexTTS2 obtain lower WER/CER with cleaner but less reference-like voices.The paper attributes stronger speaker fidelity to the base T2AV model’s generative audio prior, while noting higher WER inherited from that prior.
- No-regression checks: Every reference-free objective improves after reference-aware fine-tuning, including FAD, transcription error, perfect-transcription rate, CLAP alignment, and UTMOS naturalness.The comparison uses 400 prompts covering 30 speakers with identical seeds and texts; the zero-initialized Wfilm path stays close to identity without a reference.
- Efficient inference: ∼30× speed-up results from reducing each diffusion step from a 3.18 B-parameter joint pass to an effective ∼0.58 B audio-only pass.The audio path bypasses the video sub-block and video VAE decoder without changing weights, while K6AV LITE preserves reference timbre.
4. Conclusion
The paper shows that T2AV diffusion models can become voice-cloning systems through a minimal architectural change and short voice-aware training. K6A 5B leads speaker similarity, while K6AV LITE preserves voice cloning at much lower inference cost; proposed extensions target transferability and text fidelity.
- Core contribution: A single zero-initialized linear layer, reference-latent prepending, and speaker FiLM convert T2AV diffusion models into voice-cloning systems.The change is trained in a short voice-aware stage.
- Evaluation: 674 samples: K6A 5B achieves the highest speaker similarity across three verification networks and significantly outperforms every baseline.The benchmark evaluates voice-cloning performance on speaker-text pairs.
- Efficient inference: ∼30× lower inference cost: K6AV LITE preserves the voice-cloning behavior while evaluating the audio path without the video path.The original text-to-audio-video functionality is also retained.
- Future work: Natural extensions include transferring prepend+FiLM to asymmetric AV-DiTs with frozen speaker encoders and adding a small text-fidelity loss to address the WER gap.The suggested loss could use CTC against the prompt without disturbing speaker conditioning.
A. Implementation details · A.1. Base architectures
K5 and K5-LITE use the same asymmetric AV-DiT backbone, combining video and audio streams with a shared text encoder. K5-LITE retains the topology while reducing text and output layers to approximately 3 B parameters.
- A.1. Base architectures: 32 fused decoder blocks form the shared asymmetric AV-DiT backbone of K5 and K5-LITE.Both models use the architecture described in Section 2.1 and Figure 1.
- A.1. Base architectures: 1792 is the video-stream width, with heads of dimension 64.The video stream is denoted d_v = 1792 in the passage.
- A.1. Base architectures: 896 is the audio-stream width, with heads of dimension 64.The audio stream is denoted d_a = 896 in the passage.
- A.1. Base architectures: 2-block text encoder consumes the prompt and feeds it to both the video and audio streams.The text encoder is shared across both streams.
- A.1. Base architectures: K5 uses the full 32 fused blocks.K5 therefore uses the complete block configuration of the shared topology.
- A.1. Base architectures: K5-LITE uses a reduced configuration with the same block topology but smaller text and output layers, totaling ≈3 B parameters.The passage truncates before specifying how many parameters are attributable to individual components.
A.2. Reference window sampling
Reference windows are sampled from the same training clip as the target audio, with a buffer separating them and durations tailored to speech or non-speech content.
- Reference window sampling: Training samples use the first audio slice as the target, while references come from the remaining clip audio with a 0.5 s buffer.The target is typically 5 seconds and aligned with video latents.
- Reference window sampling: Speech references last 1.2–2.5 s from a 12 s search window, whereas non-speech references last 1.5–3.5 s from a 3.5 s search window.Both reference types are sampled from the same clip as the target.
A.3. Hyperparameters
Fine-tuning uses AdamW with a 10^-5 base learning rate, accelerated updates for the new layer, staged warmup, conditioning dropout, and modality-clean training. Inference uses a fixed 50-step diffusion configuration with split classifier-free guidance.
- AdamW uses β1, β2 = (0.9, 0.95), weight decay 10−3, and a maximum gradient norm of 1.0.
- 10−5 is the base learning rate, while Wfilm and its biases use a 5× learning-rate multiplier.
- 8000 warmup steps are followed by a constant learning rate.
- Reference and text conditioning are each dropped with probability 0.1.
- 10% of steps clamp video latents to zero and 10% clamp audio latents to zero, preserving joint audio-video behavior during fine-tuning.
- 50 diffusion steps use split CFG with wt = 5 and wr = 4 under a fixed seed at inference.
A.4. Speaker encoder
The model uses a frozen Qwen3-TTS speaker encoder to produce FiLM conditioning embeddings from 24 kHz audio. During training, encoding runs once per sample on a CPU background worker, passing only a 1024-dimensional vector to the training step.
- Encoder and conditioning: The frozen speaker encoder is the publicly released Qwen3-TTS speaker network, operating on 24 kHz audio.It computes the FiLM embedding used for conditioning.
- Training implementation: Once per training sample, the speaker encoder runs on CPU in a background worker.This separates speaker-embedding computation from the main training step.
- Training implementation: Only the resulting 1024-dim speaker vector is transmitted to the training step.The encoder itself remains frozen during this process.
B. Audio-only inference details
The asymmetric AV-DiT checkpoint supports audio-only evaluation without weight changes by disabling the video stream while preserving the audio generation path.
- Audio-only inference details: Audio-only inference disables the video stream without changing weights, short-circuiting video computation in all 32 fused decoder blocks and skipping the video VAE decoder.The audio sub-block, speaker FiLM layer, and audio classifier-free-guidance loop continue operating as during training.
C. Additional analysis
An 82-run factor study identifies how reference length, guidance, diffusion steps, language matching, and pre-processing affect voice-cloning quality. Longer references help until 4 s, guidance is best at 4–6, and 30 diffusion steps are generally sufficient.
- Study design: 82 runs evaluated inference-time and data-time choices for voice-cloning quality on a held-out benchmark subset.The study examined reference length, guidance weight, diffusion steps, language matching, and reference pre-processing.
- Reference length: 4 s marks a reference-length plateau: similarity rises from approximately 1 s, while shorter than 2 s sharply reduces Resemblyzer SECS.Beyond 4 s, additional reference audio provides negligible returns; the speaker encoder appears to bottleneck very short references.
- Reference-guidance weight: 4–6 is the optimal range for the second CFG weight wr; values above 7 slightly over-smooth audio and reduce UTMOS.Below 3, the reference is under-imposed.
- Number of diffusion steps: 30 diffusion steps essentially saturate mean SECS, while increasing to 60 adds < 0.02 and doubles wall-clock cost.Routine use therefore does not benefit from the additional steps.
- Language matching: 0.05 lower mean SECS results, on average, when prompt and reference languages differ; matching them is recommended whenever possible.Cross-language generation remains possible despite the reduction.
- Reference pre-processing: Reference de-noising and VAD help noisy captures but can hurt clean studio references, so preprocessing is safest to skip when UTMOS is approximately ≥3.The recommendation depends on reference quality rather than applying preprocessing universally.