Adding Voice Cloning to Text-to-Audio-Video Models with a Single Zero-Initialised Layer

Problem

• T2AV diffusion models synthesize video and sound from text but realize speaker instructions as arbitrary samples, offering no speaker-identity control.This limits personalized content creation, avatar dubbing, and rendering of specific people.Source: §1 · p. 1

Method

• The proposed drop-in extension adds voice-cloning conditioning with one zero-initialized linear layer on top of the audio backbone.Reference latents are prepended to the audio stream, while a global speaker embedding modulates audio signals through FiLM.Source: §1 · p. 1
• A single voice-aware fine-tuning stage enables voice cloning while preserving the base model’s original audio-visual generation quality.The recipe is designed as short fine-tuning on top of the pretrained T2AV backbone.Source: §1 · p. 1

Evaluation

• 674 samples across unique-text speaker pairs were evaluated with three speaker encoders, and the model significantly outperformed strong TTS baselines in reference speaker fidelity.The evaluation protocol measures speaker similarity using three independent encoders.Source: §1 · p. 1

Inference

• ∼30× speed-up is achieved by running the asymmetric AV-DiT audio path in isolation while preserving the full loop’s voice-cloning behaviour.This provides a practical inference variant that avoids running the video path.Source: §1 · pp. 1–2

AV-DiT backbone

• d_v = 1792 and d_a = 896 define separate video and audio dimensions in an AV-DiT backbone connected by crossmodal attention.Both streams use the CrossDiT architecture, and the text condition is shared between them.Source: §2.1 · p. 2, Figure 1 · p. 2

Reference conditioning

• Reference audio is injected through prepended VAE latents and a global speaker embedding that modulates target audio tokens with FiLM.The speaker embedding is 1024-dimensional and comes from a frozen Qwen3-TTS speaker encoder.Source: §2.2 · p. 2
• 32 fused decoder blocks let target tokens attend to fixed reference tokens through existing self-attention, maintaining consistent timbre during denoising without new attention layers.At inference, only the target portion is updated while the reference portion remains fixed.Source: §2.2 · p. 2

Initialization

• Zero-initialized FiLM weights and bias set (γ, β) = (0, 0), so the augmented model exactly matches the base T2AV model on its first forward pass.Reference tokens are excluded from FiLM modulation.Source: §2.2 · p. 2

Inference guidance

• Three-way classifier-free guidance trades text fidelity against speaker fidelity at inference without retraining.Independent 0.1 dropout probabilities for text and the joint reference signal provide unconditional, text-conditioned, and fully conditioned configurations.Source: §2.3 · p. 2

Training

• The entire model is fine-tuned in one voice-aware stage, while 10% of training steps use independently unnoised audio and video latents to preserve the original joint distribution.AdamW uses 1×10−5 for base parameters and 5×10−5 for W_film.Source: §2.4 · pp. 2–3

Benchmark

• 674 samples span 30 native-English speakers with unique 5–16-word texts, separate reference and enrollment clips, and held-out target texts.The benchmark uses native 48 kHz VCTK studio-quality audio and avoids text overlap between samples.Source: §3.1 · p. 3

Benchmark results

• K6A 5B achieves the highest speaker similarity on all six SECS columns, surpassing six competing systems across three verification networks.The lead includes WavLM-SV 0.944 vs. reference, Resemblyzer 0.866, and ECAPA-TDNN 0.766.Source: §3.2 · p. 3, §3.1 · p. 3, Table 1 · p. 4
• K6A 5B and K6AV LITE prioritize faithful reference imitation, while Qwen3-TTS and IndexTTS2 obtain lower WER/CER with cleaner but less reference-like voices.The paper attributes stronger speaker fidelity to the base T2AV model’s generative audio prior, while noting higher WER inherited from that prior.Source: §3.2 · p. 3

No-regression checks

• Every reference-free objective improves after reference-aware fine-tuning, including FAD, transcription error, perfect-transcription rate, CLAP alignment, and UTMOS naturalness.The comparison uses 400 prompts covering 30 speakers with identical seeds and texts; the zero-initialized Wfilm path stays close to identity without a reference.Source: §3.3 · pp. 3–4, Table 2 · p. 4

Efficient inference

• ∼30× speed-up results from reducing each diffusion step from a 3.18 B-parameter joint pass to an effective ∼0.58 B audio-only pass.The audio path bypasses the video sub-block and video VAE decoder without changing weights, while K6AV LITE preserves reference timbre.Source: §3.5 · p. 4

Core contribution

• A single zero-initialized linear layer, reference-latent prepending, and speaker FiLM convert T2AV diffusion models into voice-cloning systems.The change is trained in a short voice-aware stage.Source: §4 · pp. 4–5

Evaluation

• 674 samples: K6A 5B achieves the highest speaker similarity across three verification networks and significantly outperforms every baseline.The benchmark evaluates voice-cloning performance on speaker-text pairs.Source: §4 · p. 5

Efficient inference

• ∼30× lower inference cost: K6AV LITE preserves the voice-cloning behavior while evaluating the audio path without the video path.The original text-to-audio-video functionality is also retained.Source: §4 · p. 5

Future work

• Natural extensions include transferring prepend+FiLM to asymmetric AV-DiTs with frozen speaker encoders and adding a small text-fidelity loss to address the WER gap.The suggested loss could use CTC against the prompt without disturbing speaker conditioning.Source: §4 · p. 5

A.1. Base architectures

• 32 fused decoder blocks form the shared asymmetric AV-DiT backbone of K5 and K5-LITE.Both models use the architecture described in Section 2.1 and Figure 1.Source: §A.1 · p. 7
• 1792 is the video-stream width, with heads of dimension 64.The video stream is denoted d_v = 1792 in the passage.Source: §A.1 · p. 7
• 896 is the audio-stream width, with heads of dimension 64.The audio stream is denoted d_a = 896 in the passage.Source: §A.1 · p. 7
• 2-block text encoder consumes the prompt and feeds it to both the video and audio streams.The text encoder is shared across both streams.Source: §A.1 · p. 7
• K5 uses the full 32 fused blocks.K5 therefore uses the complete block configuration of the shared topology.Source: §A.1 · p. 7
• K5-LITE uses a reduced configuration with the same block topology but smaller text and output layers, totaling ≈3 B parameters.The passage truncates before specifying how many parameters are attributable to individual components.Source: §A.1 · p. 7

Reference window sampling

• Training samples use the first audio slice as the target, while references come from the remaining clip audio with a 0.5 s buffer.The target is typically 5 seconds and aligned with video latents.Source: §A.2 · p. 7
• Speech references last 1.2–2.5 s from a 12 s search window, whereas non-speech references last 1.5–3.5 s from a 3.5 s search window.Both reference types are sampled from the same clip as the target.Source: §A.2 · p. 7

• AdamW uses β1, β2 = (0.9, 0.95), weight decay 10−3, and a maximum gradient norm of 1.0.Source: §A.3 · p. 7
• 10−5 is the base learning rate, while Wfilm and its biases use a 5× learning-rate multiplier.Source: §A.3 · p. 7
• 8000 warmup steps are followed by a constant learning rate.Source: §A.3 · p. 7
• Reference and text conditioning are each dropped with probability 0.1.Source: §A.3 · p. 7
• 10% of steps clamp video latents to zero and 10% clamp audio latents to zero, preserving joint audio-video behavior during fine-tuning.Source: §A.3 · p. 7
• 50 diffusion steps use split CFG with wt = 5 and wr = 4 under a fixed seed at inference.Source: §A.3 · p. 7

Encoder and conditioning

• The frozen speaker encoder is the publicly released Qwen3-TTS speaker network, operating on 24 kHz audio.It computes the FiLM embedding used for conditioning.Source: §A.4 · p. 7

Training implementation

• Once per training sample, the speaker encoder runs on CPU in a background worker.This separates speaker-embedding computation from the main training step.Source: §A.4 · p. 7
• Only the resulting 1024-dim speaker vector is transmitted to the training step.The encoder itself remains frozen during this process.Source: §A.4 · p. 7

Audio-only inference details

• Audio-only inference disables the video stream without changing weights, short-circuiting video computation in all 32 fused decoder blocks and skipping the video VAE decoder.The audio sub-block, speaker FiLM layer, and audio classifier-free-guidance loop continue operating as during training.Source: B. Audio-only inference details · p. 7

Study design

• 82 runs evaluated inference-time and data-time choices for voice-cloning quality on a held-out benchmark subset.The study examined reference length, guidance weight, diffusion steps, language matching, and reference pre-processing.Source: C. Additional analysis · p. 7

Reference length

• 4 s marks a reference-length plateau: similarity rises from approximately 1 s, while shorter than 2 s sharply reduces Resemblyzer SECS.Beyond 4 s, additional reference audio provides negligible returns; the speaker encoder appears to bottleneck very short references.Source: C. Additional analysis · p. 7

Reference-guidance weight

• 4–6 is the optimal range for the second CFG weight wr; values above 7 slightly over-smooth audio and reduce UTMOS.Below 3, the reference is under-imposed.Source: C. Additional analysis · p. 8

Number of diffusion steps

• 30 diffusion steps essentially saturate mean SECS, while increasing to 60 adds < 0.02 and doubles wall-clock cost.Routine use therefore does not benefit from the additional steps.Source: C. Additional analysis · p. 8

Language matching

• 0.05 lower mean SECS results, on average, when prompt and reference languages differ; matching them is recommended whenever possible.Cross-language generation remains possible despite the reduction.Source: C. Additional analysis · p. 8

Reference pre-processing

• Reference de-noising and VAD help noisy captures but can hurt clean studio references, so preprocessing is safest to skip when UTMOS is approximately ≥3.The recommendation depends on reference quality rather than applying preprocessing universally.Source: C. Additional analysis · p. 8