Source-linked AI summary
Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling
Zhen Ye, Xu Tan, Aoxiong Yin, Hongzhan Lin, Guangyan Zhang, Peiwen Sun, Yiming Li, Chi-Min Chan, Wei Ye, Shikun Zhang, Wei Xue
TL;DR
Existing joint audio-video models couple modalities throughout denoising, although talking-head audio and facial motion share high-level structure but require distinct low-level rendering. Talker-T2AV separates these roles with a shared autoregressive backbone and modality-specific diffusion heads, and outperforms dual-branch baselines while supporting efficient, flexible generation. Its evidence is limited to talking-head synthesis, and long-sequence quality and video fidelity remain constrained by the continuous latent backbone and video motion autoencoder.
Problem
Existing models use pervasive cross-modal attention throughout denoising, entangling high-level semantic coordination with distinct low-level audio and visual rendering processes for talking heads.
Method
Talker-T2AV jointly models aligned audio-video patch tokens in a shared autoregressive backbone, then uses independent diffusion transformer heads for modality-specific refinement.
Results
Talker-T2AV outperforms dual-branch diffusion baselines on joint generation and matches or surpasses dedicated systems on audio-driven synthesis and video dubbing.
Takeaways & Limitations
For talking-head generation with frame-aligned audio and motion latents, factorized high-level coordination and modality-specific rendering support joint quality, cross-modal consistency, and efficient generation.
Takeaways & Limitations
Long utterances can degrade because continuous-latent prediction errors accumulate, while video fidelity is bounded by the LIA-X motion autoencoder; the claims do not extend to general scene-level audio-video generation.
Abstract
from arXiv · showhide
Joint audio-video generation models have shown that unified generation yields stronger cross-modal coherence than cascaded approaches. However, existing models couple modalities throughout denoising via pervasive attention, treating high-level semantics and low-level details in a fully entangled manner. This is suboptimal for talking head synthesis: while audio and facial motion are semantically correlated, their low-level realizations (acoustic signals and visual textures) follow distinct rendering processes. Enforcing joint modeling across all levels causes unnecessary entanglement and reduces efficiency. We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone, while low-level refinement uses modality-specific decoders. A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space. Two lightweight diffusion transformer heads decode the hidden states into frame-level audio and video latents. Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality, achieving stronger cross-modal consistency than cascaded pipelines.
1. Introduction
Talker-T2AV addresses the inefficiency and over-entanglement of dual-branch diffusion by separating shared high-level audio-video modeling from modality-specific rendering. Its autoregressive, patch-based factorization supports flexible generation and substantially faster inference while retaining unified cross-modal reasoning.
- Existing dual-branch diffusion couples audio and video through pervasive cross-modal attention, entangling semantic modeling with low-level rendering.
- Talker-T2AV uses a shared autoregressive backbone for high-level cross-modal modeling and two independent diffusion transformer heads for low-level rendering.
- Element-wise fusion enables one model to perform joint audio-video generation, audio-to-video synthesis, and video-to-audio without architectural changes or additional fine-tuning.
- A five-second clip requires about 31 autoregressive steps because temporally aligned 1-D latents are patched, while each diffusion head denoises one four-frame patch.
- 24 FPS with 1B parameters versus ≤0.41 FPS for 7–32B dual-branch models makes Talker-T2AV the fastest compared joint text-to-audio-video system.
2. Related Work
Related work has moved from cascaded audio-then-video pipelines toward jointly modeled audio-video generation, including talking-portrait systems. However, existing joint frameworks remain limited by pervasive coupling, non-causal fixed-sequence denoising, and heavy inference costs.
- Joint Audio-Video Generation: Dual-branch diffusion transformers use parallel audio and video DiT towers coupled through bidirectional cross-modal attention.
- Joint Audio-Video Generation: General-purpose joint systems consistently report stronger cross-modal coherence than cascaded pipelines.
- Talking Portraits: Talking-portrait methods model fine-grained correspondence, face-aware regions, or text-driven speech and facial motion within joint diffusion frameworks.
- Joint Audio-Video Generation: Existing joint frameworks entangle high-level semantics with low-level rendering, preclude streaming and variable-length generation, and are computationally heavy at inference.
- Audio-Driven Talking Heads: Audio-driven talking-head work evolved from lip-sync discrimination and 3D motion coefficients toward higher-fidelity diffusion, flow-matching, and explicit motion-latent methods.
3. Method
Talker-T2AV separates high-level cross-modal planning from low-level modality-specific rendering in an autoregressive diffusion architecture. It aligns audio and video temporally, compresses frames into patch tokens, and uses shared hidden states to condition independent diffusion decoders.
- Cross-Modal Modeling: Talker-T2AV jointly models text, speech, and video in a causal patch-level sequence, with audio and video embeddings summed at each position.The text sequence is placed first as a prefix, followed by aligned joint audio-video patch tokens.
- Hierarchical Generation: Each autoregressive step follows a coarse-to-fine factorization: a shared language model produces hidden states, then separate diffusion heads render audio and video patches.The shared hidden state captures cross-modal coherence while modality-specific decoders handle low-level rendering.
- Temporal Alignment: Audio and video are represented as temporally aligned one-dimensional latent sequences at 25 Hz, allowing corresponding speech and motion frames to match by construction.Frozen encoders produce one temporal vector per frame for both modalities, avoiding learned temporal alignment modules.
- Flexible Conditioning: The element-wise fusion design supports video dubbing and audio-driven talking-head synthesis without architectural modification or additional fine-tuning.At inference, an available modality is combined with the autoregressively predicted embeddings of the other modality, and only the corresponding diffusion head is activated.
- Patch Encoding: Grouping every P consecutive frames into one patch token reduces autoregressive sequence length by a factor of P and lowers attention-related computation.Dedicated patch transformer encoders compress each modality’s P frame-level latents into a single representation before element-wise fusion.
- Modality-Specific Refinement: Two independent diffusion transformer heads decode the shared hidden state into modality-specific continuous latent patches using semantic, identity, temporal-context, and noisy-target conditioning.Each head uses bidirectional attention and predicts the velocity field for flow-matching denoising.
4. Experiments
Experiments evaluate Talker-T2AV across joint text-to-audio-video generation, inference efficiency, audio-driven talking heads, video dubbing, and token-arrangement ablations. Results show strong quality, synchronization, speed, and task flexibility, with alignment-sensitive token organization.
- Evaluation setup: The evaluation covers audio quality, video quality, and audio-visual synchronization on Chinese and English test sets, using speech, video, and lip-sync metrics.The Chinese and English test sets each contain 200 videos with paired transcripts, audio, and video.
- Joint audio-video generation: Talker-T2AV achieves the lowest Chinese CER and English WER, the best FVD on both test sets, and the best or second-best FID among joint-generation methods.The authors attribute audio gains to independent modality-specific heads and variable-length autoregressive generation, which avoids fixed-length mismatches.
- Inference efficiency: The causal backbone enables streaming: the first audio-video chunk is emitted after one autoregressive step, unlike non-causal dual-DiT models that wait for the whole clip.The same efficiency advantage extends to audio-driven generation, where the model runs faster than real time and remains competitive with dedicated real-time renderers.
- Audio-driven talking heads: In audio-driven talking-head generation, Talker-T2AV ranks first or second on video fidelity and lip-sync metrics, with the strongest Sync-C scores on both test sets.No single baseline outperforms it comprehensively, and the authors report transfer of cross-modal correspondence from joint training.
- Video dubbing: On the Chem dubbing benchmark, Talker-T2AV ranks first on EMO-SIM, WER, and UTMOS, and second on DD with a negligible gap to InstructDub.The reported gains cover emotional similarity, intelligibility, naturalness, and temporal alignment without task-specific adaptation.
- Ablation studies: Token arrangement is task-dependent: aligned additive tokens work best for joint generation, whereas delaying video improves audio-driven synchronization when conditional audio is available.Audio-first interleaving is comparable but slower and prevents reverse-direction dubbing; delaying video in T2AV substantially degrades metrics.
5. Limitations
The authors identify error accumulation in continuous latent autoregression and limited video-autoencoder capacity as key limitations. They also expect further gains from scaling the paired training data.
- Continuous-latent autoregression allows prediction errors to accumulate over long sequences, causing gradual quality degradation in long utterances.The limitation arises because the backbone predicts in continuous rather than discrete token space.
- Video fidelity is bounded by the capacity of the LIA-X video motion autoencoder.The authors suggest more expressive visual representations as a future route to improve output quality.
- The authors expect continued scaling beyond the 1M paired audio-video training examples to produce further gains.
6. Conclusion
Talker-T2AV uses a shared autoregressive backbone for cross-modal modeling and independent diffusion heads for modality-specific rendering. Across three tasks, it performs strongly against relevant baselines while using substantially less computation and fewer parameters, though general scene-level transfer remains unresolved.
- Talker-T2AV decouples high-level joint audio-video modeling from low-level rendering in two independent modality-specific diffusion heads.
- A single model supports text-to-audio-video, audio-driven talking head synthesis, and video dubbing without architectural modification or additional fine-tuning.
- The factorized design outperforms dual-branch diffusion transformer baselines on joint generation and matches or surpasses dedicated systems on conditional generation tasks.
- Talker-T2AV runs two to three orders of magnitude faster than those baselines with an order of magnitude fewer parameters.
- Whether the same factorization extends to general scene-level audio-video generation remains future work.
Impact Statement
The authors state that the work aims to advance machine learning and identify no societal consequences requiring specific emphasis.
- The paper presents its goal as advancing the field of machine learning.
- The authors identify no potential societal consequences that they believe require specific highlighting.
A. LIA-X Video Motion Autoencoder
LIA-X provides a compact, frame-aligned motion representation for portrait video, which the framework normalizes, predicts, and decodes into rendered frames. The authors selected it after comparing alternative representations with different dimensionality and generalization properties.
- LIA-X is a self-supervised portrait autoencoder that supplies compact, purely temporal video motion latents for frame-level alignment with audio.It extends LIA with a sparse motion dictionary that decomposes facial dynamics into interpretable, disentangled factors.
- The frozen LIA-X encoder represents each video frame at 25 Hz as a 40-dimensional motion vector.
- Per-dimension normalization gives the motion latents zero mean and unit variance before autoregressive and diffusion processing.
- At inference, predicted normalized latents are de-normalized and passed with a source identity image to the LIA-X decoder for portrait-frame rendering.
- FLOAT's 20-dimensional representation generalized poorly to non-frontal poses, while LivePortrait's 265-dimensional representation was harder to predict reliably.
- Table 6 compares audio codec reconstruction across discrete token-based codecs and continuous representations, with Repr. reporting frame rate × latent dimensionality or VQ codebooks.
B. WhisperX-VAE Audio Autoencoder
WhisperX-VAE is a continuous 25 Hz audio autoencoder designed to provide compact latents that preserve both acoustic quality and linguistic semantics for aligned audio-video generation. It combines acoustic and Whisper-derived features before a VAE bottleneck, then reconstructs waveform and semantic features during training.
- Design motivation: WhisperX-VAE uses a continuous audio representation at 25 Hz to temporally align audio latents with the video stream.The framework operates on continuous latents and requires a compact representation suitable for autoregressive prediction.
- Semantic-acoustic fusion: WhisperX-VAE incorporates frozen Whisper Large-v3 features with acoustic encoder features through temporal alignment and element-wise addition.Whisper features are average-pooled from 50 Hz to 25 Hz before fusion with the acoustic stream.
- Latent representation: The fused 1280-dimensional features are projected into a 32-dimensional continuous latent space through a VAE bottleneck.During training, latents are sampled using the reparameterization trick; inference uses the posterior mean.
- Reconstruction: The decoder reconstructs 24 kHz waveforms and Whisper features, encouraging the latent representation to retain both acoustic signals and linguistic content.A semantic reconstruction head is attached to the acoustic decoder output.
- Training: The autoencoder is trained end-to-end with reconstruction, adversarial, feature-matching, KL-divergence, and semantic reconstruction losses.Training uses HiFi-GAN multiperiod and multi-resolution spectral discriminators.
- Reconstruction quality: 3.94 UTMOS is achieved by WhisperX-VAE among compressed representations while using one 32-dimensional continuous vector per frame at 25 Hz.Vocos-Mel remains an upper-bound reference with the best STOI and PESQ because its high-rate mel spectrogram retains nearly lossless spectral information.