Source-linked AI summary
Hierarchical Codec Diffusion for Video-to-Speech Generation
Jiaxin Ye, Gaoxiang Cong, Chenhui Wang, Xin-Cheng Wen, Zhaoyang Li, Boyuan Cao, Hongming Shan
TL;DR
VTS must generate speech from silent video despite visual-acoustic information asymmetry and the hierarchical structure of speech. HiCoDiT uses hierarchical discrete codec diffusion with disentangled visual conditioning and dual-scale normalization, and experiments report superior fidelity and expressiveness. Its discrete diffusion formulation assumes dimensional independence for efficient computation.
Problem
Existing VTS methods disregard speech hierarchy, limiting direct alignment between visual features and speech properties at matching hierarchical levels.
Method
HiCoDiT uses hierarchical discrete speech-token diffusion, assigning lip motion and identity to low-level tokens, facial emotion to high-level tokens, and dual-scale AdaLN to vocal style and prosody.
Results
HiCoDiT outperforms baselines in fidelity and expressiveness, with experiments reporting superior naturalness, expressiveness, and synchronization fidelity.
Takeaways & Limitations
The results highlight discrete speech-token modeling as a promising approach for video-to-speech generation.
Takeaways & Limitations
Efficient computation relies on the dimensional-independence assumption, applying one-dimensional diffusion independently across sequence dimensions.
Abstract
from arXiv · showhide
Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.
1. Introduction
VTS must synthesize natural, lip-synchronized speech from visual cues despite sparse visual information and cross-modal alignment challenges. HiCoDiT addresses this by aligning hierarchical speech tokens with distinct visual attributes.
- Visual cues share speech content, speaker identity, and emotional prosody, but their sparsity makes accurate visual-acoustic alignment difficult.
- Existing methods align visual information with semantic content, vocal identity, or emotional prosody using auxiliary representations and objectives.Examples include multimodal self-supervised representations, face-speaker encoder alignment, contrastive learning, and facial emotion embeddings matched to pitch and energy.
- HiCoDiT introduces a hierarchical codec diffusion transformer that uses discrete speech-token hierarchy for vision-speech alignment.
- Low-level blocks condition on synchronized lip motion and facial identity, while high-level blocks use facial emotion to generate prosodic details.
- HiCoDiT combines hierarchical diffusion, disentangled visual conditioning, and dual-scale AdaLN to improve semantic consistency, speech diversity, expressiveness, and fidelity.The contribution summary identifies global vocal style and local prosody as targets of dual-scale AdaLN.
2. Related Work
Prior VTS systems align visual and speech representations through auxiliary objectives or generative conditioning, but generally treat speech as a flat sequence. HiCoDiT instead explicitly models speech-token hierarchy in a discrete diffusion framework.
- Existing VTS approaches predict text or mel-spectrograms, condition speaker embeddings on lip motion, or minimize cross-modal embedding distances.
- FTV gradually injects hierarchical visual features into continuous mel-spectrogram generation, while VoiceCraft-Dub adapts discrete text-to-speech models to visual context.
- These approaches obscure speech representation hierarchy, where coarse linguistic content emerges earlier and fine prosodic detail is resolved later.
- HiCoDiT introduces a discrete diffusion model trained from scratch that explicitly integrates speech hierarchy for VTS.
3. Preliminary: Discrete Diffusion Models
Discrete diffusion models represent speech as discrete token sequences and corrupt them through a masked-state Markov process. Training learns reverse transitions that iteratively unmask the sequence during sampling.
- Discrete diffusion models use token sequences whose elements belong to a finite discrete state space, with transitions parameterized by a time-dependent diffusion matrix.
- For efficient computation, dimensional independence applies the same one-dimensional token-level diffusion process to each sequence dimension.
- The token-level diffusion matrix combines a noise schedule with transitions toward a masked state.
- Forward corruption either replaces tokens with [MASK] or leaves them unchanged according to time-dependent probabilities.
- A score network estimates reverse transitions, and denoising score entropy trains it to recover the reverse process.
- Sampling starts from an all-[MASK] sequence and iteratively replaces masked tokens using the learned reverse transition model.
4. Methodology
HiCoDiT formulates video-to-speech generation as hierarchical masked prediction over RVQ speech tokens, assigning visual cues to token levels according to their roles. Its architecture combines disentangled conditioning, dual-scale normalization, and multi-level diffusion training to generate content-, timbre-, and prosody-related speech tokens.
- Disentangled visual conditioning: Lip motion and facial identity condition low-level blocks to refine content- and timbre-centric tokens, while emotional expression conditions high-level blocks for prosody.The resulting score heads predict token scores used to unmask all RVQ levels during reverse diffusion.
- Disentangled visual conditioning: The lip adapter projects AV-HuBERT features into the masked low-level token dimensions, exploiting temporal alignment between lip motion and speech content.The projected features have sequence length L and channel dimension C matching the masked low-level speech tokens.
- Disentangled visual conditioning: The identity adapter projects ArcFace facial features and aligns them with GE2E acoustic identity features to model speech timbre.This cross-modal identity modeling connects facial appearance with speaker identity despite their lack of direct correspondence.
- Hierarchical speech tokenization and diffusion: HiCoDiT tokenizes speech with a 12-level RVQ codec and partitions tokens into low- and high-level components for hierarchical diffusion.Each level uses a codebook size of 1,024, and the masked tokens are generated through a discrete diffusion process.
- Dual-scale adaptive normalization: Dual-scale AdaLN captures vocal style through channel-wise conditioning and prosodic dynamics through temporal-wise conditioning.Emotion features produce temporal-level scales and channel-level scale-and-shift parameters for high-level speech-token conditioning.
- Training and inference: HiCoDiT uses 12 linear score heads and a multi-level DSE objective, with an additional identity loss reinforcing speaker consistency.Training uses the sum of DSE losses across all 12 RVQ levels, while inference uses Euler sampling with 64 steps and visual features only.
5. Experimental Results
HiCoDiT achieves strong objective and subjective performance on in-domain and out-of-domain evaluations, while ablations support its hierarchical modeling and visual-conditioning design.
- Quantitative evaluation: HiCoDiT achieves leading UTMOS, DNSMOS, WER, and LSE-C performance on untrained LRS3 and LRS2 test sets.It reaches second-best MCD, while speaker similarity is lower than FTV unless speech identity guidance is provided.
- Subjective evaluation: HiCoDiT obtains the highest MOSnat of 3.17 and MOSsync of 3.50 in subjective evaluation.Its expressiveness is slightly lower than FTV, suggesting speaker-data diversity affects this dimension.
- Subjective evaluation: HiCoDiT receives 57.0% preference over AlignDiT and 52.1% over FTV in A/B testing.Ground-truth speech is preferred over FTV at 51.5%, while HiCoDiT is preferred over ground truth at 53.9%.
- Qualitative results: HiCoDiT generates clearer mel-spectrograms with richer acoustic detail and precise lip synchronization than competing methods.Lip2Wav and MTL show over-smoothing or artifacts, while other diffusion methods retain noise in silent clips.
- Ablation studies: Removing hierarchical modeling significantly degrades performance across all metrics, supporting the speech hierarchy prior.The ablation collapses multilevel speech representations and injects visual conditioning uniformly across tokens.
- Ablation studies: HiCoDiT maintains robust intelligibility and lip synchronization on 160-utterance CinePile film data spanning 56 speakers.Removing GE2E loss reduces speaker similarity from 56.78% to 34.10%, while replacing Poster2 lowers emotion accuracy from 79.41% to 76.29%.
6. Conclusion
HiCoDiT aligns visual features with hierarchical speech tokens, synchronizing lip motion and identity at lower levels while modeling emotional and prosodic dynamics at higher levels. On LRS2 and LRS3, it outperforms state-of-the-art methods in naturalness, expressiveness, and synchronization fidelity.
- HiCoDiT aligns lip motion and facial identity with lower-level speech tokens and emotional features with higher-level prosodic tokens.This hierarchical alignment supports both speaker-aware content and expressive prosody.
- The dual-scale AdaLN captures global vocal style and local prosody dynamics during speech generation.
- HiCoDiT outperforms state-of-the-art methods on LRS2 and LRS3 in naturalness, expressiveness, and synchronization fidelity.