Source-linked AI summary
SyncVoice: Simple and Effective Automatic Video Dubbing with Vision-Augmented TTS
Kaidi Wang, Yi He, Wenhao Guan, Weijie Wu, Peijie Chen, Hongwu Ding, Xiong Zhang, Di Wu, Meng Meng, Jian Luan, Lin Li, Qingyang Hong
TL;DR
Automatic video dubbing must produce natural speech synchronized with video, yet precise synchronization and multilingual settings remain challenges. SyncVoice adds lightweight visual-linguistic fusion to pretrained TTS, achieving state-of-the-art zero-shot dubbing on LRS3 and demonstrating effectiveness in English and Chinese.
Problem
Existing approaches still face challenges in precise audio-visual synchronization, and most systems are restricted to single-language settings.
Source: §1 · p. 1Method
SyncVoice adds a lightweight Text-Visual Fusion Module to pretrained TTS, aligning textual and visual representations over time for synchronized speech synthesis.
Source: §1 · p. 1Results
Takeaways & Limitations
The authors conclude that SyncVoice improves lip synchronization and speech quality compared with baseline approaches in English and Chinese.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Automatic video dubbing aims to generate high-fidelity speech that is temporally aligned with visual content. However, existing methods still suffer from limited speech naturalness, insufficient audio-visual synchronization, and poor scalability beyond monolingual settings. To address these challenges, we propose SyncVoice, a simple and effective dubbing framework that lightly integrates a Text-Visual Fusion Module into a pretrained text-to-speech (TTS) system. This module aligns visual features with linguistic representations, enabling temporally synchronized speech synthesis without complex architectural redesign. Experiments on the LRS3 dataset show that SyncVoice achieves state-of-the-art performance in zero-shot dubbing. Further training on a large-scale bilingual audio-visual dataset improves vocal fidelity while preserving synchronization, yielding a single unified model for both Chinese and English dubbing.
1. Introduction
Automatic video dubbing must preserve speaker timbre and naturalness while synchronizing speech with visual cues. SyncVoice adds lightweight visual-linguistic fusion to pretrained TTS, targeting synchronization and multilingual limitations in prior approaches.
1. Introduction
- Prior approaches improve naturalness, but precise audio-visual synchronization remains challenging and most systems are single-language.Earlier models also rely on relatively small audio-visual datasets, limiting generalization.Source: §1 · p. 1
- SyncVoice builds on pretrained ZipVoice and uses a Text-Visual Fusion Module to align textual and visual representations over time.The design aims to preserve pretrained speech quality while improving synchronization with lip movements.Source: §1 · p. 1
- The framework injects visual cues at the linguistic level of a pretrained TTS model without complex fusion modules or auxiliary alignment objectives.It is described as a simple, non-autoregressive, flow-matching framework with lightweight multimodal conditioning.Source: §1 · p. 1
- SyncVoice achieves state-of-the-art zero-shot dubbing performance on LRS3, while large-scale Chinese–English training demonstrates strong generalization.The authors report a unified model for Chinese and English dubbing.Source: §1 · p. 1
2. Methodology
SyncVoice combines a non-autoregressive TTS backbone with a lightweight module that fuses aligned visual and linguistic features. Masked Flow Matching trains speech generation, while multi-condition guidance controls facial, lip, and text inputs.
2.1. Model Architecture
- SyncVoice pairs a non-autoregressive TTS backbone with a Text-Visual Fusion module.The architecture comprises these two components.Source: §2.1 · p. 1
2.1.1. TTS Backbone
- Average Upsampling repeats each text token to the acoustic frame rate, establishing coarse alignment and an interface for temporal visual cues.The repetition rate follows the ratio of target acoustic length to text sequence length; the passage also notes faster alignment learning and improved training convergence.Source: §2.1.1 · p. 2
2.1.2. Text-Visual Fusion Module
- Visual features are aligned to acoustic and upsampled-text frames, adapted, concatenated with text, and projected into the generation condition.The fusion builds on the frame-level alignment provided by Average Upsampling.Source: §2.1.2 · p. 2
2.2. Training Method
- Masked Flow Matching trains speech-spectrogram generation from fused text-visual features, observed speech context, and interpolated noisy speech.A binary mask marks spectrogram elements to generate; visual features are supplied only for masked regions, allowing inference with reference audio without reference video.Source: §2.2 · p. 2
2.3. Multi-Condition Classifier-Free Guidance
- At inference, multi-condition guidance combines predictions with different modality conditions, while separate scales control facial, lip, and text contributions.The LRS3 results table caption specifies st = 0.8, sf = 0.0, and sl = 0.5 for its reported setting.Source: §2.3 · p. 3, Table 1 · p. 3
3. Experimental Setup
The experiments use public English and internal bilingual zero-shot dubbing datasets, train pretrained ZipVoice-based models, and evaluate speech quality, speaker similarity, intelligibility, distortion, and synchronization.
3.1. Dataset
- The LRS3 variant trains on 439 hours of English speech and uses an EN-EN zero-shot test set, while the internal bilingual variant trains on approximately 600 hours of Chinese and 1,190 hours of English video data.The internal variant also has a ZH-ZH zero-shot test set constructed from 826 CN-CVS video pairs; for each sample, reference speech is randomly drawn from another utterance of the same speaker.Source: §3.1 · p. 3
3.2. Implementation Details
- Models start from pretrained ZipVoice weights and train for 55k LRS3 steps or 190k bilingual-data steps.Training uses Scaled Adam at 5 × 10−4; clips are resampled to 25 FPS, and inference uses a 16-step Euler ODE solver.Source: §3.2 · p. 3
3.3. Evaluation Metrics
- Evaluation combines five-point MOS-N and MOS-S ratings with objective SIM-o, UTMOS, WER, MCD, and LSE-C/LSE-D metrics.Twenty participants rate 30 utterances; the objective measures cover speaker similarity, perceived naturalness, intelligibility, spectral distortion, and audio-visual synchronization.Source: §3.3 · p. 3
4. Results
SyncVoice performs strongly against public LRS3 baselines and across English and Chinese bilingual tests. The bilingual ablation reveals a trade-off between synchronization and speech quality as visual-conditioning strength changes.
4.1. Comparative Study on Public Baselines
- Against VoiceCraft-Dub and AlignDiT on LRS3, SyncVoice leads most metrics; AlignDiT’s marginally lower WER accompanies weaker speaker similarity, while SyncVoice improves synchronization and perceptual fidelity.The comparison reports stronger SyncVoice results in SIM-o, UTMOS, MOS-N, MOS-S, LSE-C/LSE-D, and MCD.Source: §4.1 · pp. 3–4
4.2. In-depth Exploration with Bilingual Data
- On EN-EN, lip conditioning strengthens synchronization and surpasses Ground Truth on synchronization metrics; combined face and lip cues achieve the best LSE-D and MCD while largely preserving speech quality.Random initialization consistently underperforms pretrained initialization.Source: §4.2 · p. 4, Table 2 · p. 4
- On ZH-ZH, visual conditioning progressively improves synchronization but remains below Ground Truth, while slightly reducing some speech-quality metrics; the authors cite Chinese articulation as a possible factor.The authors suggest that more Chinese audiovisual data may improve cross-lingual synchronization.Source: §4.2 · p. 4, Table 3 · p. 4
- Across bilingual evaluation, large-scale training improves speech quality and generalization, while pretrained initialization is critical for competitive performance in both languages.The Chinese random-initialized model also performs worse across all metrics.Source: §4.2 · p. 4
4.3. Ablation Study
- In the EN-EN ablation, increasing text guidance improves WER at the cost of synchronization, while stronger lip guidance improves alignment but can slightly worsen WER.Facial features act as a weaker synchronization cue and minimally disrupt speech quality.Source: §4.3 · p. 4, Table 4 · p. 4
5. Conclusion
The conclusion presents SyncVoice as improving lip synchronization and speech quality over baseline approaches, with effectiveness explored in English and Chinese.
5. Conclusion
- SyncVoice improves lip synchronization and speech quality compared with baseline approaches, and the paper reports effectiveness in both English and Chinese.Future work targets more languages and improved cross-lingual adaptation.Source: §5 · p. 4