Source-linked AI summary
JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion
Anthony Chen, Naomi Ken Korem, Gal Zeevi, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or
TL;DR
Video dubbing requires translating speech while preserving identity, lip synchronization, and scene consistency, yet existing pipelines are complex and brittle in real-world conditions. JUST-DUB-IT adapts a joint audio-video diffusion foundation model with lightweight LoRA fine-tuning and synthetic multilingual pairs. The resulting system produces high-quality, robust dubbed videos with improved visual fidelity and lip synchronization compared with existing pipelines.
Problem
Video dubbing must change spoken language while preserving speaker identity, precise lip synchronization, and visual consistency, but modular pipelines struggle with complex real-world conditions.
Method
The method adapts a foundational audio-video diffusion model with lightweight LoRA fine-tuning and trains it on synthetic multilingual pairs created through language-switching and audio-face inpainting.
Results
The approach produces high-quality dubbed videos with preserved facial and voice identity, accurate lip synchronization, and improved robustness and visual fidelity over existing dubbing pipelines.
Takeaways & Limitations
Joint audio-visual generation lets translated speech, facial motion, scene dynamics, and non-speech events co-evolve under a unified generative prior.
Takeaways & Limitations
The approach does not perfectly preserve speaker voice identity in all cases, indicating a need for stronger disentanglement or explicit identity supervision.
Abstract
from arXiv · showhide
Audio-Visual Foundation Models, which are pretrained to jointly generate sound and visual content, have recently shown an unprecedented ability to model multi-modal generation and editing, opening new opportunities for downstream tasks. Among these tasks, video dubbing could greatly benefit from such priors, yet most existing solutions still rely on complex, task-specific pipelines that struggle in real-world settings. In this work, we introduce a single-model approach that adapts a foundational audio-video diffusion model for video-to-video dubbing via a lightweight LoRA. The LoRA enables the model to condition on an input audio-video while jointly generating translated audio and synchronized facial motion. To train this LoRA, we leverage the generative model itself to synthesize paired multilingual videos of the same speaker. Specifically, we generate multilingual videos with language switches within a single clip, and then inpaint the face and audio in each half to match the language of the other half. By leveraging the rich generative prior of the audio-visual model, our approach preserves speaker identity and lip synchronization while remaining robust to complex motion and real-world dynamics. We demonstrate that our approach produces high-quality dubbed videos with improved visual fidelity, lip synchronization, and robustness compared to existing dubbing pipelines.
1 Introduction
Video dubbing must translate speech while preserving speaker identity, precise lip synchronization, and visual consistency in complex real-world scenes. JUST-DUB-IT addresses brittle modular pipelines with a joint audio-video diffusion model trained using synthetic multilingual paired videos.
- Video dubbing translates spoken content while preserving facial appearance, voice identity, accurate lip synchronization, and the surrounding video.Small errors in lip motion, voice characteristics, or identity cues are especially perceptible in videos with complex motion, pose changes, and varying visual conditions.
- Existing dubbing systems often use complex specialized stages whose assumptions break under speaker motion, non-speech sounds, mouth occlusions, and changing scene conditions.Separating and remixing audio can also lose synchronization when translated speech differs in duration from the original.
- The proposed model frames dubbing as joint audio-video generation, allowing audio and visual cues to inform each other through cross-modality attention.This unified formulation jointly models speech, facial motion, scene dynamics, and environmental sounds while requiring only lightweight LoRA fine-tuning.
- Training data are synthesized by generating multilingual clips with one speaker, then inpainting each half’s face and audio to match the other language while preserving semantic content.The resulting aligned bilingual pairs provide supervision without requiring naturally occurring paired videos.
- The method produces high-quality dubbed videos with preserved facial and voice identity, accurate lip synchronization, and improved robustness, visual fidelity, and perceptual quality over existing pipelines.Evaluation covers diverse language pairs, unconstrained real-world videos, complex motion, non-frontal views, partial occlusions, and expressive facial behavior.
2 Related Work
Related work spans unified audio-visual foundation models, audio-driven talking-face generation, and zero-shot voice cloning. These approaches provide complementary generative, facial-animation, and speech-synthesis capabilities, but voice-cloning methods remain unimodal.
- Audio-Visual Generative Models: Audio-visual generation is shifting from cascaded pipelines toward unified foundation models that jointly synthesize high-fidelity sound and visual content.Diffusion Transformers and architectures such as Ovi and UniAVGen target modality alignment through joint or cross-modal designs.
- Audio-driven Talking Face Generation: Audio-driven talking-face methods evolved from graphics-based concatenation to neural systems such as Wav2Lip and modern inpainting-based approaches.The cited literature includes synchronization discriminators and methods that reconstruct masked lip regions.
- Zero-shot Voice Cloning: Zero-shot voice cloning moved from data-intensive systems to neural codec language models and large-language-model adaptations.Methods including IndexTTS2 and SSPO address precise duration alignment but remain unimodal and omit visual cues needed for accurate dubbing.
3 Method
The method adapts a pretrained audio-visual diffusion model with a lightweight in-context LoRA for joint video dubbing. It synthesizes identity-consistent multilingual training pairs and uses masked, modality-aware generation to preserve synchronized audio, facial motion, and scene context.
- 3 Method: The approach adapts a pretrained audio-visual diffusion model for dubbing without masks, explicit face tracking, or modular pipelines.A lightweight in-context LoRA enables the constrained editing behavior while retaining the foundation model.
- 3 Method: Synthetic “perfect pairs” use multilingual clips with one speaker, preserving identity and visual context while changing spoken language.The data-generation pipeline creates language-switching videos and counterfactual inpainted pairs by masking face and audio regions.
- 3 Method: The identity–pronunciation trade-off is addressed by conditioning audio generation on a reference clip that combines speaker identity with target-language phonetic style.Denoising from scratch causes voice drift, while source-audio conditioning causes prosody leakage across languages.
- 3 Method: Lip augmentation prompts exaggerated character-level articulation to increase visual phonetic diversity and reduce indistinct or mumbling lip movements.Two inpainting passes produce translated audio and visually diverse lip motion before merging them into training context.
- 3 Method: Modality-specific cross-attention masking prevents noisy audio and video from receiving conflicting clean-context guidance or leaking across modalities.The training framework concatenates clean context pairs with noised target pairs while fine-tuning only LoRA adapters on a frozen audio-visual transformer.
- 3 Method: Joint generation lets visual dynamics, acoustic events, paralinguistic cues, and dialogue pacing co-evolve within the scene.This design preserves interactions between the speaker and environment rather than treating translated speech as a detached stream.
4 Experiments
The experiments evaluate unified audiovisual dubbing across standard and challenging datasets using video, audio, synchronization, and user-study measures. Results indicate strong robustness to unconstrained conditions, with improved audiovisual coherence, temporal alignment, and user preference over baselines.
- Evaluation Setup: The framework is assessed on generation success, identity preservation, visual fidelity, temporal coherence, mouth articulation, audio quality, and audiovisual synchronization.
- Evaluation Setup: The evaluation covers standard benchmarks and a challenging set of 25 real and 25 synthetic videos with profile views, pose shifts, occlusions, and stylized appearances.
- Quantitative Results: 100% generation success is achieved on both standard and challenging benchmarks, compared with 80% for LatentSync and 74% for MuseTalk on complex samples.
- Quantitative Results: The model achieves the lowest FVD across datasets and higher MAR, indicating stronger temporal coherence, fewer motion artifacts, and more expressive mouth articulation than modular baselines.
- Quantitative Results: The model achieves the lowest duration error and highest intensity correlation, adapting speech duration to video context while preserving scene interactions and non-dialogue audio cues.
- Qualitative Results: Audiovisual offset remains approximately 1–2 frames, while qualitative results show preserved timing, pauses, laughter, and visually related audio events across challenging scenes.
- User Study: Participants prefer the method over LatentSync and HeyGen for Lip Synchronization, Prompt Adherence, and Overall Quality.
- Ablation Study: Ablations show that training and lip augmentation are jointly necessary to avoid source-video reconstruction and produce prompt-aligned, linguistically accurate lip motion.
5 Conclusions
The paper reframes video dubbing as constrained joint audio–visual generation, regenerating speech, facial motion, and scene dynamics under one generative prior. It reports improved robustness in challenging real-world settings, while acknowledging imperfect voice-identity preservation.
- 5 Conclusions: Joint audio–visual diffusion regenerates translated speech, facial motion, and scene dynamics together while preserving temporal structure and identity cues.The formulation also models non-speech events within the unified generative process.
- 5 Conclusions: The method does not perfectly preserve speaker voice identity in all cases, motivating stronger disentanglement or more explicit identity supervision.This is identified as a limitation of the current approach.
- 5 Conclusions: Future work targets longer temporal contexts and richer conversational settings.The paper also positions joint audio–visual priors as promising for broader multimodal editing tasks.
A.1 Training Data
The training-data procedure uses generated multilingual, language-switching videos and targeted audio–facial inpainting to create paired examples. Lip motion is quantified with landmark distance and mouth-aspect-ratio measures for filtering and diversity assessment.
- A.1 Training Data: Gemini generates 100 structured multilingual prompts spanning seven languages, with detailed visual attributes, temporal segments, and explicit language switches.The languages are English, Spanish, Russian, French, German, Italian, and Greek.
- A.1 Training Data: Generated language-switching videos use a rainy-alley example with timed speech, a silent pause, and non-speech actions such as lighting a cigarette.The prompt specifies language and temporal segments within one clip.
- A.1 Training Data: The two video halves are processed independently by detecting the lip region, encoding masked and empty videos, and thresholding their latent difference at τ=0.1.The resulting mask selectively noises the lip region while fully noising the audio before joint denoising.
- A.1 Training Data: A second inpainting pass uses nonsensical speech prompts, then merges first-pass audio with second-pass video to augment lip motion.Attention isolation limits cross-modal interference despite temporal asynchrony between the merged streams.
- A.1 Training Data: Lip Landmark Distance measures structural deviation between reference and generated lip landmarks, while MAR diversity measures the standard deviation of mouth aspect ratio over time.The landmark set contains 20 lip landmarks, and MAR diversity summarizes sequence-level variation.
- A.1 Training Data: Quality–Diversity is defined after introducing LMD and MAR-based lip-motion measurements.The supplied passages identify the QD score as a named evaluation quantity but do not provide a readable prose interpretation of its equation.
A.2 Benchmark
The evaluation uses standard and challenging benchmark tiers to assess video-to-video dubbing across multilingual pairs and unconstrained visual conditions. The challenging tier emphasizes robustness to pose, lighting, occlusion, and stylization.
- A.2 Benchmark: The benchmark has two tiers: 150 standard resampled pairs and 50 high-complexity samples for robustness testing.The standard tier combines TalkVid and HDTF, while the challenging tier combines edited movie clips and in-the-wild synthetic scenes.
- A.2 Benchmark: The standard benchmark contains 100 multilingual TalkVid pairs and 50 English-to-English HDTF pairs across English, German, Spanish, French, and Russian.These datasets provide high-fidelity facial animations and natural speaking styles.
- A.2 Benchmark: The challenging benchmark includes 25 edited movie clips and 25 synthetic scenes with low light, profile views, occlusions, and non-human stylized characters.Its samples are designed to stress-test robustness under complex visual conditions.
A.3 Training
Training adapts LTX-2 with a masked audio-video IC-LoRA using separate video and audio learning rates. Evaluation supplements benchmark testing with a 25-participant preference study over easy and hard samples.
- A.3 Training: The masked audio-video IC-LoRA uses rank-128 modules across attention and feed-forward layers in both video and audio branches.Training applies 2 × 10^-4 to video modules and 1×10^-5 to audio modules to limit audio-domain overfitting.
- A.3 Training: The training uses masked loss with a 10:1 foreground-to-background weighting ratio.The supplied passage specifies foreground mask_loss_weight=1.0 and indicates corresponding background weighting, but truncates the remaining implementation details.
- A.3 Training: The user study compares the model with state-of-the-art baselines on 18 videos, including 6 easy and 12 hard cinematic samples.The hard set contains complex motion, multiple speakers, and varied lighting.
A.5 Evaluation Metrics Implementation
The evaluation reports generation success, identity preservation, visual fidelity, temporal coherence, mouth-motion diversity, and audiovisual synchronization using defined metrics. It also examines latent leakage and shows that lip augmentation improves reconstructed mouth-movement diversity.
- Metric definitions and evaluation: 100% Succ is achieved by the proposed method, while competing methods fail when they cannot detect a face in the input.Succ is the percentage of inputs producing an output video.
- Metric definitions and evaluation: CSIM measures identity preservation through cosine similarity between generated-video and reference face-identity embeddings.FID measures visual fidelity, while FVD measures spatiotemporal consistency using video-feature distributions.
- Metric definitions and evaluation: MAR Div. captures mouth-motion diversity as the temporal standard deviation of normalized Mouth Aspect Ratio.The measure is analogous to Eye Aspect Ratio.
- Metric definitions and evaluation: ASync uses SyncNet to estimate a global synchronization offset by averaging pairwise audio-visual embedding similarity across time.The metric is later shown to be vulnerable to profile-view artifacts in unconstrained scenarios.
- Lip augmentation findings: Lip augmentation produces substantially richer reconstructed mouth-motion variation, improving expressiveness and temporal realism over inpainting without augmentation.Without augmentation, reconstructed lip shapes tend to collapse toward similar shapes across frames.
B.3 ASync metric problem
The ASync metric can overfit to frontal-video behavior and reward visually incorrect profile reconstructions. In contrast, the proposed method matches ground-truth synchronization in the reported profile-view example.
- ASync metric problem: In profile views, LatentSync produces a misleadingly superior ASync score of 0.00 despite visible artifacts caused by a frontal-facing reconstruction bias.The baseline distorts profile lips to minimize temporal offset.
- ASync metric problem: Latent-Aware Fine Masking eliminates leakage-induced mouth and lower-face artifacts while regenerating coherent, audio-aligned facial motion.Without fine masking, latent information leakage produces pronounced green-light artifacts around the masked region.
- ASync metric problem: Lip augmentation increases data-generation diversity by producing richer variation in reconstructed mouth movements.The figure compares original frames with inpainting outputs with and without augmentation.
- ASync metric problem: ASync metrics overfit to frontal videos, limiting their reliability for unconstrained profile-view evaluation.The figure specifically flags this metric behavior.
- ASync metric problem: The proposed method achieves an ASync score of 3.00, identical to ground truth, indicating physically consistent video results in the reported case.The comparison contrasts metric-driven distortion with the proposed result.