PaperlayerView exact revision on arXiv

Source-linked AI summary

X-Dubbing: Speech-to-Speech Translation under Temporal and Visual Constraints

Jeongsoo Choi, Ji-Hoon Kim, Junseok Ahn, Joon Son Chung

arXiv:2610.32042v1cs.CLcs.CVeess.AS

TL;DR

Fixed-video dubbing must produce faithful target speech that fits the source video, while visual constraints and limited paired dubbing data complicate adaptation. X-Dubbing joins timing-aware translation with video-conditioned synthesis and refines translation using synthesized-speech feedback. Evaluations report improvements across translation quality, isochrony, lip-sync, and perceived dubbing quality.

  • Problem

    Fixed-video dubbing must preserve translation meaning while matching source timing and lip movements, but end-to-end speech-to-speech systems do not explicitly model visual constraints and paired dubbing data are scarce.

    Source: §1 · pp. 1–2

  • Method

    X-Dubbing couples a timing-aware Dubbing Translator with a video-aware Voice Actor and uses synthesized-speech feedback to refine scripts, without requiring paired dubbing data.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    X-Dubbing improves translation quality, isochrony, lip-sync, and perceived dubbing quality in extensive evaluations.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The results suggest that downstream speech feedback offers a scalable path toward dubbing-aware translation.

    Source: §6 · p. 8

    Experiments focus on French-to-English monologue videos with one visible speaker, and dubbed-speech quality may depend on source audio and video characteristics.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Cross-lingual dubbing aims to convert source speech into target-language speech that remains synchronized with the original video. The key to this task is to produce a faithful translation that is naturally speakable within the source timing, which is non-trivial because languages differ in duration, rhythm, and articulation. In this paper, we propose X-Dubbing, which formulates the task as a joint problem of time-aware speech-to-text translation and video-aware text-to-speech. Specifically, our method consists of a Dubbing Translator that generates temporally structured scripts and a Voice Actor that renders them in synchrony with the source video, both grounded in a language-agnostic timing budget. We further refine the translator with dubbing-aware feedback from the synthesized speech, encouraging scripts that yield well-synchronized dubbing. This design requires no paired dubbing data, as the two components are trained on speech-text and monolingual audio-visual data respectively. Extensive experiments show that X-Dubbing outperforms recent baselines across diverse metrics, including isochrony and audio-visual correspondence.

1 Introduction

Fixed-video dubbing must preserve meaning while fitting source timing and lip movements, but translation and speech synthesis optimized separately can conflict. X-Dubbing coordinates both through speech-output feedback without paired dubbing data.

1 Introduction

  • Fixed-video dubbing requires translated speech to preserve meaning while fitting source timing and lip movements, which separate translation and synthesis may not coordinate.Time-aware translation alone can miss visual boundaries, while video-aware synthesis with unconstrained scripts can distort speaking rate.Source: §1 · pp. 1–2, Figure 1 · p. 1
  • The framework pairs a Dubbing Translator that uses source timing and explicit pause markers with a Voice Actor guided by those markers and source-video frames.Candidate translations are evaluated using both text-based criteria and synthesized speech.Source: §1 · p. 2
  • X-Dubbing formulates fixed-video dubbing as coordination of time-aware translation and video-aware speech generation, using synthesized-speech feedback without paired cross-lingual dubbing data.The feedback optimizes the translator toward scripts that can be rendered naturally under source-video constraints.Source: §1 · p. 2
  • X-Dubbing achieves strong performance across translation quality, audio-visual synchrony, and perceptual dubbing quality.The framework coordinates a time-aware translator and video-aware speech generator through feedback from synthesized speech.Source: §1 · p. 2

2 Related Work

Prior work addresses timing in translation, visual conditioning in speech synthesis, and speech-to-speech translation, but each leaves challenges for fixed-video cross-lingual dubbing. X-Dubbing couples these aspects using generated-speech feedback.

2.1 Time-Aware Machine Translation

  • Duration-aware and pause-aware translation methods address target duration and segment boundaries, but local alignment also depends on speech-generator timing control.The paper therefore couples time-aware translation with a controllable speech generator conditioned on the source video.Source: §2.1 · p. 2

2.2 Video-Aware Speech Generation

  • Video-aware synthesis can secure visual alignment, but fixed translated scripts may require unnatural speaking rates when their length differs from the source.X-Dubbing uses generated-speech feedback to optimize the translator for the source video’s temporal and visual constraints.Source: §2.2 · p. 2

2.3 Speech-to-Speech Translation

  • Speech-to-speech translation can preserve translation quality and paralinguistic traits, but fixed-video dubbing also needs timing, pause, and articulation alignment with scarce paired training data.X-Dubbing instead trains its translator on speech–text pairs and its speech generator on monolingual audio-visual data.Source: §2.3 · p. 3

3 Method

X-Dubbing coordinates a timing-aware script generator and a video-conditioned voice actor, then uses evaluations of synthesized speech to refine scripts.

3 Method

  • X-Dubbing splits dubbing between a Dubbing Translator that inserts source-structured pause markers and a Voice Actor that renders them using temporal and visual cues.The translator generates a target-language script; the Voice Actor uses source audio as a speaker reference.Source: §3 · p. 3, §3.3 · p. 5

3.1 Preliminary

  • Multimodal speech translation conditions target text directly on acoustic embeddings, while the TTS backbone supports cross-lingual voice transfer and duration control.X-Voice can preserve a source speaker’s timbre across languages, and its output length is set by the initial noise sequence.Source: §3.1 · pp. 3–4

3.2 Dubbing Translator: Time-Aware S2TT

  • The translator uses VAD-derived speech intervals and pauses as a language-agnostic timing budget, avoiding token-level source-target alignment.The budget records when speech and silence occur in the source utterance.Source: §3.2 · p. 4
  • Isochrony-normalized projection estimates target duration from syllable counts and language-level rates, then rescales timing while preserving relative speech-segment and pause positions.The resulting training targets allocate target syllables in proportion to source segment durations and place pause markers near word boundaries.Source: §3.2 · pp. 4–5

3.3 Voice Actor: Video-Aware TTS

  • The Voice Actor uses source audio as a speaker reference and VAD speech-activity frames plus SyncNet visual embeddings as aligned synthesis controls.The temporal and visual cues enter acoustic conditioning and each DiT block, providing early access and persistent frame-level control.Source: §3.3 · p. 5

3.4 Preference Optimization with End-to-End Feedback

  • Preference optimization renders candidate scripts with the frozen Voice Actor, scores timing, translation quality, lip sync, and pause consistency, then uses Pareto-dominant pairs for DPO.Candidates that violate the pause-count constraint are discarded before preference pairs are constructed.Source: §3.4 · p. 6

4 Experiment

The experiments use French-to-English mTEDx data and English LRS3 audio-visual data, assessing dubbing with objective metrics and human ratings. The setup also specifies model adaptation and compares systems addressing different aspects of cross-lingual dubbing.

4.1 Datasets

  • Translator fine-tuning uses 11,092 mTEDx French-English clips, while Voice Actor fine-tuning uses 95,474 English LRS3 audio-visual clips.The datasets contain 14.9 and approximately 200 hours, respectively; translator preference optimization uses 7,752 pairs, and evaluation uses the mTEDx French-to-English test subset.Source: §4.1 · p. 6

4.2.1 Objective Metrics

  • Objective evaluation covers translation quality, speaker similarity, temporal alignment, lip synchronization, and speech quality.Translation uses ASR-BLEU and ASR-chrF++; speaker similarity uses SpkSim, lip synchronization uses LSE-C and LSE-D, and speech quality uses UTMOS.Source: §4.2.1 · pp. 6–7
  • SyllRate ρ measures source–generated syllable-rate correlation, Pause IoU compares pause intervals, and SLC@p counts clips within a specified duration range.SLC@p uses p ∈ {0.2, 0.4} around the source active-speech duration.Source: §4.2.1 · pp. 6–7

4.2.2 Subjective Metrics

  • Human evaluation uses 15 raters and 12 test clips to score Audio-Visual Correspondence and Overall Naturalness on a five-point scale.Raters assess lip movement and timing correspondence, and naturalness in speaking rate, pauses, and rhythm.Source: §4.2.2 · p. 7

4.3 Implementation Details

  • Both models use rank-8 LoRA with α = 32, while the translator audio projection and Voice Actor pause and visual conditioning modules are fully trainable.Speech activity detection uses a 0.5 probability threshold and 100 ms minimum silence duration; LRS3 word-level alignments use UnitY2.Source: §4.3 · p. 7

4.4 Baseline Models

  • The compared baselines address synchronized audio-visual translation, prosody preservation, pause timing, or duration-controlled textless speech translation.They include AV2AV-Speech, SeamlessExpressive, Hibiki, and Dub-S2ST.Source: §4.4 · p. 7

5 Experimental Results

Across fixed-video dubbing experiments, X-Dubbing leads on temporal alignment and lip-sync while also scoring highest in translation quality and speaker similarity, with competitive speech quality. Ablations and human ratings support the contributions of timing and visual conditioning, dubbing-aware translator training, and the system’s perceived quality.

5.1 Main Results

  • X-Dubbing achieves the best temporal alignment and lip-sync scores, the highest translation and speaker similarity scores, and competitive speech quality.These are the reported comparisons with baseline systems in fixed-video dubbing.Source: §5.1 · pp. 7–8

5.2 Comparison with Cascaded Baselines

  • Without inference-time candidate search, X-Dubbing achieves the best translation quality and lip synchrony among the compared cascades.The alternatives use repeated synthesis and scoring; re-ranking and paraphrase-based search improve lip synchrony but require that additional inference-time work.Source: §5.2 · p. 8

5.3.1 Dubbing Translator

  • DPO improves translation quality, speaking-rate consistency, and duration compliance over SFT, while removing INP from SFT weakens pacing control.Together, the results suggest that INP-based supervision and dubbing-aware preference optimization help produce scripts that better fit source timing.Source: §5.3.1 · p. 8, Table 3 · p. 8

5.3.2 Voice Actor

  • Removing frame-level FiLM weakens pause alignment and lip synchrony; removing pause conditioning increases Seg. WER, while removing video conditioning degrades both alignment measures.The results support combining pause and video conditioning through frame-level FiLM to align synthesized speech with source timing and lip movements.Source: §5.3.2 · p. 8, Table 4 · p. 8

5.4 Human Evaluation

  • In human evaluation, X-Dubbing receives the highest ratings for audiovisual correspondence and overall naturalness.The authors report that these ratings confirm that improvements in automatic timing and lip-sync metrics translate to perceived dubbing quality.Source: §5.4 · p. 8, Table 5 · p. 8

6 Conclusion

X-Dubbing coordinates translation and speech synthesis using a language-agnostic timing budget and feedback from synthesized speech. The authors report improvements across translation, synchrony, and perceived dubbing quality without paired dubbing data.

6 Conclusion

  • X-Dubbing treats translation and speech synthesis as a coordinated process rather than isolated stages.A time-aware translator and video-aware speech generator are connected through a language-agnostic timing budget and feedback from synthesized speech.Source: §6 · p. 8
  • The framework generates synchronized dubbed speech without requiring paired dubbing data.The conclusion describes the translator as refined using feedback from synthesized speech.Source: §6 · p. 8
  • Evaluations report improvements in translation quality, isochrony, lip-sync, and perceived dubbing quality.The authors suggest downstream speech feedback offers a scalable path toward dubbing-aware translation.Source: §6 · p. 8

Limitations

The experiments focus on French-to-English monologue videos with one visible speaker; broader language pairs, multi-speaker conversations, and complex visual scenes remain future directions.

Limitations

  • Experiments focus on French-to-English monologue videos with a single visible speaker.The authors identify more diverse language pairs, multi-speaker conversations, and complex visual scenes as future directions.Source: Limitations · p. 9
  • The reported experimental scope does not cover more diverse language pairs or multi-speaker conversations.Both are named as directions for future work.Source: Limitations · p. 9
  • The authors note that dubbed-speech quality may depend on the characteristics of the source audio and video.Source: Limitations · p. 9

Ethical Considerations

The paper notes that reproducing a speaker’s voice in another language may be misused to generate misleading content. It calls for speaker consent and clear disclosure of synthetic speech.

Ethical Considerations

Related papers

Loading 2610.32042v1…