PaperlayerView exact revision on arXiv

Source-linked AI summary

Not Quite My Tempo: Voice Activity-aware Speech Synthesis for Lip-Synchronous Dubbing

Alejandro Pérez-González-de-Martos, Florian Lux, Angelina Elizarova, Milana Shkhanukova, Andreas Kellner, Mattia Antonino Di Gangi

arXiv:2609.26486v1eess.AScs.AI

TL;DR

Lip-synchronous dubbing requires precise pause timing across source and target speech, but many approaches condition synthesis on visual lip cues. This work conditions inpainting-based speech synthesis on source-derived voice activity and makes that guidance optional at inference. Alignment accuracy is high, while subjective prosody ratings show no statistically significant degradation and pauses remain linguistically coherent.

  • Problem

    Lip-synchronous dubbing must map pause structures precisely across source and target speech, while prior approaches often use video-derived lip cues.

    Source: §1 · p. 1

  • Method

    The model uses frame-level binary voice-activity masks to condition inpainting-based synthesis, with training-time random masking enabling optional guidance at inference.

    Source: §1 · p. 1, §2.2 · p. 2

  • Results

    Frame-level VAD alignment accuracy rises from roughly 73% without conditioning to about 96% with conditioning for LibriTTS-R; prosody differences are not significant.

    Source: §3.3 · p. 3, §3.4 · p. 3

  • Takeaways & Limitations

    Editors can use or relax VAD guidance at inference, while the model can place pauses at linguistically coherent boundaries without explicit pause constraints.

    Source: §1 · p. 1, §2.2 · p. 2

    The method assumes translations have timing structure sufficiently matched to the source, though it does not require explicit additional prosodic information.

    Source: §1 · p. 2

Abstract

from arXiv · show

Automatic lip-synchronous dubbing requires a speech synthesis model to generate alternating voice and silence patterns in the target language that match the timing of the source clip precisely to ensure an optimal viewing experience. Prior works address this problem by conditioning the speech synthesis process on lip movements extracted from the video signal. In this work, we condition the speech generation on a binary voice-activity signal, which has a lightweight representation and can be produced in multiple ways. We show that the model follows the voice-activity signal with high accuracy while maintaining natural prosody and semantically appropriate pause placement within sentences, as demonstrated through extensive objective and subjective evaluations. By randomly masking this condition during training, we make the feature entirely optional during inference, allowing editors to enforce or relax lip-sync constraints when desired.

1. Introduction

Lip-synchronous dubbing must map pause timing across source and target speech while preserving a coherent cross-lingual experience. The paper uses source-derived voice activity to condition inpainting-based synthesis, while leaving linguistic content within voiced regions to the model.

1. Introduction

  • The method transfers source-language voice-activity timing to target synthesis, while leaving linguistic content within voiced regions to the model.The binary pattern maps across the fixed-length source and target frame sequences.Source: §1 · p. 1
  • The model learns to place pauses at linguistically coherent boundaries despite having no explicit pause-placement constraints beyond the activity mask.The authors point to examples from subjective evaluation and qualitative analysis.Source: §1 · p. 1
  • Unlike visual-cue approaches that condition TTS on lip movements [13], this method uses audio-text data and requires no paired video during training.It is also decoupled from detecting the primary speaking face and modeling lip dynamics.Source: §1 · p. 1
  • The approach assumes the translation has timing structure sufficiently matched to the source, but does not require explicit additional prosodic information.Prior exact-prosody settings assume identical source and target texts, unlike dubbing,,.Source: §1 · pp. 1–2

2. Methods

The methods combine a cross-lingual TTS architecture with frame-level voice-activity conditioning. Randomly masking the conditioning during training supports optional VAD guidance at inference.

2.1. Basic TTS Setup

  • The model follows F5-TTS with average upsampling, explicit speaker embeddings and Global Style Tokens, and a modified pretrained SoundStream vocoder.The architecture uses these choices for temporal alignment bias, speaker/style transfer, and waveform generation, respectively.Source: §2.1 · p. 2

2.2. Voice Activity Conditioning

  • A binary voice-activity indicator for each acoustic frame is embedded and added to encoder representations as temporal cues for the DiT Flow Matching decoder.This is the method’s stated architectural novelty.Source: §2.2 · p. 2
  • Randomly masking ground-truth activity embeddings during training enables selective masking near speech–silence transitions or fully unconditioned inference.The flexibility supports workflows that may not require strict lip-sync constraints; the authors also note alignment may compromise voice quality in some cases.Source: §2.2 · p. 2
  • One large-scale study reports lip-sync alignment in movies only about 12% of the time, while rigid constraints can be difficult with non-adapted translations.The authors also report that enforcing alignment may compromise other aspects of voice quality in some cases.Source: §2.2 · p. 2

3. Experiments

The experiments test voice-activity alignment, pause and prosody naturalness, and synthesis robustness on multilingual dubbing data. VAD conditioning improves temporal alignment, while subjective ratings show no significant naturalness decrease; poorly timed translations can still raise synthesis errors.

3.1. Data

  • The evaluation uses 291 mTEDx samples across four source languages, while the models include a reproducible LibriTTS-R English model and a multilingual model.The test utterances last 7–15 seconds and each contains at least one pause longer than 500 ms.Source: §3.1 · p. 2

3.2. Experimental Setup

  • The system uses frame-level Silero VAD masks and randomly masks voice-activity embeddings during training to allow controllable conditioning at inference.Its setup also uses phoneme inputs, speaker and style embeddings, and a DiT decoder.Source: §3.2 · p. 2

3.3. Alignment Consistency

  • About 73% to 96% alignment accuracy with VAD conditioning, versus without conditioning, for the LibriTTS-R model; the multilingual model shows a similar improvement.The comparison holds across source languages; Table 1 reports per-language Silero VAD alignment accuracy.Source: §3.3 · p. 3, Table 1 · p. 3
  • Conditioned synthesis scores higher across Pyannote, TEN3, and Silero VAD evaluators, and its silence-boundary deviations cluster near zero while unconditioned deviations spread more widely.The cross-evaluator comparison checks that the alignment result is not tied to one VAD system.Source: §3.3 · p. 3, Figure 3 · p. 3, Figure 4 · p. 3

3.4. Prosody Naturalness

  • VAD conditioning causes a marginal, statistically nonsignificant decrease in pause-placement and prosody ratings, with appropriate pause placement maintained.Pairwise Mann-Whitney U tests report p > 0.05 for all pairwise comparisons; Table 2 summarizes the ratings.Source: §3.4 · p. 3, Table 2 · p. 3, Figure 5 · p. 4

3.5. Robustness

  • VAD-conditioned synthesis shows a slight WER increase, attributed primarily to translations poorly adapted to source VAD constraints.Under extreme timing mismatch, the model may alter pacing or introduce phonetic deletions and repetitions; Table 3 reports WER and TTSDS scores.Source: §3.5 · pp. 3–4

4. Qualitative Analysis

The qualitative analysis links low evaluation scores to synthesis artifacts and identifies translation timing and punctuation as triggers. When timing budgets are exceeded, the model preserves VAD alignment at the cost of omissions, repetitions, or text reordering.

4. Qualitative Analysis

  • Low evaluation scores correlate strongly with synthesis artifacts, including phonetic deletions, repetitions, and word reordering.The analysis investigates the worst-scoring samples and identifies distinct triggers for each error type.Source: §4 · p. 4
  • When translations exceed the available temporal budget, the model can omit words to preserve pause structure, even at speaking rates beyond the training distribution.In constructed examples, omissions begin after a critical threshold as the model maintains time-axis alignment.Source: §4 · p. 4
  • Translations with too little text for the target duration can trigger repetitions once the model cannot slow speech further without changing VAD structure.The passage describes the lower speaking-rate bound but does not specify its value.Source: §4 · p. 4
  • Punctuation near a VAD silence can prompt speed changes around the pause, but extreme rate modulation may lead the model to reorder text instead.The constructed examples indicate that the model has learned a correlation between pauses and punctuation marks.Source: §4 · p. 4
  • Manually refining translations to match the source VAD structure mitigates instability; the authors report high precision with isochronous input.Such input may come from automated isochronous MT or human-in-the-loop intervention.Source: §4 · p. 4

5. Conclusion

The conclusion presents VAD-conditioned inpainting TTS for automated lip-synchronous dubbing, reporting accurate temporal following with only minor prosody and robustness degradation linked to poorly adapted translations.

5. Conclusion

  • The proposed VAD-conditioned inpainting TTS enables automated lip-synchronous dubbing and follows the reference audio's temporal structure with high accuracy.Source: §5 · p. 4
  • Only minor prosody and robustness degradation is observed, which the authors link to poorly adapted translations.Source: §5 · p. 4
  • Future work will use specially constructed training data to reduce reliance on accurate isochronous MT or a human-in-the-loop, and add finer-grained lip-dynamic information.The proposed lip-dynamic information includes extracting phonetic content and mapping it to articulatory configurations.Source: §5 · p. 4

Disclosure of Generative AI Tool Use

Loading 2609.26486v1…