PaperlayerView exact revision on arXiv

Source-linked AI summary

Watch Your Speech: Text-aware Video-to-Speech Synthesis with Textual Conditioning

Gunwoo Lee, Yoori Oh, Yoseob Han

arXiv:2610.01012v1cs.CVcs.SDeess.AS

TL;DR

Visually similar lip movements can correspond to different utterances, making phonetic accuracy difficult in video-to-speech synthesis. WYS conditions on text from training onward and fuses it with video features for speech generation. On LRS2 and LRS3, it improves synchronization and perceptual quality while maintaining competitive textual accuracy.

  • Problem

    Visually similar lip movements can correspond to different phonemes, making accurate speech content difficult to determine from video alone.

    Source: §1 · p. 2

  • Method

    WYS uses ground-truth text during training and lip-reading-predicted text at inference, fusing text and video features to condition speech generation.

    Source: §1 · p. 2

  • Results

    WYS improves audio-visual synchronization and perceptual quality on LRS2 and LRS3 while maintaining competitive textual accuracy.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The reported results support text-aware video-to-speech generation that balances synchronization and perceptual quality with competitive textual accuracy.

    Source: §1 · p. 2, §6 · p. 14

    At inference, WYS relies on the accuracy of the lip-reading model used to predict text.

    Source: §6 · p. 14

Abstract

from arXiv · show

Video-to-speech synthesis aims to generate natural-sounding speech from silent talking-face videos while ensuring phonetic accuracy. A fundamental challenge in this task is the inherent one-to-many mapping problem, where visual dynamics often lack sufficient information to uniquely determine the corresponding utterance. To address this, we propose Watch Your Speech (WYS), a video-to-speech synthesis framework that incorporates textual conditioning as an explicit linguistic cue to mitigate visual ambiguity. Our framework features an attention-based embedding fusion module that synergistically integrates textual context with video sequences, coupled with a conditional flow matching objective for high-fidelity speech generation. Extensive experiments on the LRS2 and LRS3 datasets demonstrate that WYS achieves superior performance, establishing new state-of-the-art results in audio-visual synchronization (LSE-C/D) while maintaining highly competitive textual accuracy (WER). Subjective evaluations further confirm that our model generates speech with near-human naturalness, validating the effectiveness of textual conditioning in content-controlled video-to-speech synthesis. Project page: https://github.com/gunwoo5034/Watch-your-Speech

1 Introduction

Video-to-speech must recover intelligible, natural speech from silent talking-face video, but visually similar lip shapes can correspond to different phonemes. WYS adds text conditioning during training and fuses textual and visual features; evaluations report improved synchronization and perceptual quality with competitive textual accuracy.

1 Introduction

  • Visually similar lip shapes can correspond to distinct phonemes, making speech content difficult to recover accurately from video alone.Bilabial phonemes such as /b/ and /m/ can be visually indistinguishable.Source: §1 · p. 2
  • Earlier text-aware approaches either underuse textual information or add it only during inference, limiting learned audio-text alignment and potentially affecting naturalness.LipVoicer uses an external lip-reading classifier during inference, while text is absent during its training.Source: §1 · p. 2
  • WYS uses ground-truth text during training and lip-reading predictions at inference, with attention-based fusion and conditional flow matching for speech generation.The framework directly conditions generation on text rather than relying on external classifier guidance.Source: §1 · pp. 2–3
  • On LRS2 and LRS3, WYS improves audio-visual synchronization and perceptual quality while maintaining competitive textual accuracy.Subjective evaluations also indicate improved perceived naturalness compared with baseline methods.Source: §1 · pp. 2–3

2 Related Work

Video-to-speech research has progressed from visual-to-acoustic mappings through adversarial and diffusion-based generation, but precise textual content remains challenging. WYS proposes training-time fusion of text and video features without an auxiliary classifier.

2 Related Work

  • Early systems mapped visual features to acoustic representations, and generative adversarial networks were later introduced to improve audio realism.Examples include Vid2Speech and LTBS, followed by VCA-GAN.Source: §2 · p. 3
  • Diffusion-based methods improve audio fidelity, but methods without explicit text conditioning can leave one-to-many content ambiguity unresolved.LipVoicer adds lip-reading classifier guidance at inference, but does not train with text and may compromise naturalness.Source: §2 · p. 3
  • WYS proposes attention-based fusion of text and video from training onward, using classifier-free guidance without an auxiliary classifier.The approach is intended to constrain the visual-to-acoustic mapping while improving intelligibility, naturalness, and synchronization.Source: §2 · p. 3

3 Methodology

WYS combines textual, visual, and speaker-identity features to condition flow-based speech synthesis. Its front end uses ground-truth text during training and lip-reading estimates at inference.

3.1 Linguistic Information via Lip-Reading

  • During training, WYS pairs ground-truth text with video clips; at inference, a pre-trained lip-reading model estimates the text supplied to the text encoder.The resulting text embeddings provide linguistic conditioning in both phases.Source: §3.1 · p. 4

3.2 Multimodal Representation via Feature Extraction

  • The framework derives text, speaker-identity, and lip-motion embeddings from the video and its associated text using dedicated encoders.The text encoder uses frozen BERT with LoRA adaptation; speaker identity comes from a reference frame, and lip dynamics are encoded by a 3D CNN, ShuffleNet, and TCN.Source: §3.2 · pp. 4–6

3.3 Cross-modal Alignment via Attention-based Embedding Fusion

  • A dual-stage attention module enriches video and text features within each modality, then exchanges information across modalities to form a synchronized representation.The speaker embedding is broadcast across time and concatenated with that representation to produce the final fused embedding.Source: §3.3 · pp. 6–7

3.4 Speech Synthesis via Conditional Flow Matching with Classifier-Free Guidance

  • Conditional flow matching uses the fused embedding to generate a mel-spectrogram, with classifier-free guidance trained on both conditioned and null-token inputs.At inference, guided velocity predictions drive ODE synthesis, and a pre-trained HiFi-GAN converts the generated mel-spectrogram into a waveform.Source: §3.4 · p. 7

Experimental Setup

The experiments use LRS2 and LRS3, with videos processed at 25 fps and audio at 16 kHz. Evaluation combines objective measures of content, synchronization, and audio quality with human listening ratings.

4.1 Datasets

  • LRS2 contains 144K clips and 224 hours; LRS3 contains 151K clips and 475 hours, with greater speaker, accent, and recording-condition variability.Both datasets are processed at 25 fps with audio sampled at 16 kHz.Source: §4.1 · p. 8

4.2 Implementation Details

  • Training uses Adam for 1,000,000 iterations with batch size 64; inference uses a 100-step first-order Euler ODE solver and guidance scale 2.0.The model was trained on a single NVIDIA RTX PRO 6000 GPU.Source: §4.2 · p. 8

4.3 Evaluation Metrics

  • WER measures transcript accuracy, LSE-C/D measure lip-speech alignment, and STOI-Net and DNSMOS assess audio clarity and human-likeness.Human raters scored 10 samples per dataset across WYS and four baselines on five criteria using a 5-point MOS scale.Source: §4.3 · p. 8

Experimental Results

WYS is compared with a range of video-to-speech baselines, then evaluated on benchmark performance, component ablations, and additional tests of text quality and generalization. Results show a trade-off in WER versus other metrics, alongside gains in subjective ratings and robustness across several tested conditions.

5.1 Main Results

  • On LRS2/LRS3, LipVoicer has lower WER (18.8991% vs. 20.1321%; 22.2423% vs. 23.7349%), while WYS leads other metrics; human ratings favor WYS on every criterion.The quantitative comparison covers synchronization and perceptual-quality metrics; subjective ratings are reported on both datasets.Source: §5.1 · p. 9, Table 1 · p. 10, Table 2 · p. 10

5.2 Ablation Studies

  • Video-query cross-attention reaches 21.0629% WER versus 80.8254% for text-query attention; the final esyn fusion reaches 20.1321% WER.The final fusion combines refined self-attention features with dual cross-attention.Source: Table 3 · p. 11, §5.2 · p. 11
  • Removing the speaker identity input drops spkSIM by 10.0839%, while core metrics fluctuate only marginally.The study interprets identity injection and cross-modal fusion as serving distinct roles.Source: Table 4 · p. 11, §5.2 · pp. 11–12
  • With guidance scale ω = 2.0, WYS peaks across evaluation dimensions; disabling conditioning yields WER above 100%, and ω = 3.0 slightly declines.The authors use ω = 2.0 as the default setting.Source: Table 5 · p. 12, §5.2 · p. 12
  • T = 10 preserves robust performance at 0.7 sec per sample, while T = 1000 brings negligible gains and 84.2 sec latency.The authors select T = 100 for the most refined balance across metrics.Source: Table 6 · p. 12, §5.2 · p. 12

5.3 Additional Experiments

  • GT text at inference lowers WER from 20.1321% to 9.9784%; training with GT text also halves WER versus predicted text (20.1 vs. 40.9).The authors report that noisy predicted text mainly degrades linguistic supervision, and use GT text in training and predicted text only at inference.Source: §5.3 · pp. 12–13
  • Lipreading-plus-TTS trails WYS in synchronization and content accuracy despite competitive audio quality; on LRS3-to-LRS2 transfer, WYS leads four of five metrics.For cross-dataset evaluation, WYS ranks second on STOI-Net.Source: §5.3 · p. 13, Table 7 · p. 13
  • With moderately corrupted text, WYS retains a substantial WER reduction versus text-free models and maintains superior synchronization and perceptual quality across text conditions.The evaluation stratifies LRS2 samples by lip-reading WER into accurate, moderately corrupted, and severely corrupted groups.Source: §5.3 · pp. 13–14, Table 8 · p. 14

6 Conclusion

WYS uses explicit textual conditioning from training onward, with attention-based fusion and conditional flow matching to address ambiguity in video-to-speech. Experiments on LRS2 and LRS3 report improved synchronization and perceived naturalness while maintaining competitive textual accuracy; inference remains reliant on lip-reading accuracy.

6 Conclusion

  • WYS addresses the one-to-many mapping problem with textual conditioning from training onward, attention-based embedding fusion, and conditional flow matching.The framework integrates linguistic constraints for high-fidelity speech generation.Source: §6 · p. 14
  • On LRS2 and LRS3, WYS improves audio-visual synchronization while maintaining highly competitive WER, and subjective evaluations indicate improved perceived naturalness.Source: §6 · p. 14
  • Inference relies on lip-reading accuracy; future work will explore noisy-text-robust training and advances in lip-reading models.Source: §6 · p. 14

Supplementary Material

The supplementary algorithms describe training with aligned GT text and inference with lip-reading-predicted text, followed by iterative mel-spectrogram generation and vocoding. Supplementary configuration details compare model training and vocoder choices.

A Algorithm

  • Training extracts lip-region video, a reference face frame, and aligned GT text; inference substitutes lip-reading-predicted text and generates a mel spectrogram through iterative denoising before vocoding.The inference process initializes a latent variable from Gaussian noise and estimates a velocity field using classifier-free guidance.Source: A Algorithm · p. 18, B Configuration Details · p. 19

B Configuration Details

  • Most compared models use a fixed learning rate and Adam or AdamW; V2SFlow uses eight GPUs, LipVoicer four, and other methods one.VCA-GAN uses Griffin-Lim, SVTS uses WaveGAN, and several other systems, including WYS, use HiFi-GAN.Source: B Configuration Details · p. 18

C Data preprocessing

WYS prepares separate lip-motion, speaker-identity, and text inputs from the source video, aligning training text to video timestamps while using the full sentence at inference.

Lip-Region Video

  • 68 facial landmarks guide grayscale 96×96 lip crops, which form the temporal lip-region video tensor.The tensor contains one channel and one element per frame.Source: C Data preprocessing · p. 19

Speaker Identity Frame

  • A randomly sampled video frame is resized to 224×224 and kept in RGB format for speaker identity.The identity frame has three channels.Source: C Data preprocessing · p. 20

Temporal Synchronization

  • Training text is selected by word-level timestamps with a small margin, whereas inference uses the full sentence.The margin helps ensure that words are fully captured in the aligned video segment.Source: C Data preprocessing · p. 20

D Compare Speaker Similarity

WYS is evaluated on speaker identity using Speaker Similarity and scores 0.6823, below only IntelligibleL2S among the baselines.

Compare Speaker Similarity

  • WYS scores 0.6823 on Speaker Similarity, outperforming every baseline except IntelligibleL2S.The authors characterize this as competitive performance, not the highest score.Source: D Compare Speaker Similarity · p. 20
  • Speaker identity is assessed with the Speaker Similarity (spkSIM) metric.The reported comparison is on LRS2.Source: D Compare Speaker Similarity · p. 20
  • The authors interpret the result as maintaining reasonable speaker identity while focusing primarily on natural speech generation from visual input.Source: D Compare Speaker Similarity · p. 20

E Lip-Reading Backbone Selection

The LRS2 and LRS3 WER comparison favors Auto-AVSR over AV-Hubert, so the authors use Auto-AVSR as their lip-reading backbone.

Lip-Reading Backbone Selection

F Robustness to Imperfect Text

Across robustness and supplementary comparisons, WYS balances linguistic accuracy and perceptual quality, while attention configurations, text inputs, guidance, and sampling steps affect generated spectrograms. Inference remains limited by errors in lip-reading pseudo-text.

F Robustness to Imperfect Text

  • Across LRS2 text-corruption levels, WYS maintains low LSE-D and balanced linguistic accuracy and perceptual quality, while WER rises sharply and STOI-Net and DNSMOS change more gradually.Text-free models achieve slightly higher STOI-Net scores in some conditions, while WYS remains competitive overall.Source: F Robustness to Imperfect Text · p. 21, Table S4 · p. 22

G.1 Comparison with Other Models

  • WYS spectrograms most closely match ground truth across LRS2 and LRS3 samples, while other models show blur or fine-detail differences.The passage also reports the most stable ASR performance across samples.Source: §G.1 · p. 22

G.2 Failure Case

  • Lip-reading pseudo-text errors can yield suboptimal reconstructions and impose an upper bound on inference performance.Source: §G.2 · p. 22

G.3 Effect of Attention Mechanism

  • The proposed attention configuration produces the closest ground-truth spectrogram match, particularly in low-frequency regions; other configurations can fail to reflect text information accurately.The passage identifies the relevant outputs as self-attention and cross-attention features.Source: §G.3 · pp. 22–23

G.4 Effect of Lip-Reading Model

  • Providing ground-truth text instead of lip-reading pseudo-text yields higher text accuracy and a spectrogram more similar to ground truth.Source: §G.4 · p. 23

G.5 Effect of Guidance Scale Factor

  • Guidance produces stable, coherent spectrograms resembling ground truth, unlike the distorted and unstable output without guidance.Source: §G.5 · p. 23

G.6 Effect of Sampling Steps

  • With as few as 10 sampling steps, WYS produces perceptually meaningful spectrograms whose overall structure remains largely unchanged across step counts.The original-paper baseline uses 100 steps.Source: §G.6 · p. 23

H Crowdsource Evaluation

Crowdsourced MOS assessments compared WYS, four other systems, and ground truth across five speech criteria. MTurk participants evaluated samples through 20 HITs split between LRS2 and LRS3.

H Crowdsource Evaluation

  • The MTurk evaluation used 20 HITs—10 per dataset—with five video samples per HIT, 10 unique participants per HIT, and a $0.50 reward per HIT.Each task unit was called a Human Intelligence Task (HIT).Source: H Crowdsource Evaluation · p. 23
  • MOS assessments compared IntelligibleL2S, DiffV2S, LipVoicer, V2SFlow-V, WYS, and ground truth across five criteria using randomized samples.The criteria were quality, alignment, intelligibility, synchronization, and naturalness.Source: H Crowdsource Evaluation · p. 23, Figure S9 · p. 28

Related papers

Loading 2610.01012v1…