PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Audiovisual Speech Recognition through Synthetic Visual Data Augmentation

Pol Buitrago, Pol Gàlvez, Javier Hernando

arXiv:2609.31961v1eess.AScs.CLcs.SDeess.IV

TL;DR

AVSR development is constrained by scarce labeled audiovisual data, and whether synthetic visual data benefits AVSR remains an open question. This work pairs synthetic visuals with real speech audio and evaluates the approach for Spanish augmentation and Catalan training without real AV data. The results report relative WER reductions of up to 16.2% for Spanish augmentation and suggest synthetic data alone can provide cross-modal information for zero-AV-resource Catalan training.

  • Problem

    AVSR fine-tuning still requires scarce labeled audiovisual data, and whether synthetic visual data benefits AVSR remains an open question.

    Source: §1 · p. 1

  • Method

    The work generates synthetic visual data from speech recordings and face images to pair with real audio for AVSR training.

    Source: §1 · p. 1

  • Results

    Up to 16.2% relative WER reductions were reported when synthetic visual data augmented Spanish AVSR, while the Catalan case study suggests synthetic data alone provides sufficient cross-modal information for zero-AV-resource training.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The methodology offers a scalable strategy for training AVSR systems in low-resource scenarios with less dependence on language-specific audiovisual datasets.

    Source: §5 · p. 11

    The annotation tool was developed primarily for Catalan benchmarking; extending it as a general-purpose AV annotation tool for low-resource languages lies beyond this work.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

Audiovisual Speech Recognition (AVSR) is a multimodal approach to speech recognition that incorporates visual information from lip movements to enhance model performance. Despite its advantages, its development remains constrained by the limited availability of labeled audiovisual (AV) datasets. This work explores the use of synthetic visual data as a solution, using an audio-driven talking-head pipeline to generate lip-synchronized visual content from existing audio data. We evaluate the effectiveness of synthetic visual data both as an augmentation strategy and as a standalone training resource, applying our approach to Spanish and Catalan. Our results show that augmenting real AV data with synthetic samples yields relative Word Error Rate (WER) reductions of up to 16.2%, demonstrating the potential of this approach. Moreover, we demonstrate that synthetic data alone can serve as a baseline for AVSR training in languages lacking AV datasets. These findings provide evidence that synthetic visual data can serve as a scalable solution to AVSR data scarcity, enabling broader language coverage.

1. Introduction

AVSR can improve transcription by combining visual speech cues with audio, but scarce labeled audiovisual data limits its development across languages. This work investigates synthetic visual data for Spanish augmentation and Catalan training without real AV data.

  • Labeled audiovisual data remains scarce, and AVSR fine-tuning still depends on it despite self-supervised pretraining.This shortage limits development across languages and motivates approaches that reduce reliance on real AV annotations.Source: §1 · p. 1
  • The proposed approach pairs real speech audio with synthetic visual data generated from speech recordings and face images to expand AV training resources.It aims to reduce dependence on manually annotated video corpora, particularly for under-resourced languages.Source: §1 · p. 1
  • Recent generative models can produce realistic, temporally coherent visual speech with lip movements that preserve articulatory structure.Synthetic visual data has shown potential in visual speech tasks such as lipreading, but its effectiveness for AVSR is the open question.Source: §1 · p. 1
  • Whether synthetic visual data benefits AVSR remains open because domain shift or audiovisual misalignment can disrupt joint audio-visual learning.Prior work has explored synthetic visuals mainly for unimodal lipreading, so those benefits may not transfer directly to AVSR.Source: §1 · pp. 1–2
  • The study tests synthetic visuals as an augmentation strategy for low-resource Spanish AVSR and as a standalone training resource for Catalan without real AV data.Source: §1 · p. 2

2. Methodology

The methodology builds synthetic lip-synchronized audiovisual samples, fine-tunes a pretrained AVSR model on configured datasets, and adds a semi-automatic pipeline to annotate a Catalan test set.

2. Methodology

  • The methodology combines speech-driven visual synthesis, fine-tuning on real, synthetic, and mixed datasets, and experiments evaluating synthetic data's impact on AVSR performance.Source: §2 · p. 2

2.1. Speech-Driven Visual Synthesis

  • AVSynthGen filters face images for visible mouths, animates the lip region from real speech audio, and pairs generated videos with original-corpus transcriptions.It uses a talking-head model to produce synchronized visual speech while preserving the rest of each face.Source: §2.1 · pp. 2–3

2.2. Semi-automatic Annotation Pipeline

  • The Catalan test-set pipeline segments and filters raw clips, pseudo-labels and normalizes them, then manually reviews and corrects labels with Label-Inspector.The tool's broader use as a general-purpose annotation tool for low-resource languages is beyond this work's scope.Source: §2.2 · p. 4, Figure 5 · p. 4

2.3. AVSR Model and Training

  • Experiments fine-tune a pretrained AV-HuBERT checkpoint on English LRS3 and VoxCeleb2 data, using a Transformer decoder to produce SentencePiece subword units.The selected checkpoint was pretrained on unlabeled audiovisual data; the decoder is randomly initialized.Source: §2.3 · p. 4

3. Experimental Setup

The experimental setup compares synthetic visual data in Spanish, where real audiovisual data is available, and Catalan, where training uses synthetic audiovisual data alone. It specifies the datasets, training configurations, and modality-specific evaluations used to assess these settings.

3.1.1. Synthetic Data Generation

  • Synthetic AV datasets pair selected speech recordings with face images; Spanish uses Common Voice, while Catalan uses TV3Parla and ParlamentParla, excluding audio test splits from synthesis.Source: §3.1.1 · p. 5

3.1.2. Real Audiovisual Data

  • Spanish tests synthetic data as an augmentation to real AV resources, while Catalan tests training with synthetic AV data alone.Source: §3.1.2 · p. 5
  • The Spanish real-AV training corpora are LIP-RTVE, CMU-MOSEASes, and MuAViCes, with test subsets reserved for evaluation.Source: §3.1.2 · p. 5
  • AV-CAT is a manually curated Catalan evaluation benchmark comprising 51 minutes and 38 seconds of footage, created because no existing Catalan AV corpus was available.Source: §3.1.2 · p. 5
  • Synthetic datasets are longer than real AV datasets; the study prioritizes their scalability as an alternative when real AV resources are limited or unavailable, not a size-matched comparison.Source: §3.1.2 · p. 5

3.2. Experiments

  • Experiments report WER and select models by validation WER, across Spanish low-resource and Catalan zero-AV-resource settings.Source: §3.2 · pp. 5–6

3.2.1. Training configurations

  • Spanish compares real-only, synthetic-only, and mixed visual training data; each configuration trains AV, audio-only, and video-only variants.Source: §3.2.1 · p. 6

3.2.2. Experiments in Spanish

  • Spanish experiments first test video-only variants for useful articulatory information, then evaluate AV models against benchmarks and prior AVSR baselines, with modality, attribution, and noise analyses.Source: §3.2.2 · p. 6

4. Results

Synthetic visual data improves Spanish AVSR when added to real training data, and synthetic-only training produces a functional Catalan AVSR model. The results also examine modality contributions, visual attribution, and robustness under noise.

4.2. AVSR Performance with Synthetic Video

  • 12.9%, 16.2%, and 5.4% relative WER improvements on LIP-RTVE, CMU-MOSEASes, and MuAViCes, respectively, accompany MixedVisuales’ lower WER across all benchmarks.The gains are smaller on MuAViCes, where visual cues are less informative; Figure 7 illustrates the dataset-dependent improvements.Source: Table 4 · p. 7, §4.2 · pp. 7–8, Figure 7 · p. 8

4.1. VSR Performance with Synthetic Video

  • 24.7% and 26.0% relative WER reductions on LIP-RTVE and CMU-MOSEASes, and 3.9% on MuAViCes, show that MixedVisuales improves video-only recognition on two benchmarks substantially and one modestly.Real video is higher fidelity but limited in scale, while synthetic video adds scale and diversity despite lower intelligibility.Source: §4.1 · p. 7, Table 3 · p. 7

4.3. Comparison of AVSR with Unimodal Baselines

  • 1.3%–5.8% relative WER reductions versus audio-only show that the audiovisual MixedVisuales variant performs best across all three Spanish benchmarks.The video-only variant performs substantially worse, especially on MuAViCes.Source: Table 6 · p. 8, §4.3 · p. 8

4.4. Input Attribution Analysis in AVSR

  • 1.87% higher mean video attribution for RealVisuales than SynthVisuales accompanies largely overlapping attribution distributions, indicating comparable reliance on visual information.The analysis summarizes each video input’s spatio-temporal Integrated Gradients map as a mean attribution score.Source: §4.4 · p. 9, Figure 8 · p. 9

4.5. Noise Robustness in AVSR

  • Across babble and technical or environmental noise, the audiovisual model generally outperforms audio-only, with gains and the point where video-only overtakes AV varying by noise type and SNR.Under babble, AV gains are approximately 5%–10% from 0 to -20 dB, while under technical and environmental noise gains remain above 4% in most conditions.Source: §4.5 · pp. 9–10, Figure 9 · p. 10

4.6. Zero-AV-Resource Adaptability

  • 15.2% and 81.3% relative WER reductions versus audio-only and video-only, respectively, show that synthetic-only Catalan AVSR outperforms both unimodal variants.The model is evaluated on the manually labeled real-speech AV-CAT benchmark and has slightly higher overall WER than Spanish MixedVisuales.Source: §4.6 · p. 10, Table 7 · p. 10

5. Conclusions

The conclusions present synthetic visual data as an augmentation strategy for Spanish AVSR and a potential training resource for Catalan AVSR without real labeled AV training data.

5. Conclusions

  • Generating lip-synchronized visual speech from existing audio supports both augmentation of limited real AV datasets and AVSR training where real labeled audiovisual data are unavailable.The proposed methodology is presented as a scalable strategy that reduces dependence on language-specific audiovisual datasets.Source: §5 · p. 11
  • Up to 16.2% relative WER reductions across Spanish benchmarks show that synthetic visual data improves AVSR when augmenting real audiovisual data while retaining noisy-condition robustness.The Catalan case study also suggests synthetic visual data alone can provide sufficient cross-modal information for AVSR training in a zero-AV-resource setting.Source: §5 · p. 11

Funding

The paper reports that it received no specific grant from funding agencies.

Funding

  • No specific grant funding was received from public, commercial, or not-for-profit funding agencies.Source: Funding · p. 11

CRediT authorship contribution statement

The contribution statement assigns conceptualization and methodology to Pol Buitrago and Javier Hernando, with distinct roles in data curation, software, analysis, and writing.

CRediT authorship contribution statement

  • Pol Buitrago led methodology, software, investigation, formal analysis, and the original draft; Pol Gálvez handled data curation and review; Javier Hernando contributed conceptualization, supervision, and review.Source: CRediT authorship contribution statement · p. 11

Data availability

The implementation and experiment-reproduction scripts are publicly available, but the synthetic audiovisual datasets and manually annotated Catalan benchmark are not. The datasets used can be obtained from their original sources.

Data availability

  • The study datasets can be obtained from their original sources, and the implementation and experiment-reproduction scripts are publicly available.Source: Data availability · p. 11
  • The synthetic audiovisual datasets and manually annotated Catalan benchmark are not publicly available because third-party audiovisual content cannot be redistributed.Source: Data availability · p. 11
  • Additional information may be available from the corresponding author upon request, subject to the original sources’ licensing restrictions.Source: Data availability · p. 11

Related papers

Loading 2609.31961v1…