Source-linked AI summary
Recurrent Neural Network Transducer for Audio-Visual Speech Recognition
Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, Olivier Siohan
TL;DR
The paper addresses robust large-vocabulary speech recognition by combining audio and visual information in an RNN-T system trained on 31k hours of YouTube-derived audio-visual data. Across YouTube and LRS3-TED evaluations, the combined system improves over unimodal baselines, while visual information is especially relevant under degraded audio and the reported LRS3-TED setting.
Problem
ASR systems can fail in noisy or overlapping speech, motivating use of visual signals that provide phonetic and contextual information.
Method
The authors mine 31k hours of matched YouTube audio, captions, and speaking-face tracks and train an RNN-T system on audio, visual, or concatenated synchronized features.
Results
The audio-visual system reaches 20.5% WER on YTDEV18 versus 21.5% for audio-only and 48.5% for visual-only, and the paper reports state-of-the-art results on LRS3-TED.
Takeaways & Limitations
Combining synchronized audio and visual inputs improves the reported YTDEV18 result and enables evaluation of visual contributions under noise and overlapping speech.
Takeaways & Limitations
The strongest visual-only performance discussed for LRS3-TED applies to professionally produced, well-lit video with high frame rates and cooperative speakers facing the camera.
Abstract
from arXiv · showhide
This work presents a large-scale audio-visual speech recognition system based on a recurrent neural network transducer (RNN-T) architecture. To support the development of such a system, we built a large audio-visual (A/V) dataset of segmented utterances extracted from YouTube public videos, leading to 31k hours of audio-visual training content. The performance of an audio-only, visual-only, and audio-visual system are compared on two large-vocabulary test sets: a set of utterance segments from public YouTube videos called YTDEV18 and the publicly available LRS3-TED set. To highlight the contribution of the visual modality, we also evaluated the performance of our system on the YTDEV18 set artificially corrupted with background noise and overlapping speech. To the best of our knowledge, our system significantly improves the state-of-the-art on the LRS3-TED set.
1. INTRODUCTION
The paper develops an audio-visual ASR system using a large mined dataset and an RNN-T architecture, evaluating audio, visual, and combined modalities on YouTube and LRS3-TED benchmarks.
- Visual speech information can help ASR in noisy or overlapping-speech conditions by constraining possible phonemes and supplying contextual cues.The motivation includes spelling support for obscure names and terms and visual-scene information.
- 31k hours of mined audio-visual training data support evaluation of audio-only, visual-only, and audio-visual speech recognition.The corpus is filtered from YouTube videos for matching speech, captions, and on-screen speaking faces.
- The system uses an RNN-T encoder-decoder with character outputs and a joint network, avoiding the conditional-independence limitation of CTC models.The encoder processes audio or visual features, while the decoder models output symbols conditioned on previous symbols.
- Audio-visual ASR is evaluated against audio-only and visual-only systems on YouTube transcription and the LRS3-TED benchmark.The evaluation also includes YTDEV18 utterances corrupted with babble noise or overlapping speech.
2. DATASETS
The paper constructs a large YouTube-derived audio-visual corpus and evaluates it with manually transcribed YouTube utterances plus the LRS3-TED benchmark, including corrupted audio conditions.
- The training set is mined from YouTube using forced alignment, face tracking, and matching between transcripts, audio, and visible speaking faces.The approach extends prior semi-supervised YouTube corpus construction.
- 31k hours of short utterances form the training corpus, with matched user captions, audio, and on-screen speaking-face tracks.The corpus exceeds prior V2P and TM-seq2seq training quantities and covers more varied speaking-face content than professional datasets.
- YTDEV18 contains 20k manually transcribed segmented utterances totaling 25 hours, selected from videos with matching face tracks.The evaluation data begin with 1000 hours of transcribed videos and are processed with the same face-tracking approach as training.
- YTDEV18 is corrupted with babble noise at 0, 10, and 20 dB SNR or equal-energy overlapping speech at utterance boundaries.These conditions are designed to assess the contribution of the visual modality under degraded audio.
3. SYSTEM ARCHITECTURE
The system synchronizes audio and video features by varying the audio extraction rate, then feeds audio, visual, or concatenated features into an RNN-T architecture.
- 3.1. Synchronized Audio-Visual Frames: Audio features use 80 mel filters after a 25 ms Hanning-window STFT, while videos are restricted to 23–30 fps and smoothed tracked-face landmarks.These preprocessing choices address heterogeneous video standards and audio-video rate differences.
- 3.1. Synchronized Audio-Visual Frames: Variable-rate audio feature extraction matches video timing while retaining the 25 ms STFT window, avoiding visual downsampling.The STFT shift is advanced proportionally to the video frame rate.
- 3.2. RNN-T for Audio-Visual Speech Recognition: The RNN-T architecture uses separate encoder and decoder components connected by a joint network that produces character probabilities.Audio-only and visual-only operation is controlled through modality switches.
- 3.2. RNN-T for Audio-Visual Speech Recognition: The full audio-visual model concatenates audio and visual features into a 912-dimensional encoder input.The visual frontend produces a 512-coefficient embedding for each input frame.
4. EXPERIMENTS AND RESULTS
Experiments show that audio-visual RNN-T models improve recognition on clean and noisy speech, while modality dropout improves robustness when inputs are missing. The system also achieves a strong LRS3-TED result, including 4.5% WER with combined audio and visual input.
- 4.1. Baseline Results: 20.5% WER was obtained on YTDEV18 with matched audio-visual training and testing, versus 21.5% for audio-only and 48.5% for video-only systems.These results show that the audio-visual model was the best matched-modality configuration.
- 4.2. Training with Drop-out: Thirty-percent audio dropout improved the overall audio-visual result to 19.8% WER and made visual-only performance 50.3%, near the 48.5% visual-only baseline.The trade-off was a worse audio-only result of 46%.
- 4.3. Variable Versus Fixed Frame Rate: Variable frame rate was as effective as fixed frame rates for the RNN-T model on YTDEV18.The fixed 30fps model was only slightly better under a different learning-rate schedule.
- 4.4. Impact of Visual Features on Noisy Speech: Audio-visual models gained about 5% relative improvement over audio-only models on clean YTDEV18 and a larger advantage on noisier versions.Under severe 0dB babble noise, the visual-only model outperformed both audio-only and audio-visual systems.
- 4.4. Impact of Visual Features on Noisy Speech: With overlapping speech added during training 10% of the time, audio-visual models showed greater relative improvements, especially on 0dB babble and overlapping-speech tests.The training corruption used levels uniformly sampled between 0 and 20dB.
- 4.5. Performance on LRS3-TED Dataset: 4.5% WER was achieved on LRS3-TED with combined audio and visual input, improving over the reported video-only result of 33.6%.The system used models trained only on the 31k-hour YouTube dataset.
5. AUDIO-VISUAL ASR & AI PRINCIPLES
The paper highlights privacy concerns alongside strong visual-only performance, while noting that results depend substantially on recording quality and speaker conditions. It also states that the technology is intended to improve ASR access and support assistive applications, under stated data-governance constraints.
- AI Principles: The authors identify privacy issues as important because visual-only lip-reading performance can be strong under favorable recording conditions.The stated LRS3-TED conditions include no shadows, at least 24 fps, and a cooperative speaker facing the camera.
- AI Principles: 33.6% visual-only WER on LRS3-TED versus 48.5% on YTDEV18 illustrates the gap between studio-quality and in-the-wild recordings.The LRS3-TED task uses professionally produced content with good lighting, high frame rate, and cooperative speakers facing the camera.
- AI Principles: 57.0% visual-only WER on low-quality YTDEV18 utterances versus 37.1% on high-quality utterances shows sensitivity to image quality.Seventy percent of utterances were classified as low quality using face-size and pose criteria.
- AI Principles: The work aims to extend ASR access and support assistive technology for people with impaired speech.The authors specifically mention people with Lou Gehrig’s disease as a potential beneficiary group.
- AI Principles: The data and models are restricted to a small group of project researchers and handled in compliance with the European Union General Data Protection Regulation.
6. CONCLUSIONS
The paper presents an RNN-T audio-visual speech recognition system trained on a large automatically mined YouTube corpus. Audio-visual input improves robustness to babble noise and overlapping speech, while the large training set supports state-of-the-art performance on LRS3-TED.
- 6. CONCLUSIONS: The RNN-T audio-visual system achieves significant performance improvement under babble noise or overlapping speech compared with audio-only recognition.With equal training data, the audio-visual system only slightly improves over audio-only recognition in clean conditions.
- 6. CONCLUSIONS: 31k hours of automatically mined YouTube audio-visual training data support state-of-the-art performance on the publicly available LRS3-TED set.The corpus pairs user-uploaded captions with matching audio and an on-screen speaking face.
- 6. CONCLUSIONS: The system uses synchronized audio and visual features concatenated at the RNN-T encoder input for audio-only, visual-only, and audio-visual models.