Source-linked AI summary
ASR is all you need: cross-modal distillation for lip reading
Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman
TL;DR
Visual speech recognition can be trained without human transcriptions by distilling knowledge from a pretrained ASR model into a VSR model. The combined distillation and CTC approach uses unlabelled audio-visual data, achieves strong public-data results on LRS2 and LRS3, and supports scalable pretraining, though CTC alignment issues and limited scale remain open boundaries.
Problem
Public VSR datasets are much smaller than audio-only ASR corpora, while training typically depends on aligned human transcriptions and forced alignment.
Method
The method distills a pretrained ASR teacher into a VSR student by combining CTC on ASR-generated transcripts with frame-wise posterior distillation.
Results
The method achieves performance that exceeds existing lip reading systems except those trained with proprietary data, including 58.2% WER on LRS2 and 65.6% on LRS3 without ground-truth transcriptions.
Takeaways & Limitations
Unlabelled audio-visual data can support effective VSR pretraining without manually annotated transcriptions, with optional fine-tuning on a small amount of annotation.
Takeaways & Limitations
The study leaves cross-modal CTC distillation alignment problems for future work and uses only VoxCeleb2 with a relatively small network because of resource constraints.
Abstract
from arXiv · showhide
The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data. We achieve this by distilling from an Automatic Speech Recognition (ASR) model that has been trained on a large-scale audio-only corpus. We use a cross-modal distillation method that combines Connectionist Temporal Classification (CTC) with a frame-wise cross-entropy loss. Our contributions are fourfold: (i) we show that ground truth transcriptions are not necessary to train a lip reading system; (ii) we show how arbitrary amounts of unlabelled video data can be leveraged to improve performance; (iii) we demonstrate that distillation significantly speeds up training; and, (iv) we obtain state-of-the-art results on the challenging LRS2 and LRS3 datasets for training only on publicly available data.
1 Visual Geometry Group, Department of Engineering Science, University of Oxford 2 Naver Corporation
The paper concerns lip reading and cross-modal distillation.
- Lip reading is a central topic of the paper.
- Cross-modal distillation is the paper’s other named focus.
- The index terms pair lip reading with cross-modal distillation.
1. INTRODUCTION
The introduction motivates cross-modal distillation as a way to train visual speech recognition using abundant unlabelled audio-visual data. The approach combines distillation with CTC to avoid transcription and alignment requirements while accelerating training.
- Smaller public VSR datasets motivate exploiting larger audio-visual resources that lack text annotations.
- The proposed teacher-student approach distills an ASR model into a VSR model using unlabelled talking-head videos.
- Distillation removes the need for professionally transcribed subtitles and forced alignment when creating VSR training data.
- The method combines a distillation loss with conventional CTC rather than relying only on ASR transcriptions.
- The introduction reports that distillation significantly accelerates training compared with using CTC alone on ASR transcriptions.
2. DATASETS
The paper uses public audio-visual datasets with different annotation and domain properties, filtering VoxCeleb2 to obtain cleaner training utterances. LRS2 and LRS3 provide transcriptions but are relatively small.
- LRS2 and LRS3 are public transcribed audio-visual datasets, but they are relatively small and come from different domains.
- VoxCeleb2 is audio-visual and similar in scale to Librispeech but lacks transcriptions, motivating its use for distillation.
- The study focuses on publicly available datasets, pretraining with distillation on VoxCeleb2 before fine-tuning and evaluation.
- Filtering VoxCeleb2 uses ASR-derived language and quality proxies, followed by transcript comparison with a second ASR model.
- The filtering procedure retains approximately 140k clean VoxCeleb2 utterances from 1M total samples.
3. CROSS-MODAL DISTILLATION
The method adapts an ASR architecture for lip reading and trains it with CTC plus frame-wise knowledge distillation from teacher posteriors. CTC handles unaligned sequence targets, while distillation supplies frame-level supervision.
- 3. CROSS-MODAL DISTILLATION: The student adapts the Jasper ASR architecture for lip reading by replacing its first strided convolution with a transposed convolution and using visual features as input.The visual features come from a spatio-temporal residual CNN.
- 3. CROSS-MODAL DISTILLATION: CTC trains sequence-to-sequence models without explicit alignment by marginalizing over paths that collapse blanks and repeated labels into output sequences.The network predicts a distribution over graphemes and a blank symbol for each input frame.
- 3. CROSS-MODAL DISTILLATION: Frame-wise distillation minimizes the divergence between teacher and student CTC posterior distributions, providing the student with teacher-derived supervision at each frame.The paper identifies this objective as frame-wise knowledge distillation.
- 3. CROSS-MODAL DISTILLATION: The training objective combines the CTC loss on transcriptions with the frame-wise knowledge-distillation loss from teacher posteriors.The combined objective is illustrated in Fig. 1 and uses separate weights for the two loss terms.
4. EXPERIMENTAL SETUP
The experiments compare supervised, teacher-supervised, and unsupervised-pretraining regimes using VoxCeleb2, LRS2, and LRS3. Training uses a Jasper-based implementation with specified optimization settings and hardware.
- 4. EXPERIMENTAL SETUP: The study trains on VoxCeleb2, LRS2, and LRS3, evaluates on the LRS2 and LRS3 test sets, and compares full supervision, no supervision, and unsupervised pre-training followed by fine-tuning.The fine-tuning experiments use either all target-dataset annotations or only a small fraction of them.
- 4. EXPERIMENTAL SETUP: The no-supervision setting relies only on transcriptions and posteriors generated by the ASR teacher, whereas supervised training uses ground-truth transcriptions.Unsupervised pre-training first uses distillation on unlabeled data before supervised fine-tuning.
- 4. EXPERIMENTAL SETUP: The implementation uses the Nvidia Seq2Seq framework, a 10x5 Jasper Librispeech ASR teacher, a pretrained visual frontend, NovoGrad, four GPUs, and batch size 64 per GPU.The loss weights are set to λCTC = 0.1 and λKD = 10.
5. EXPERIMENTS
The method matches supervised performance without ground-truth transcriptions, improves further when extra unlabeled audio-visual data are used, and accelerates training through distillation. It achieves strong WER results on LRS2 and LRS3 using public data.
- 5. EXPERIMENTS: Without ground-truth transcriptions, the method reaches 58.2% WER on LRS2 and 65.6% on LRS3, matching or improving the supervised baseline.The supervised baseline records 58.5% on LRS2 and 68.8% on LRS3.
- 5. EXPERIMENTS: 51.3% WER on LRS2 and 59.8% on LRS3 are achieved after unsupervised pre-training followed by full supervised fine-tuning.These are the best results reported for the paper’s training scenarios.
- 5. EXPERIMENTS: 54.2% WER on LRS2 and 61.7% on LRS3 are obtained without ground-truth transcriptions after training on extra available data.The results improve over training on each target dataset individually.
- 5. EXPERIMENTS: Distillation significantly accelerates training compared with a CTC model trained on ground-truth transcriptions.The authors attribute this acceleration to explicit alignment information from distillation rather than CTC’s implicit signal.
6. DISCUSSION AND FUTURE WORK
The paper presents ASR-based distillation as an effective way to train visual speech recognition without manual annotations, while accelerating training. It also identifies cross-modal CTC distillation challenges and outlines extensions to larger datasets, models, and languages.
- 6. DISCUSSION AND FUTURE WORK: Distillation accelerates training relative to CTC alone by providing explicit alignment information instead of CTC’s implicit signal.Figure 2 compares validation greedy WER learning curves for the proposed method and a CTC-only baseline.
- 6. DISCUSSION AND FUTURE WORK: Distilling knowledge from a pretrained ASR model trains strong visual speech recognition systems without manually annotated data.The method can pretrain on unlabeled datasets and optionally use a small amount of annotations for fine-tuning.
- 6. DISCUSSION AND FUTURE WORK: The method exceeds existing lip reading systems except those trained with proprietary data, using only publicly available data.
- 6. DISCUSSION AND FUTURE WORK: The approach could extend lip reading to languages with limited annotations when pretrained ASR models and unlabeled talking-head video are available.
- 6. DISCUSSION AND FUTURE WORK: Cross-modal distillation may face CTC teacher–student spike-timing misalignment, but its extent remains open because sequence-level knowledge distillation showed no improvement.
- 6. DISCUSSION AND FUTURE WORK: Resource constraints limited experiments to VoxCeleb2 and a relatively small network, motivating future scaling in both dataset and model size.