Source-linked AI summary
Visual Speech Recognition for Multiple Languages in the Wild
Pingchuan Ma, Stavros Petridis, Maja Pantic
TL;DR
VSR has advanced largely through larger datasets, leaving the contribution of model design less clear. The paper combines auxiliary prediction tasks with optimization and augmentation, achieving strong multilingual results and further gains from additional data.
Problem
VSR improvements have often relied on increasingly large datasets, making it unclear how much progress comes from better models rather than more data.
Method
The paper revises a VSR architecture through hyperparameter optimization, time-masking, and auxiliary prediction of audio and visual representations.
Results
The approach achieves state-of-the-art performance across six languages, outperforms publicly trained methods by a large margin, and improves further with additional data.
Takeaways & Limitations
VSR progress can come from careful model design and optimization as well as larger training sets.
Takeaways & Limitations
VSR is dual-use technology whose potential surveillance applications raise privacy and misuse concerns, motivating appropriate regulation.
Abstract
from arXiv · showhide
Visual speech recognition (VSR) aims to recognize the content of speech based on lip movements, without relying on the audio stream. Advances in deep learning and the availability of large audio-visual datasets have led to the development of much more accurate and robust VSR models than ever before. However, these advances are usually due to the larger training sets rather than the model design. Here we demonstrate that designing better models is equally as important as using larger training sets. We propose the addition of prediction-based auxiliary tasks to a VSR model, and highlight the importance of hyperparameter optimization and appropriate data augmentations. We show that such a model works for different languages and outperforms all previous methods trained on publicly available datasets by a large margin. It even outperforms models that were trained on non-publicly available datasets containing up to to 21 times more data. We show, furthermore, that using additional training data, even in other languages or with automatically generated transcriptions, results in further improvement.
1 Introduction
The paper argues that VSR progress depends on model design and optimization as well as training-set size, and evaluates a prediction-auxiliary-task approach across multiple languages and datasets.
- 1 Introduction: VSR research has largely relied on increasingly large transcribed datasets, leaving improvements attributable to model design versus data scale unclear.Some prior systems use tens of thousands of hours of non-public training data.
- 1 Introduction: State-of-the-art performance is achieved across English, Mandarin, Spanish, Italian, French, and Portuguese, extending evaluation beyond predominantly English-only VSR work.The paper reports this result using publicly available datasets.
- 1 Introduction: Additional data, including videos in other languages and automatically transcribed English videos, reduces WER across languages, although the reduction is smaller in Mandarin because of language mismatch.The authors interpret these findings as evidence that dataset size contributes substantially to recent improvements.
- 1 Introduction: The proposed approach combines hyperparameter optimization, time-masking, and prediction-based auxiliary tasks that jointly predict audio and visual representations.The auxiliary prediction task supplies additional supervision for learning visual representations.
- 1.4 Results on LRS2: 5 % lower best WER is achieved than the previous state of the art on LRS2, rising to 9.2 % with the same training-set size and 12.4 % with additional data.The model also achieves a 22.8 % absolute best-WER improvement over a comparable method using non-public data.
- 1.5 Results on LRS3: 8.2 % absolute improvement in best WER is achieved over the state of the art on LRS3 with the same training data, increasing to 11.8 % with additional data.With a smaller training set, the reported absolute improvement is 5.4 %.
- 1.6 Results on CMLR / 1.7 Results on CMU-MOSEAS-Spanish: 12.9 % absolute improvement over the state of the art is achieved on CMLR, while Spanish results show a 7.7 % absolute WER reduction and a further 6.5 % reduction with additional data.The CMLR metric is CER, whereas the Spanish metric is WER.
- 1 Introduction: Performance comparisons report mean and best results over ten runs, with mean WER up to 0.8 % worse than best WER except on CMLR.This distinction matters particularly for comparisons on relatively small test sets.
2 Applications, Challenges and Ethical Considerations
VSR supports communication, transcription, and audiovisual speech applications, but its broader deployment remains constrained by visual variability, language and speech-context differences, privacy risks, regulation, and potential dataset bias.
- Applications: VSR can enhance ASR in noisy environments because visual speech is largely unaffected by acoustic noise.The improvement over ASR increases as the signal-to-noise ratio decreases.
- Applications: Silent speech interfaces could enable communication without audible speech, including for people with aphonia or dysphonia.VSR can also transcribe silent or otherwise audio-free video and assist face-forgery detection.
- Challenges: Visual ambiguities and sensitivity to lighting, occlusion, blur, compression, resolution, and frame-rate mismatches limit VSR robustness.Further studies are needed to establish the impact of each visual noise type.
- Challenges: Unknown speakers and head poses can alter mouth appearance, making person-independent and pose-invariant recognition difficult.Multi-view cameras offer an opportunity, but the optimal lip views and their combination remain unclear.
- Challenges: VSR remains underexplored for silent speech, speech in noise, and non-English languages with distinct characteristics and accents.Existing evidence reports performance drops in some altered speech conditions.
- Ethical Considerations: VSR is dual-use: surveillance through CCTV or smartphone cameras creates privacy concerns, while regulation and auditing are advocated before commercialization.The authors also warn that datasets may underrepresent demographic groups, creating potential bias across age, gender, culture, and speaker background.
3 Visual Speech Recognition
The approach combines multilingual VSR evaluation with data preprocessing, performance metrics, augmentation, auxiliary prediction tasks, and progressively larger training sets. These design and training choices improve performance across languages and datasets.
- The method outperforms state-of-the-art VSR methods by a large margin across multiple languages.
- Performance metrics: WER measures the distance between predicted and target word sequences using substitutions, deletions, insertions, and target-word count.
- Data augmentation: Time-masking randomly replaces consecutive video frames with the mean frame, encouraging contextual use and disambiguation of similar lip movements.
- Ablation study: Removing time-masking or either auxiliary loss increases mean WER, while removing both auxiliary losses and time-masking causes a substantial performance decrease.
- Prediction-based auxiliary tasks: The auxiliary task predicts audio and visual representations learned by pretrained ASR and VSR encoders from intermediate model layers.
- Additional training data: Performance keeps improving as training incorporates more datasets, including additional data with automatically generated transcriptions, although gains are smaller for Mandarin than English.
4 Conclusions
The paper concludes that strong multilingual VSR performance can result from both larger datasets and careful model design. It highlights hyperparameter optimization, time-masking, auxiliary prediction tasks, and additional data as important components.
- State-of-the-art VSR performance is achieved through careful model design as well as larger datasets.
- Hyperparameter optimization, time-masking, and auxiliary tasks improve the proposed VSR approach.
- The approach outperforms existing VSR works trained on publicly available datasets in English, Spanish, and Mandarin by a large margin.
Authors’ Contributions
The manuscript and experiments were conducted by the listed authors, with the entire project supervised by M.P.
- P.M. wrote the code and conducted experiments with S.P., while P.M., S.P., and M.P. wrote the manuscript.
- M.P. supervised the entire project.
S1 Datasets Details
The study contrasts publicly available audio-visual datasets with substantially larger non-public datasets. Non-public datasets are one to two orders of magnitude larger.
- Non-publicly available audio-visual datasets are one to two orders of magnitude larger than publicly available datasets.
S2 Architecture Details
The VSR architecture combines visual front-end processing, a 12-block Conformer back-end, hybrid CTC/transformer decoding, and auxiliary predictors.
- The model contains a front-end encoder, VSR encoder, back-end encoder, hybrid CTC and transformer decoder, and two predictors.
- The front end uses a 3D convolution followed by ResNet-18 and global average pooling to produce visual feature sequences.
- The back end projects features to 256 dimensions and processes them through 12 Conformer blocks combining feed-forward, self-attention, and convolution modules.
- The decoder embeds target prefixes, applies residual multi-head attention blocks, and uses encoder-decoder attention to focus on relevant encoded inputs.
- A linear softmax layer maps back-end features to character sequences under CTC training, while predictors use sixth-block features to estimate pretrained audio or visual features.
S3 Pre-trained VSR and ASR models
The pretrained ASR and VSR models provide modality-specific representations for the full model and are trained on the same data as it.
- The pretrained VSR model shares the full model architecture but omits the predictors, while the pretrained ASR model replaces the VSR encoder with an ASR encoder.
- Increasing batch size from 8 to 16 reduced validation WER on LRS2, with the same pattern on LRS2 and LRS3 test sets.
- Six monolingual transformer language models were trained for 50 epochs using combined text corpora for the paper’s languages.
S6 Time Masking
Time masking replaces consecutive video frames with the video’s mean frame, with mask duration and count scaled to sequence length.
- Time masking replaces n consecutive frames with the video mean frame, sampling duration uniformly from zero to an upper bound.
- The method uses one mask per second, with each mask lasting at most 0.4 seconds.
- The number of masks is proportional to sequence length to accommodate the datasets’ large variance in video duration.
S7 Loss Functions
The system uses a hybrid CTC/attention objective, language-model and auxiliary-loss weighting, and evaluates configurations across datasets and training choices.
- The hybrid architecture maps audio or visual input sequences to target characters using CTC and attention-based decoding.
- CTC estimates sequence probabilities under conditional independence, whereas attention-based decoding estimates the posterior using the chain rule.
- Supplementary tables report pretrained-model performance, hyperparameter and language-model investigations, curriculum-learning experiments, and layer-placement effects across datasets.
- The speech-recognition objective linearly combines CTC and cross-entropy losses with α = 0.1.
- Grid search selected auxiliary-loss weights βa = 0.4 and βv = 0.4 on the LRS2 validation set for all experiments.
S8 Curriculum Learning
Training the end-to-end model from scratch performs poorly on LRS2 and LRS3 because their long utterances make learning difficult. Curriculum learning resolves this by progressively introducing longer videos.
- S8 Curriculum Learning: Poor performance on LRS2 and LRS3 results when the end-to-end model is trained from scratch.The authors attribute this difficulty to the datasets’ many very long utterances.
- S8 Curriculum Learning: Table S11 reports the curriculum learning experiments conducted on the LRS3 dataset.
- S8 Curriculum Learning: Curriculum learning trains first on videos shorter than 100 frames, then on utterances up to 150 frames.The model trained on shorter videos initializes the subsequent longer-utterance training stage.
S9 Additional Results
Additional experiments examine auxiliary-loss placement, multilingual performance, pretrained-model choices, time masking, and architecture variations. Across these settings, the proposed approach reduces WER, while auxiliary-loss gains become smaller with larger training sets.
- S9 Additional Results: Layer 6 is the optimal attachment point for the auxiliary loss on the LRS2 validation set.The tested layer positions range from 0 to 12 in increments of 2.
- S9 Additional Results: 5.6% absolute WER reduction is achieved on Multilingual TEDx-Spanish, with a further 4.2% reduction from additional training data.
- S9 Additional Results: 5.6% absolute WER reduction is achieved on Multilingual TEDx-Italian, with a further 8% reduction from additional training data.The Italian training corpus contains 26,387 videos totaling 45.8 hours after manual cleaning.
- S9 Additional Results: 4.2% absolute WER reduction is achieved on Multilingual TEDx-Portuguese, with a further 3.9% reduction from additional training data.
- S9 Additional Results: 8.5% absolute WER reduction is achieved on CMU-MOSEAS-Portuguese, with a further 5.6% reduction from additional training data.
- S9 Additional Results: 11.5% absolute WER reduction is achieved on CMU-MOSEAS-French, with a further 9.3% reduction when additional training data is included.
- S9 Additional Results: Removing either auxiliary loss or time masking increases mean WER on LRS3, while auxiliary-loss gains are smaller with larger training sets.The LRS3 ablation uses LRW, LRS2, LRS3, and AVSpeech, totaling 1,459 hours.
- S9 Additional Results: Replacing the proposed front ends with ShuffleNet v2 increases WER by 4.4% and 0.6%, while auxiliary losses still outperform the baseline across architectures.The authors connect auxiliary losses to improved supervision of intermediate-layer visual representations.