PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

arXiv:2201.02184v2eess.AScs.CVcs.SD

TL;DR

Self-supervised speech representation learning has underused the correlated lip movements and audio in video. AV-HuBERT masks both streams and predicts iteratively refined multimodal hidden units, improving lip-reading and audio-only recognition across benchmark settings.

  • Problem

    Self-supervised speech representation learning has largely used audio alone despite the coupled audio and visual information in speech video.

  • Method

    AV-HuBERT masks audio and image sequences and predicts discrete cluster assignments whose targets are iteratively refined from learned audio-visual features.

  • Results

    AV-HuBERT improves both lip-reading and audio-only speech recognition, including 32.5% lip-reading WER with 30 labeled hours and 1.3% audio-only WER.

  • Takeaways & Limitations

    The learned audio-visual representations transfer to visual-only lip-reading and can also pre-train audio-based speech recognition models.

  • Takeaways & Limitations

    The dataset speaker distribution may not represent the global human population, creating potential societal, gender, racial, and other biases.

Abstract

from arXiv · show

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert

1 INTRODUCTION

Speech is inherently multimodal, yet self-supervised speech representation learning has largely relied on audio alone. AV-HuBERT addresses this gap by learning from masked audio-visual inputs and improves both lip-reading and audio-based recognition.

  • Visual speech cues support language learning, speech understanding in noise, and communication for people with speech impairments.
  • Audio and visual lip-movement signals provide coupled supervision for learning speech representations, but prior self-supervised frameworks were mostly audio-only.
  • AV-HuBERT masks audio and image sequences, predicts discrete cluster assignments, and iteratively refines targets using learned audio-visual features.
  • 32.5% lip-reading WER with 30 hours of labeled LRS3 data outperforms 33.6% from a prior model trained on 31,000 hours of transcribed video.
  • 26.9% lip-reading WER results from combining AV-HuBERT with self-training using all 433 labeled LRS3 hours.
  • 1.3% audio-only speech-recognition WER outperforms the previous state-of-the-art model at 2.3% and unimodal HuBERT at 1.5%.

2 RELATED WORK

Related work has developed multimodal self-supervision for general video understanding and audio-visual speech, including synchronization, translation, contrastive learning, and audio-guided visual representation learning.

  • General multimodal video representation learning targets high-level tasks such as action recognition and audio-event detection.
  • Prior general-video methods learn through audiovisual synchronization prediction and cyclic translation between modalities.
  • Audio-visual speech representation learning primarily targets improved lip-reading from untranscribed speech data.
  • Existing speech approaches include contrastive learning from synchronization and matching visual latent features to pretrained audio embeddings.

3 METHOD

AV-HuBERT extends HuBERT to audio-visual speech by iteratively clustering multimodal features and predicting masked cluster assignments. Its design combines audio-visual inputs, modality dropout, multimodal clustering, and visual masking by substitution.

  • Single-modal and cross-modal visual HuBERT: The visual HuBERT variants cluster image features into discrete units and train a visual encoder to predict those units from masked image sequences.Visual features can initially come from engineered extractors such as HoG and later from intermediate HuBERT layers.
  • AV-HuBERT: AV-HuBERT alternates feature clustering with masked prediction, using iteratively refined discrete targets for audio-visual speech representation learning.Targets begin with signal-processing acoustic features and are later refined by clustering features from the learned encoder.
  • Audio-visual input: Audio and image sequences pass through modality-specific encoders, are fused, and enter a shared transformer that predicts masked cluster assignments.Targets are generated from clustered audio features or features extracted from the preceding AV-HuBERT iteration.
  • Modality dropout: Modality dropout masks an entire audio or visual stream before fusion, training the model with visual-only, audio-only, or audio-visual inputs.This prevents the model from ignoring video and bridges multimodal pre-training with visual-only fine-tuning and inference.
  • Audio-visual clustering: Audio-visual clustering produces multimodal target assignments after the first iteration by combining complementary lip-movement and audio information.The paper reports that combining both modalities produces higher-quality cluster assignments than audio-based targets alone.
  • Masking by substitution: Masking by substitution replaces visual segments with random segments from the same video, creating temporally smooth imposter segments for masked prediction.The model must identify the substituted frames and infer labels for the original frames; an ablation evaluates this strategy.

4 EXPERIMENT

Experiments evaluate AV-HuBERT on lip-reading and ASR, including low-resource, multilingual, target-label, and pre-training comparisons. AV-HuBERT consistently improves recognition, while audio-visual targets and iterative refinement contribute to performance.

  • Experimental setup: AV-HuBERT uses LRS3 and VoxCeleb2, with English data for pre-training and two model sizes, BASE and LARGE.The experiments use 433 hours of transcribed LRS3 videos and 1,326 hours of English VoxCeleb2 data; BASE has 12 transformer blocks and LARGE has 24.
  • Lip-reading results: Using 1,759 hours of unlabeled data and 30 hours of labels, AV-HuBERT-LARGE outperforms prior lip-reading models trained with substantially more labeled data.The comparison includes a prior model trained with 1,000 times more labeled data; combining AV-HuBERT with self-training achieves a new state-of-the-art result using 7% of that data.
  • Lip-reading results: 32.5% WER is achieved with 30 labeled hours, compared with 33.6% for the prior state of the art trained on 31,000 hours of transcribed videos.Using the complete 433-hour LRS3 training set further reduces lip-reading WER to 28.6%.
  • Lip-reading results: Pre-training reduces WER versus the same architecture trained from scratch, from 92.3% to 32.5% in low-resource and from 62.3% to 28.6% in high-resource settings.With LRS3 alone, pre-training followed by fine-tuning achieves 41.6% WER versus 62.3% from scratch.
  • Ablation studies: Audio-visual targets improve iteratively, and AV-HuBERT transfers audio knowledge into the visual encoder more effectively than cross-modal or single-modality visual variants.Starting target clustering from MFCC features is critical; MFCC clusters have 30.3% phone-label purity versus 16.4% for HoG clusters.
  • Multilingual and ASR results: Non-English pre-training helps with little English data but can hurt when English data increases, while audio-visual targets improve audio-HuBERT ASR across resource settings.The best audio-HuBERT model reaches 1.4% WER, outperforming the prior 2.3% state of the art without an external language model.

5 CONCLUSION

AV-HuBERT uses correlated audio and lip movements for masked-prediction speech representation learning, achieving strong visual and audio speech-recognition results with substantially less labeled data than prior work.

  • AV-HuBERT alternates feature clustering with masked prediction, using targets initialized from MFCC features and refined through iterative training.The model consumes masked image and audio frames to predict target cluster assignments.
  • 433 hours of text transcriptions yield state-of-the-art visual speech recognition with two orders of magnitude less labeled data than the prior best approach.
  • Using one-thousandth of the labeled data, AV-HuBERT outperforms the prior lip-reading state of the art by more than 10% relative.
  • An audio-HuBERT model trained with AV-HuBERT-generated targets achieves state-of-the-art audio-based speech recognition on LRS3.

ETHICAL STATEMENT

The paper identifies representativeness, privacy, security, and misuse concerns associated with its publicly available audio-visual speech data and technology.

  • The speaker distribution may not represent the global human population, creating potential societal, gender, racial, and other biases.
  • The method can support security and crime investigations but could also enable malicious surveillance and wiretapping.
  • The authors distribute code and models carefully, emphasizing potential security and privacy concerns.

REPRODUCIBILITY STATEMENT

The paper states that its code and models are publicly available and provides implementation details alongside architectural illustrations of the compared HuBERT variants.

  • The code and models are publicly available, and the paper includes as many implementation details as possible.
  • A MODEL ILLUSTRATION: Figure A.1 compares the proposed AV-HuBERT with single-modal and cross-modal visual HuBERT.
  • A MODEL ILLUSTRATION: The illustration labels single-modal visual HuBERT, cross-modal visual HuBERT, and the proposed audio-visual HuBERT.
  • A MODEL ILLUSTRATION: The model illustration identifies masked frames, multimodal cluster ids, and contextualized audio-visual representations.

B.1 DATASETS

The experiments use labeled and unlabeled audio-visual speech data, with preprocessing, iterative pretraining, fine-tuning, decoding, self-training, and a LiRA comparison described for reproducibility.

  • DATASETS: LRS3 contains over 400 hours of sentence-level English TED and TEDx video, including 403 hours of pretraining data and 30 hours of trainval data.The low-resource setup uses only trainval as labeled data, with 1,200 sequences selected for validation.
  • DATASETS: VoxCeleb2 contains over 2,442 hours from more than 6,000 speakers; the experiments use its 1,326-hour English portion.The dataset is unlabeled and has substantial domain discrepancy from LRS3.
  • PREPROCESSING: Video frames are aligned using 68 facial keypoints, cropped to a 96×96 mouth-centered region, converted to grayscale, and augmented during training.Training uses random 88 × 88 crops and horizontal flips; testing uses center-cropped 88 × 88 regions.
  • PRETRAINING: Pretraining begins with k-means targets from 39-dimensional MFCC features and subsequently refines targets using intermediate model features.The models are trained for five iterations, with BASE generating clusters and LARGE trained only in the fifth iteration.
  • PRETRAINING: Training masks fused features in iterations 1–4 and uses modality dropout with both modality probabilities set to 0.5.
  • FINE-TUNING AND DECODING: Fine-tuning removes the audio encoder and supports CTC or sequence-to-sequence decoding on labeled video-text pairs.CTC uses a 4-gram language model, whereas sequence-to-sequence decoding uses no language model.
  • SELF-TRAINING: Self-training generates pseudo-labels with a fine-tuned LARGE model and combines pseudo-labeled videos with ground-truth labeled videos.

C.1 AMOUNT OF LABELED DATA

Pre-training provides consistent gains across fine-tuning regimes, including low-resource settings, while speaker overlap substantially changes reported lip-reading WER. Self-training alone is weaker than AV-HuBERT-based approaches.

  • Pre-training brings large and consistent gains when fine-tuning with 1, 10, or 100 hours of labeled data.The experiments use 433 hours of unlabeled LRS3 data.
  • CTC-based fine-tuning outperforms S2S-based fine-tuning with 1 or 10 hours of labeled data.The passage attributes this trend partly to S2S overfitting in low-resource settings.
  • 18.0 ± 0.5% WER is achieved for seen speakers, versus 30.5% for unseen speakers under the open-speaker setting.The closed-speaker evaluation uses five randomly sampled test groups.
  • Self-training alone has significantly higher WER than AV-HuBERT and self-trained AV-HuBERT.This indicates that the combined approach’s gain is primarily attributed to AV-HuBERT.

C.4 FULL RESULTS WITH CTC FINE-TUNING

Full CTC results support the section’s S2S-based conclusions: AV-HuBERT benefits different labeled-data settings and improves over prior comparisons on LRS3.

  • CTC results generally preserve the conclusions drawn from S2S fine-tuning.These include benefits from AV-HuBERT pre-training across settings and improvement over LiRA.
  • The self-trained AV-HuBERT comparison includes models trained from scratch, with self-training only, and with AV-HuBERT only.All compared models are Transformer-LARGE.
  • Table C.4 reports AV-HuBERT results alongside prior works on the LRS3-TED dataset.Ma et al. is re-implemented with the same architecture for a fairer comparison.

D ABLATION STUDIES

Ablations identify masking design, modality-specific masking probabilities, modality dropout, and prediction-loss placement as important components of AV-HuBERT training.

  • Masking Strategy: Substitution masking with a segment from the same utterance achieves the best result among tested masking strategies.It preserves temporal smoothness while replacing masked content with visually similar frames.
  • Masking Strategy: Feature masking performs worst because information leaks to the ResNet, while artificial masked frames enlarge the pre-training–fine-tuning domain gap.Direct masking with Gaussian noise or learned embeddings slightly improves performance but corrupts the raw image sequence.
  • Masking probability: Audio and visual streams require independent masking probabilities because their optimal masking probabilities differ.The last iteration uses 0.8 for acoustic frames and 0.3 for image frames.
  • Modality dropout: Randomly dropping the audio sequence prevents over-reliance on audio and helps visual representation learning.This is the purpose of modality dropout in the ablation.
  • Where to compute prediction loss: The loss weight on unmasked regions has little impact on fine-tuning performance.The passage contrasts this with Audio HuBERT, where masked prediction is more important.

E.1 MEASURING CLUSTERING QUALITY

Cluster quality is evaluated against frame-level phonetic labels using purity and NMI, and generally improves through iterative refinement. The best layer shifts later as target quality improves.

  • Cluster assignments are evaluated using purity and Normalized Mutual Information against forced-alignment phonetic labels.The labels come from a monophone HMM-GMM ASR model trained on LRS3.
  • Input-level masking yields worse audio-visual clusters than feature-level masking, with NMI of 27.2%.Feature-level masking better aligns the two original input streams in the learned features.
  • Cluster assignment quality generally improves with more training iterations.Figure E.1 compares features across layers, iterations, and cluster counts.
  • 44.2% NMI is achieved by the selected 12th layer in the last iteration.More iterations provide stable gains that are less sensitive to which layer is selected.
  • The best feature layer shifts toward later layers as target quality improves across iterations.In the first iteration, middle layers show higher cluster quality than other layers.
  • Lip-reading errors are notably higher for short utterances, while ASR shows a different sentence-length pattern.Homophone substitutions are identified as another common lip-reading error source.
Loading 2201.02184v2…