Source-linked AI summary
AVSRBench: A Multi-Condition AVSR Benchmark
Rishabh Jain, Naomi Harte
TL;DR
AVSR’s strong broadcast-benchmark performance leaves its broader generalization uncertain. The paper evaluates three architectures across six conditions, introduces RoomReader-AV and a unified preprocessing pipeline, and finds that visual-only systems fail beyond broadcast speech while fusion helps mainly in Lombard conditions. LLM-based decoding further brings poor out-of-domain behavior and substantial deployment overhead.
Problem
Broadcast-focused AVSR benchmarks provide limited evidence about generalization to varied speech, speakers, views, and conversational settings.
Method
The paper evaluates three AVSR architectures across six conditions and releases RoomReader-AV plus a unified preprocessing pipeline for comparable multi-condition testing.
Results
Visual-only recognition fails outside broadcast speech, audio-video fusion mainly helps Lombard speech, and extreme profile views expose reliance on acoustic fallback.
Takeaways & Limitations
Standard LRS2 and LRS3 evaluation is insufficient to characterize real-world AVSR generalization, motivating broader benchmark coverage.
Takeaways & Limitations
Llama-AVSR requires 17.91 GB of GPU memory for AV inference versus 2.29 GB for Auto-AVSR and has slow autoregressive generation.
Abstract
from arXiv · showhide
While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true generalization or just domain adaptation. To investigate this gap, we evaluate three AVSR architectures across six conditions: controlled broadcast speech, fixed-grammar utterances, hyper-articulated Lombard speech, read speech from professional lipspeakers and non-professional speakers, and spontaneous multi-party video conversations. We find that visual-only performance deteriorates rapidly beyond broadcast domains, and audio-video fusion mainly benefits Lombard speech environments. Visual understanding degrades sharply at 90° profile views, with multimodal systems relying largely on acoustic fallback. Additionally, speaker articulation proves more critical than minor camera shifts, and LLM-based architectures suffer from poor out-of-domain generalization. Our work highlights a significant generalization gap in current AVSR research. To address this, we also introduce RoomReader-AV as a new benchmark for AVSR and release a unified data preprocessing pipeline to make comprehensive multi-condition evaluation accessible.
I. INTRODUCTION
AVSR’s near-perfect broadcast-benchmark results do not establish generalization beyond controlled speech. This paper addresses that gap with standardized multi-condition evaluation, RoomReader, and a reusable preprocessing toolkit.
- Motivation: Broadcast benchmarks can make AVSR appear nearly solved, despite limited evidence about performance beyond controlled speech.LRS2 and LRS3 results approach near-perfect WER, while their broadcast conditions constrain what is tested.
- Motivation: Standardized evaluation is difficult because existing model code is tightly coupled to LRS2 and LRS3, requiring custom preparation for new corpora.This limits fair, systematic comparison across datasets.
- Contributions: The paper releases a compatible preprocessing toolkit, introduces RoomReader for spontaneous multi-party conversations, and evaluates three AVSR systems across six datasets.The evaluation includes nonfrontal views, speaker articulation, Lombard speech, and video conferencing.
- Findings: Visual-only recognition fails outside broadcast conditions, while audio-video fusion mainly helps Lombard speech and otherwise offers little advantage over audio-only recognition.The evaluation compares visual-only, audio-only, and audio-visual settings.
II. DATASETS AND PROCESSING
The benchmark spans broadcast, fixed-grammar, Lombard, read-speech, and spontaneous conversational conditions. Its datasets vary systematically in vocabulary, articulation, camera angle, and recording environment.
- Dataset Overview: The evaluation covers six datasets ranging from broadcast speech to fully spontaneous conversation.LRS2 and LRS3 provide broadcast conditions, while GRID, LombardGrid, TCD-TIMIT, and RoomReader add controlled and conversational challenges.
- Dataset Overview: GRID tests generalization to unseen vocabulary and sentence structure using clean frontal recordings with a fixed six-word grammar.Its 34 speakers use command utterances absent from training.
- Dataset Overview: LombardGrid elicits louder, higher-pitched, hyper-articulated speech and pairs synchronized frontal and 90° profile views.The matched views enable direct camera-angle comparisons within the same recording session.
- Dataset Overview: TCD-TIMIT separates professional lipspeakers from non-professional volunteers and further divides both groups by 0° and 30° camera angles.The dataset contains 62 speakers and 6,913 phonetically rich sentences.
- Dataset Overview: RoomReader contains spontaneous Zoom tutorials with 118 participants, corrected transcripts, and separate session-level and speaker-isolated audio streams.The reported evaluation uses individual participant streams and Easy/Hard subsets.
B. Preprocessing Pipeline
The preprocessing pipeline standardizes heterogeneous datasets for direct use with AV-HuBERT and Auto-AVSR. It reduces format incompatibility without modifying either model codebase.
- B. Preprocessing Pipeline: The released pipeline converts GRID, LombardGrid, TCD-TIMIT, and RoomReader into formats compatible with AV-HuBERT and Auto-AVSR.It is designed to make multi-condition evaluation easier and more comparable.
- B. Preprocessing Pipeline: The pipeline addresses a practical barrier to multi-condition evaluation: substantial undocumented engineering required to adapt new datasets to established frameworks.The released documentation describes dataset-specific cleaning steps.
- B. Preprocessing Pipeline: The pipeline applies consistent face detection, 96×96 mouth-region extraction, 16 kHz mono audio standardization, and transcript normalization.RetinaFace and 68-point landmarks are used for mouth-region extraction.
- B. Preprocessing Pipeline: GRID transcripts are generated from its fixed six-word vocabulary, while RoomReader disfluency markers are removed before manifest generation.The resulting manifests support the two target frameworks.
III. AVSR MODELS
The study evaluates three modern AVSR architectures under identical decoding and WER protocols across visual-only, audio-only, and audio-visual modalities. The models represent supervised, self-supervised, and LLM-decoder approaches.
- III. AVSR MODELS: Three pretrained AVSR models are evaluated with fixed official decoding settings, using WER across visual-only, audio-only, and audio-visual conditions.The protocol uses each model’s official pretrained checkpoint.
- III. AVSR MODELS: Auto-AVSR represents large-scale supervised learning with a ResNet-18 visual frontend, Conformer encoder, and hybrid CTC/attention decoder.The main checkpoint uses 3,448 hours of audiovisual data.
- III. AVSR MODELS: AV-HuBERT Large represents self-supervised multimodal pretraining followed by LRS3 fine-tuning with a shared Transformer.The evaluated checkpoint uses noise augmentation.
- III. AVSR MODELS: Llama-AVSR combines AV-HuBERT visual features and Whisper acoustic features with a Llama3.1-8B decoder fine-tuned using LoRA.Fine-tuning uses LRS3 and VoxCeleb2.
IV. EXPERIMENT AND RESULTS
On LRS2 and LRS3, audio-visual recognition remains near audio-only performance, while visual-only recognition is substantially weaker, especially outside matched training conditions.
- LRS3 AV performance is at or near AO for all models, with Llama-AVSR achieving 0.79% AV WER and 0.74% AO WER.
- Table II reports AVSR results on LRS2 and LRS3, while Table III is identified as the GRID benchmark rather than an LRS result table.
- AV-HuBERT reaches 38.00% VO WER on unseen LRS2 versus 41.57% for Llama-AVSR, yet VO remains dramatically weaker than AO and AV.
B. Structured Read Speech: GRID
On the unseen fixed-grammar GRID corpus, visual recognition collapses and audiovisual fusion can underperform audio-only recognition, unlike the LombardGrid condition where hyper-articulation helps.
- B. Structured Read Speech: GRID: GRID VO WER reaches 66.53% for Auto-AVSR, 83.80% for AV-HuBERT, and over 116% for Llama-AVSR, while AV underperforms AO for Auto-AVSR and Llama-AVSR.
- B. Structured Read Speech: GRID: AO errors on GRID primarily involve homophones, whereas VO errors reflect difficulty recognizing visually ambiguous words under the fixed grammar.
- C. Lombard Speech Under Noise: LombardGrid: On LombardGrid, Auto-AVSR improves from 14.02% AO WER to 11.73% AV WER, reversing its -8.90% AV degradation on standard GRID.
- C. Lombard Speech Under Noise: LombardGrid: At a 90° profile view, VO WER rises from 64.88% to 92.50% for Auto-AVSR, while AV changes only from 11.59% to 11.87%.
D. Camera Angle and Speaker Articulation: TCD-TIMIT
TCD-TIMIT separates camera-angle effects from speaker-articulation effects by comparing frontal and semi-profile recordings across professional lipspeakers and volunteers.
- D. Camera Angle and Speaker Articulation: TCD-TIMIT: TCD-TIMIT uses 0° and 30° recordings to examine camera angle independently from speaker articulation.
- D. Camera Angle and Speaker Articulation: TCD-TIMIT: Table VII reports WER across 0° frontal and 30° semi-profile views, while Table VIII compares lipspeakers, volunteers, and matched volunteers across modalities.
1) Effect of Camera Angle in TCD-TIMIT:
Across TCD-TIMIT, moderate camera changes from 0° to 30° produce little WER variation, indicating tolerance to small frontal offsets.
- 1) Effect of Camera Angle in TCD-TIMIT:: WER changes remain below 3% absolute between 0° frontal and 30° semi-profile views across all configurations.
2) Speaker Articulation in TCD-TIMIT:
TCD-TIMIT results show that professional articulation strongly improves visual recognition, while moderate camera offsets have little effect. In spontaneous RoomReader-AV speech, visual performance collapses and short utterances are especially difficult.
- Articulation quality: 25.91% versus 45.57% WER: Auto-AVSR performs substantially better visually on professional lipspeakers than matched volunteers.At 0°, the matched-volunteer gap demonstrates the importance of articulation quality for visual recognition.
- RoomReader-AV: RoomReader-AV is the most challenging condition: VO collapses, while Auto-AVSR and AV-HuBERT retain viable AO WERs of 25.40% and 36.18%.Llama-AVSR reaches 88.01% AO WER and suffers a severe multimodal penalty.
- RoomReader-AV: Short RoomReader-AV utterances have much higher WER, which drops rapidly as utterance duration increases.The analysis attributes the short-clip difficulty to limited temporal context, especially below 2 seconds.
F. Scaling and Generalization in Visual Speech Recognition
Scaling broadcast training data improves visual recognition in structured settings but does not resolve out-of-domain failure. Across conditions, fusion, pose, articulation, and decoder choice reveal that visual robustness remains limited and LLM-based decoding adds deployment costs.
- Scaling and generalization: Doubling Auto-AVSR training data from 1.7k to 3.4k hours improves most datasets but leaves RoomReader-AV nearly unchanged.The result indicates that scaling broadcast data does not remove the out-of-domain visual bottleneck.
- Generalization: 66–313% WER out-of-domain versus 19–44% in-domain shows that video-only recognition fails to generalize beyond broadcast speech.A 19% LRS3 VO WER can coexist with failure on every other tested condition.
- Multimodal fusion: AV fusion mainly improves in-domain and LombardGrid recognition; elsewhere it offers little benefit or worsens AO performance.The discussion attributes this behavior to fusion mechanisms that continue trusting unreliable visual features.
- Articulation and pose: At 90° profile views, AV performance is maintained primarily through audio fallback after visual features become uninformative.A 30° offset remains within the encoder’s frontal range, unlike the extreme profile condition.
- Articulation and pose: A 17% to 44% articulation gap is much larger than the minimal 2% change from a 30° camera offset.This comparison places speaker articulation above moderate camera position as a determinant of visual performance.
- LLM-based decoders: Llama-AVSR requires 17.91 GB of GPU memory for AV inference versus 2.29 GB for Auto-AVSR, alongside unstable out-of-domain outputs.The reported overhead and slow autoregressive generation limit suitability for low-latency real-time use.
VI. CONCLUSION AND FUTURE WORK
The paper finds that current AVSR progress is narrow: visual-only systems fail beyond broadcast speech, apparent pose robustness often reflects acoustic fallback, and LLMs introduce domain-shift challenges. It responds with RoomReader-AV and a unified preprocessing pipeline for broader evaluation.
- Conclusion: Visual-only AVSR fails to generalize outside broadcast speech across architectures and training scales.The conclusion frames standard LRS2 and LRS3 evaluation as insufficient for characterizing real-world generalization.
- Future work: The paper introduces RoomReader-AV for spontaneous multiparty conversational AVSR and releases a pipeline supporting evaluation across multiple datasets and frameworks.The pipeline also simplifies extension to newer ASR, VSR, and AVSR systems.