PaperlayerView exact revision on arXiv

Source-linked AI summary

AVSRBench: A Multi-Condition AVSR Benchmark

Rishabh Jain, Naomi Harte

arXiv:2609.10366v1eess.AScs.CVcs.MM

TL;DR

AVSR’s strong broadcast-benchmark performance leaves its broader generalization uncertain. The paper evaluates three architectures across six conditions, introduces RoomReader-AV and a unified preprocessing pipeline, and finds that visual-only systems fail beyond broadcast speech while fusion helps mainly in Lombard conditions. LLM-based decoding further brings poor out-of-domain behavior and substantial deployment overhead.

  • Problem

    Broadcast-focused AVSR benchmarks provide limited evidence about generalization to varied speech, speakers, views, and conversational settings.

    Source: §I · p. 1

  • Method

    The paper evaluates three AVSR architectures across six conditions and releases RoomReader-AV plus a unified preprocessing pipeline for comparable multi-condition testing.

    Source: §I · p. 1, B. Preprocessing Pipeline · p. 2, §VI · p. 6

  • Results

    Visual-only recognition fails outside broadcast speech, audio-video fusion mainly helps Lombard speech, and extreme profile views expose reliance on acoustic fallback.

    Source: §V · p. 6

  • Takeaways & Limitations

    Standard LRS2 and LRS3 evaluation is insufficient to characterize real-world AVSR generalization, motivating broader benchmark coverage.

    Source: §VI · p. 6

    Llama-AVSR requires 17.91 GB of GPU memory for AV inference versus 2.29 GB for Auto-AVSR and has slow autoregressive generation.

    Source: §V · p. 6

Abstract

from arXiv · show

While AVSR has achieved sub-1% word error rates on the standard LRS3 benchmark, its reliance on broadcast speech obscures whether this reflects true generalization or just domain adaptation. To investigate this gap, we evaluate three AVSR architectures across six conditions: controlled broadcast speech, fixed-grammar utterances, hyper-articulated Lombard speech, read speech from professional lipspeakers and non-professional speakers, and spontaneous multi-party video conversations. We find that visual-only performance deteriorates rapidly beyond broadcast domains, and audio-video fusion mainly benefits Lombard speech environments. Visual understanding degrades sharply at 90° profile views, with multimodal systems relying largely on acoustic fallback. Additionally, speaker articulation proves more critical than minor camera shifts, and LLM-based architectures suffer from poor out-of-domain generalization. Our work highlights a significant generalization gap in current AVSR research. To address this, we also introduce RoomReader-AV as a new benchmark for AVSR and release a unified data preprocessing pipeline to make comprehensive multi-condition evaluation accessible.

I. INTRODUCTION

AVSR’s near-perfect broadcast-benchmark results do not establish generalization beyond controlled speech. This paper addresses that gap with standardized multi-condition evaluation, RoomReader, and a reusable preprocessing toolkit.

Motivation

  • Broadcast benchmarks can make AVSR appear nearly solved, despite limited evidence about performance beyond controlled speech.LRS2 and LRS3 results approach near-perfect WER, while their broadcast conditions constrain what is tested.Source: §I · p. 1
  • Standardized evaluation is difficult because existing model code is tightly coupled to LRS2 and LRS3, requiring custom preparation for new corpora.This limits fair, systematic comparison across datasets.Source: §I · p. 1

Contributions

  • The paper releases a compatible preprocessing toolkit, introduces RoomReader for spontaneous multi-party conversations, and evaluates three AVSR systems across six datasets.The evaluation includes nonfrontal views, speaker articulation, Lombard speech, and video conferencing.Source: §I · pp. 1–2

Findings

  • Visual-only recognition fails outside broadcast conditions, while audio-video fusion mainly helps Lombard speech and otherwise offers little advantage over audio-only recognition.The evaluation compares visual-only, audio-only, and audio-visual settings.Source: §I · p. 2

II. DATASETS AND PROCESSING

The benchmark spans broadcast, fixed-grammar, Lombard, read-speech, and spontaneous conversational conditions. Its datasets vary systematically in vocabulary, articulation, camera angle, and recording environment.

Dataset Overview

  • The evaluation covers six datasets ranging from broadcast speech to fully spontaneous conversation.LRS2 and LRS3 provide broadcast conditions, while GRID, LombardGrid, TCD-TIMIT, and RoomReader add controlled and conversational challenges.Source: A. Dataset Overview · p. 2
  • GRID tests generalization to unseen vocabulary and sentence structure using clean frontal recordings with a fixed six-word grammar.Its 34 speakers use command utterances absent from training.Source: A. Dataset Overview · p. 2
  • LombardGrid elicits louder, higher-pitched, hyper-articulated speech and pairs synchronized frontal and 90° profile views.The matched views enable direct camera-angle comparisons within the same recording session.Source: A. Dataset Overview · p. 2
  • TCD-TIMIT separates professional lipspeakers from non-professional volunteers and further divides both groups by 0° and 30° camera angles.The dataset contains 62 speakers and 6,913 phonetically rich sentences.Source: A. Dataset Overview · p. 2
  • RoomReader contains spontaneous Zoom tutorials with 118 participants, corrected transcripts, and separate session-level and speaker-isolated audio streams.The reported evaluation uses individual participant streams and Easy/Hard subsets.Source: A. Dataset Overview · p. 2

B. Preprocessing Pipeline

The preprocessing pipeline standardizes heterogeneous datasets for direct use with AV-HuBERT and Auto-AVSR. It reduces format incompatibility without modifying either model codebase.

B. Preprocessing Pipeline

  • The released pipeline converts GRID, LombardGrid, TCD-TIMIT, and RoomReader into formats compatible with AV-HuBERT and Auto-AVSR.It is designed to make multi-condition evaluation easier and more comparable.Source: B. Preprocessing Pipeline · pp. 2–3
  • The pipeline addresses a practical barrier to multi-condition evaluation: substantial undocumented engineering required to adapt new datasets to established frameworks.The released documentation describes dataset-specific cleaning steps.Source: B. Preprocessing Pipeline · pp. 2–3
  • The pipeline applies consistent face detection, 96×96 mouth-region extraction, 16 kHz mono audio standardization, and transcript normalization.RetinaFace and 68-point landmarks are used for mouth-region extraction.Source: B. Preprocessing Pipeline · p. 2
  • GRID transcripts are generated from its fixed six-word vocabulary, while RoomReader disfluency markers are removed before manifest generation.The resulting manifests support the two target frameworks.Source: B. Preprocessing Pipeline · pp. 2–3

III. AVSR MODELS

The study evaluates three modern AVSR architectures under identical decoding and WER protocols across visual-only, audio-only, and audio-visual modalities. The models represent supervised, self-supervised, and LLM-decoder approaches.

III. AVSR MODELS

  • Three pretrained AVSR models are evaluated with fixed official decoding settings, using WER across visual-only, audio-only, and audio-visual conditions.The protocol uses each model’s official pretrained checkpoint.Source: §III · p. 3
  • Auto-AVSR represents large-scale supervised learning with a ResNet-18 visual frontend, Conformer encoder, and hybrid CTC/attention decoder.The main checkpoint uses 3,448 hours of audiovisual data.Source: §III · p. 3
  • AV-HuBERT Large represents self-supervised multimodal pretraining followed by LRS3 fine-tuning with a shared Transformer.The evaluated checkpoint uses noise augmentation.Source: §III · p. 3
  • Llama-AVSR combines AV-HuBERT visual features and Whisper acoustic features with a Llama3.1-8B decoder fine-tuned using LoRA.Fine-tuning uses LRS3 and VoxCeleb2.Source: §III · p. 3

IV. EXPERIMENT AND RESULTS

On LRS2 and LRS3, audio-visual recognition remains near audio-only performance, while visual-only recognition is substantially weaker, especially outside matched training conditions.

B. Structured Read Speech: GRID

On the unseen fixed-grammar GRID corpus, visual recognition collapses and audiovisual fusion can underperform audio-only recognition, unlike the LombardGrid condition where hyper-articulation helps.

B. Structured Read Speech: GRID

C. Lombard Speech Under Noise: LombardGrid

D. Camera Angle and Speaker Articulation: TCD-TIMIT

TCD-TIMIT separates camera-angle effects from speaker-articulation effects by comparing frontal and semi-profile recordings across professional lipspeakers and volunteers.

D. Camera Angle and Speaker Articulation: TCD-TIMIT

1) Effect of Camera Angle in TCD-TIMIT:

Across TCD-TIMIT, moderate camera changes from 0° to 30° produce little WER variation, indicating tolerance to small frontal offsets.

1) Effect of Camera Angle in TCD-TIMIT:

2) Speaker Articulation in TCD-TIMIT:

TCD-TIMIT results show that professional articulation strongly improves visual recognition, while moderate camera offsets have little effect. In spontaneous RoomReader-AV speech, visual performance collapses and short utterances are especially difficult.

Articulation quality

  • 25.91% versus 45.57% WER: Auto-AVSR performs substantially better visually on professional lipspeakers than matched volunteers.At 0°, the matched-volunteer gap demonstrates the importance of articulation quality for visual recognition.Source: 2) Speaker Articulation in TCD-TIMIT · p. 5

RoomReader-AV

F. Scaling and Generalization in Visual Speech Recognition

Scaling broadcast training data improves visual recognition in structured settings but does not resolve out-of-domain failure. Across conditions, fusion, pose, articulation, and decoder choice reveal that visual robustness remains limited and LLM-based decoding adds deployment costs.

Scaling and generalization

  • Doubling Auto-AVSR training data from 1.7k to 3.4k hours improves most datasets but leaves RoomReader-AV nearly unchanged.The result indicates that scaling broadcast data does not remove the out-of-domain visual bottleneck.Source: F. Scaling and Generalization in… · p. 6

Generalization

  • 66–313% WER out-of-domain versus 19–44% in-domain shows that video-only recognition fails to generalize beyond broadcast speech.A 19% LRS3 VO WER can coexist with failure on every other tested condition.Source: §V · p. 6

Multimodal fusion

  • AV fusion mainly improves in-domain and LombardGrid recognition; elsewhere it offers little benefit or worsens AO performance.The discussion attributes this behavior to fusion mechanisms that continue trusting unreliable visual features.Source: §V · p. 6

Articulation and pose

  • At 90° profile views, AV performance is maintained primarily through audio fallback after visual features become uninformative.A 30° offset remains within the encoder’s frontal range, unlike the extreme profile condition.Source: §V · p. 6
  • A 17% to 44% articulation gap is much larger than the minimal 2% change from a 30° camera offset.This comparison places speaker articulation above moderate camera position as a determinant of visual performance.Source: §V · p. 6

LLM-based decoders

  • Llama-AVSR requires 17.91 GB of GPU memory for AV inference versus 2.29 GB for Auto-AVSR, alongside unstable out-of-domain outputs.The reported overhead and slow autoregressive generation limit suitability for low-latency real-time use.Source: §V · p. 6

VI. CONCLUSION AND FUTURE WORK

The paper finds that current AVSR progress is narrow: visual-only systems fail beyond broadcast speech, apparent pose robustness often reflects acoustic fallback, and LLMs introduce domain-shift challenges. It responds with RoomReader-AV and a unified preprocessing pipeline for broader evaluation.

Conclusion

  • Visual-only AVSR fails to generalize outside broadcast speech across architectures and training scales.The conclusion frames standard LRS2 and LRS3 evaluation as insufficient for characterizing real-world generalization.Source: §VI · p. 6

Future work

  • The paper introduces RoomReader-AV for spontaneous multiparty conversational AVSR and releases a pipeline supporting evaluation across multiple datasets and frameworks.The pipeline also simplifies extension to newer ASR, VSR, and AVSR systems.Source: §VI · p. 6
Loading 2609.10366v1…