Source-linked AI summary
Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
Umberto Cappellazzo, Xubo Liu, Pingchuan Ma, Stavros Petridis, Maja Pantic
TL;DR
Existing LLM-based approaches often handle ASR, VSR, and AVSR with separate models and fixed compression rates, limiting cross-task sharing and flexibility. Omni-AVSR brings the tasks and elastic inference into one model through efficient multi-granularity training and LoRA adaptation. On LRS2 and LRS3, it matches or surpasses state-of-the-art baselines while reducing resource use.
Problem
Many LLM-based approaches train separate models for ASR, VSR, and AVSR, increasing resource use and overlooking cross-task synergies; fixed-rate compression also limits flexibility.
Source: §I · p. 1, §II · p. 2Method
Omni-AVSR unifies ASR, VSR, and AVSR with elastic inference, using efficient matryoshka-based multi-granularity training and LoRA adaptation.
Source: §I · p. 1Results
Omni-AVSR matches or surpasses state-of-the-art baselines on LRS2 and LRS3; Omni-AVSR-S and -T achieve average WERs as low as 7.9 across tasks on LRS3.
Source: §III · p. 3, §IV · p. 4Takeaways & Limitations
A single model supports joint tasks and elastic inference while reducing training and deployment resource use.
Source: §IV · p. 4
Abstract
from arXiv · showhide
Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech Recognition (AVSR). Despite this progress, current LLM-based approaches typically address each task independently, training separate models that raise computational and deployment resource use while missing potential cross-task synergies. They also rely on fixed-rate token compression, which restricts flexibility in balancing accuracy with efficiency. These limitations highlight the need for a unified framework that can support ASR, VSR, and AVSR while enabling elastic inference. To this end, we present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation. Specifically, we adapt the matryoshka representation learning paradigm to efficiently train across multiple audio and visual granularities, reducing its inherent training resource use. Furthermore, we explore three LoRA-based strategies for adapting the backbone LLM, balancing shared and task-specific specialization. Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines while training a single model at substantially lower training and deployment resource use. The model also remains robust under acoustic noise, and we analyze its scaling behavior as LLM size increases, providing insights into the trade-off between performance and efficiency.
I. INTRODUCTION
Prior LLM-based speech-recognition systems often train separate models for ASR, VSR, and AVSR, increasing resource use while missing potential cross-task synergies. Omni-AVSR unifies these tasks and elastic inference in one framework, with evaluation on LRS2 and LRS3 showing comparable or superior WER results across all three tasks.
I. INTRODUCTION
- AVSR incorporates visual cues such as lip movements that remain unaffected by acoustic noise, addressing a weakness of audio-only recognition.The introduction also situates recent progress in Transformers and multimodal learning.Source: §I · p. 1
- Separate task-specific models increase computational cost and complexity while overlooking potential cross-task synergies.Prior unified approaches either use cumbersome student-teacher pseudo-labeling or underperform task-specific models.Source: §I · p. 1
- Omni-AVSR jointly supports ASR, VSR, and AVSR with elastic inference under a single set of weights.It combines efficient multi-granularity training with three LoRA-based adaptation strategies.Source: §I · p. 1
- On LRS2 and LRS3, Omni-AVSR achieves comparable or superior WER results across all three tasks while substantially reducing training and deployment resource use.The paper also reports competitive performance in clean and noisy conditions and analyzes scaling with LLM size.Source: §I · p. 1
II. OMNI-AVSR
Omni-AVSR combines modality-specific encoders, rate-controlled token compression, and an LLM backbone to support ASR, VSR, and AVSR with flexible granularity. Its training procedure limits the LLM to three passes per iteration, while LoRA variants explore shared and task-specific adaptation.
II. OMNI-AVSR
- Pre-trained audio and video encoders feed projected tokens to an LLM backbone, with selected compression rates controlling audio-visual granularity.The framework supports ASR, VSR, and AVSR within one model and one set of weights.Source: §II · p. 1, Figure 1 · p. 2
II-A. Multi-Granularity via Efficient Matryoshka Training
- Fixed-rate compression limits adaptability, while finer-grained tokens improve recognition accuracy at higher inference compute cost.Matryoshka training uses multiple audio and video rates to retain flexible granularity at inference.Source: §II · p. 2
- Randomly selecting one audio rate and one video rate per iteration reduces LLM forward/backward passes to three, one per task.This replaces CA + CV + CA·CV passes while preserving flexible inference granularity.Source: §II · p. 2
II-B. Joint ASR-VSR-AVSR Training Formulation
- Training averages autoregressive next-token prediction losses for ASR, VSR, and AVSR, weighted by task-specific coefficients.Each task loss is the negative log probability of the target transcription given its task-specific input.Source: §II · p. 2
II-C. Efficient LLM Adaptation via Omni-LoRA
- With the pre-trained LLM frozen, Omni-LoRA-S shares one module across tasks, Omni-LoRA-T uses task-specific modules, and Omni-LoRA-ST combines both.The variants compare parameter sharing with task specialization; task-specific modules are selectively used at inference.Source: §II · pp. 2–3
III. EXPERIMENTS AND RESULTS
On LRS2 and LRS3, Omni-AVSR variants match or surpass multi-task baselines across ASR, VSR, and AVSR, while one model supports multiple tasks and granularities. The experiments also report reduced training computation, noise robustness, and a scaling trade-off between accuracy and efficiency.
Experiment Settings
- Experiments use LRS2 (225 hours) and LRS3 (433 hours), with ASR rates {4,16}, VSR rates {2,5}, and their four AVSR combinations.The compression rates capture a range of inference efficiency–performance trade-offs.Source: §III · p. 3
Results under Acoustic Noise
- Across noise levels, Omni-AVSR-ST outperforms Llama-AVSR and Llama-MTSK and remains competitive with Llama-MT, often surpassing it at lower SNRs.The comparison evaluates AVSR on LRS3 with babble noise from NOISEX.Source: Table III · p. 3, §III · p. 4
Comparison with Other Multi-task Methods
- At (4,2), Omni-AVSR-ST performs best across all tasks while using significantly fewer parameters; at (16,5), it remains competitive within one set of weights.The comparison includes u-HuBERT and AV-CPL, each trained on 1759 hours of LRS3 and VoxCeleb2 data.Source: Table IV · p. 3, §III · p. 4
Main results
- On LRS3, Omni-AVSR-S and -T attain average WERs as low as 7.9 across tasks and match or surpass multi-task baselines.On LRS2, Omni-AVSR-T performs best among the proposed variants; on LRS3, all three variants yield comparable results.Source: §III · p. 3
AVSR Comparison with Sota Methods
- On LRS3 AVSR at (4,2), Omni-AVSR-ST achieves competitive WERs with substantially fewer parameters and training-data hours than the compared state-of-the-art methods.The comparison also tracks scaling trends as the LLM size increases.Source: Figure 2 · p. 4, §III · p. 4
Computational cost
- Omni-AVSR trains one model and reduces LLM training passes to T, versus rate- and task-multiplied passes for the baselines.It samples one audio and one video rate per iteration and computes the loss once per task.Source: §III · p. 4
Optimal Task-specific Weights
- On LRS2, the best reported task weights are λASR = λAVSR = 1 and λVSR = 1.5.The authors state that giving the more challenging VSR task a higher weight improves overall results.Source: Table V · p. 4, §III · p. 4
LLM Scaling Trend
- As LLM size increases, performance improves, especially on VSR and highly compressed ASR, but training computation, memory use, and inference time also increase.The paper identifies 1–3B-parameter LLMs as a favorable accuracy–efficiency trade-off.Source: §III · p. 4
IV. CONCLUSION
Omni-AVSR combines unified ASR, VSR, and AVSR with elastic inference under one set of weights. Experiments report competitive performance, noise robustness, computational savings, and favorable trade-offs as LLM size scales.
Unified framework
- Omni-AVSR jointly supports ASR, VSR, and AVSR with elastic inference under a single set of weights.It combines matryoshka-based multi-granularity training with LoRA adaptation strategies.Source: §IV · p. 4
Conclusions
- On LRS2 and LRS3, Omni-AVSR matches or surpasses state-of-the-art baselines, remains robust in noisy conditions, and offers favorable LLM-scaling trade-offs.The conclusion also reports one model and fewer LLM passes during training as sources of computational savings.Source: §IV · p. 4
Efficiency
- Omni-AVSR reduces training and deployment resource use while requiring only one model and fewer LLM passes during training.Source: §IV · p. 4