Source-linked AI summary
Prompting Large Language Models with Speech Recognition Abilities
Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, Mike Seltzer
TL;DR
Text-only LLMs do not directly process speech, limiting their access to information encoded in audio. The paper attaches a conformer audio encoder and prepends its embeddings to text embeddings so a decoder-only LLM can perform ASR. On MLS, the resulting system performs multilingual recognition, outperforms monolingual baselines by 18% on average WER, and remains effective with frozen LLMs or nearly one-second strides.
Problem
Text-only interaction limits LLM access to information encoded in audio, motivating methods that extend LLMs to speech recognition.
Method
A conformer audio encoder produces variable-length embeddings that are projected to the LLM dimension, prepended to text embeddings, and used by a decoder-only LLM for next-token prediction.
Results
18% lower average WER than monolingual baselines is achieved by the 80ms-stride decoder-only LLaMA system, while large-stride systems remain competitive with those baselines.
Takeaways & Limitations
Multilingual ASR is possible with an LLM trained overwhelmingly on English text, including when the LLM is frozen or audio strides approach 1 second.
Abstract
from arXiv · showhide
Large language models have proven themselves highly flexible, able to solve a wide range of generative tasks, such as abstractive summarization and open-ended question answering. In this paper we extend the capabilities of LLMs by directly attaching a small audio encoder allowing it to perform speech recognition. By directly prepending a sequence of audial embeddings to the text token embeddings, the LLM can be converted to an automatic speech recognition (ASR) system, and be used in the exact same manner as its textual counterpart. Experiments on Multilingual LibriSpeech (MLS) show that incorporating a conformer encoder into the open sourced LLaMA-7B allows it to outperform monolingual baselines by 18% and perform multilingual speech recognition despite LLaMA being trained overwhelmingly on English text. Furthermore, we perform ablation studies to investigate whether the LLM can be completely frozen during training to maintain its original capabilities, scaling up the audio encoder, and increasing the audio encoder striding to generate fewer embeddings. The results from these studies show that multilingual ASR is possible even when the LLM is frozen or when strides of almost 1 second are used in the audio encoder opening up the possibility for LLMs to operate on long-form audio.
1 Introduction
LLMs are flexible across text-based generative tasks, but text-only interaction cannot capture all information conveyed by modalities such as audio. This paper investigates adding speech recognition to an LLM while addressing the adaptation trade-off between efficiency and performance.
- LLMs learn broad world knowledge from unsupervised text and support tasks including summarization, question answering, retrieval, generation, and translation.
- Audio and images encode information that can be difficult to capture through text, motivating extensions of LLMs to other modalities.
- Multimodal systems have combined pretrained modality-specific encoders with LLMs to perform tasks such as robotics.
- Adapting very large multimodal systems can be computationally expensive because of their parameter counts.
- The paper equips an LLM with speech recognition by conditioning it on a variable-length sequence of audio embeddings.
- The study evaluates encoder size and frame rate, LoRA adaptation, text token masking, and LLM choice as factors affecting recognition performance.
2 Methodology
The method conditions a decoder-only LLM on audio embeddings produced by a conformer encoder, treating multimodal embedding sequences similarly to text sequences. Audio embeddings are compressed, projected into the LLM dimension, prepended to text embeddings, and used for next-token prediction under frozen or parameter-efficient LLM adaptation.
- The approach models embedding sequences irrespective of modality and uses a pretrained audio encoder to produce variable-length audial embeddings.
- A conformer audio encoder is initially trained with CTC, then consecutive outputs can be stacked to create fewer, larger embeddings.
- The experiments primarily adapt LLaMA-7B with LoRA while freezing other parameters, and also test fully frozen LLMs and BLOOM models.
- The projected audio embeddings are prepended to text embeddings and fed to the LLM for next-token prediction.
3 Experimental Evaluation
Experiments evaluate the audio-conditioned LLaMA system on multilingual LibriSpeech and test how encoder design, LLM adaptation, masking, and language-model choice affect recognition.
- 3.1 Dataset: Multilingual LibriSpeech contains 50k hours across eight languages, with substantial English dominance and low-resource Portuguese and Polish subsets.Utterances are up to 20 seconds, and lower-resource languages are oversampled during training.
- 3.2 Model: The conformer audio encoder produces embeddings that are stacked, projected to LLaMA’s 4096-dimensional space, and prepended to text embeddings.Stacking reduces sequence length and permits frame rates up to 960ms.
- 3.4 Main Results: At 80ms frame rate, decoder-only LLaMA outperforms monolingual models by 18% and 10% on average word error rate; 480/960ms strides remain competitive with monolingual baselines.The high-striding systems reduce the original filterbank sequence by up to a factor of 96.
- 3.5 Ablation Studies: A 36-layer encoder with 240ms striding reaches 9.7% average WER, matching an 18-layer encoder with 80ms striding.This result highlights the effect of encoder size and striding on embedding quality.
- 3.5 Ablation Studies: Freezing LLaMA yields 10.9% average WER, while rank R = 32 reaches 9.5% average WER through stronger low-rank adaptation.Increasing the trainable rank improves performance, whereas fully tuning the LLM would be more expensive.
- 3.5 Ablation Studies: Masking 25.0% of text tokens improves average WER by 5.7% over no masking, but heavier masking harms Portuguese and Polish.The authors leave language-specific masking levels for future work.
- 3.5 Ablation Studies: LLaMA-7B and similarly sized BLOOM-7B1 show no significant difference in average WER, while scaling the LLM improves performance with a fixed conformer encoder.The results suggest multilingual pretraining is less impactful after multilingual speech training, whereas LLM scale remains beneficial.
4 Analysing Audio Encoder Text Alignment
The analysis tests whether audio embeddings align monotonically with text embeddings and finds that larger strides make this alignment increasingly difficult.
- Audio and text embeddings should be monotonically aligned when the encoder places them in the same semantic space.The ASR task is framed as cleaning and repeating information from the audio sequence.
- Cosine similarity is computed between every audio–text embedding pair for an English test example.The analysis uses the LLaMA models with different encoder strides.
- The alignment plots support monotonic alignment between audio and text embeddings.
- As striding increases, aligning audio embeddings to text becomes harder.
- Strides from 80ms to 960ms are compared across the Figure 3 subfigures.
5 Conclusion
The paper presents a simple procedure for multilingual speech recognition with a decoder-only language model conditioned on audio embeddings. It also examines encoder and language-model factors affecting recognition and observes alignment between audio and text embeddings.
- Prepending an audio embedding sequence enables a large language model to perform multilingual speech recognition in a decoder-only fashion.
- The study analyzes audio-encoder stride and size, LLaMA versus BLOOM, low-rank adaptation, and input masking.
- After joint training, audio embeddings tend to align with text embeddings.
- Future work could directly train the audio encoder for alignment with the language model.