Source-linked AI summary
An Embarrassingly Simple Approach for LLM with Strong ASR Capacity
Ziyang Ma, Guanrou Yang, Yifan Yang, Zhifu Gao, Jiaming Wang, Zhihao Du, Fan Yu, Qian Chen, Siqi Zheng, Shiliang Zhang, Xie Chen
TL;DR
Whether complex task-specific designs are needed for LLM-based ASR remains an open question. The paper benchmarks a simple system with only a trainable linear projector and reports that it outperforms other LLM-based and previous neural ASR models on Librispeech.
Problem
The paper examines whether the complex designs used in LLM-based ASR are necessary for effective speech recognition.
Source: §1 · p. 2Method
The authors benchmark combinations of speech encoders and LLMs, then build SLAM-ASR with only a trainable linear projector.
Source: §1 · p. 2Results
SLAM-ASR outperforms other LLM-based and previous neural ASR models on the Librispeech benchmark.
Source: §6 · p. 8Takeaways & Limitations
The findings challenge the notion that greater design complexity yields better LLM-based ASR.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In this paper, we focus on solving one of the most important tasks in the field of speech processing, i.e., automatic speech recognition (ASR), with speech foundation encoders and large language models (LLM). Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task. To be more specific, we benchmark and explore various combinations of LLMs and speech encoders, leading to the optimal LLM-based ASR system, which we call SLAM-ASR. The proposed SLAM-ASR provides a clean setup and little task-specific design, where only the linear projector is trained. To the best of our knowledge, SLAM-ASR achieves the best performance on the Librispeech benchmark among LLM-based ASR models and even outperforms the latest LLM-based audio-universal model trained on massive pair data. Finally, we explore the capability emergence of LLM-based ASR in the process of modal alignment. We hope that our study can facilitate the research on extending LLM with cross-modality capacity and shed light on the LLM-based ASR community.
1 Introduction
The paper argues that strong LLM-based ASR can be achieved with a simple speech-encoder–LLM composition, and presents SLAM-ASR as evidence for this approach.
1 Introduction
- The authors argue that a powerful speech encoder, suitable LLM, and single trainable linear projector suffice, challenging the need for complex ASR designs.Prior designs include temporal compression of encoder outputs, projector-based modal alignment, and partial or full LLM fine-tuning.Source: §1 · pp. 1–2
- With only a trained linear projector and 4 GPUs for 4 hours, SLAM-ASR achieves state-of-the-art Librispeech performance among LLM-based ASR models and prior best-performing NN-based models.The authors describe the setup as clean and requiring little task-specific design.Source: §1 · p. 2
- Benchmarks show that supervised-fine-tuned chat LLMs outperform raw pretrained LLMs, while limited-data-fine-tuned self-supervised encoders outperform supervised foundation ASR encoders.These findings guide the component choices in SLAM-ASR.Source: §1 · p. 2
- The authors report capability emergence during LLM-based ASR training and explore this phenomenon in depth.They frame their benchmark and experiments as a step-by-step exploration using a clean setup with little task-specific design.Source: §1 · p. 2
2 Speech Recognition Meets Large Language Model
The section surveys established neural and LLM-based ASR approaches, then presents SLAM-ASR as a simple speech-to-LLM pipeline. Its method uses a frozen speech encoder and LLM with a trainable projector between speech features and LLM embeddings.
2.1 Previous NN-based ASR
- CTC, AED, transducer, Pruned RNN-T, and CIF exemplify neural ASR paradigms that address alignment and frame-independence limitations in different ways.Quartznet uses CTC; Whisper uses an attention-based encoder-decoder; Branchformer combines CTC and AED; Conformer uses a neural transducer; Zipformer adopts Pruned RNN-T; and Paraformer uses CIF for soft monotonic alignment.Source: §2.1 · p. 2
2.2 Existing LLM-based ASR
- Existing LLM-based ASR systems use decoder-only LLMs, but many require expensive fine-tuning of the encoder, LLM, or LoRA components.LauraGPT fully fine-tunes parameters and SpeechGPT uses multistage fine-tuning; other approaches use trainable encoders, while Fathullah et al. (2023) requires LoRA. Yu et al. (2024) instead uses a segment-level Q-Former projector and random concatenation to address Whisper’s 30-second input limit.Source: §2.2 · pp. 2–3
2.3 Proposed Method
- SLAM-ASR freezes the speech encoder and LLM, training only a linear projector to align speech features with the LLM’s input embedding dimension.It downsamples speech by concatenating each k consecutive frames along the feature dimension; the projector uses a hidden layer, ReLU, and regression layer.Source: Figure 1 · p. 3, §2.3 · p. 3
- The model input combines projected speech, prompt, and transcript embeddings during training, while inference uses speech and prompt embeddings.The transcript embedding is omitted from the inference template.Source: §2.3 · p. 3
3 Experiment Setup
The experiments follow a KISS approach, investigating key factors in LLM-based ASR with a systematic component sweep and standard LibriSpeech evaluation.
3.1.1 Speech Encoder
- The encoder sweep covers supervised Whisper variants and Qwen-Audio, plus raw or fine-tuned HuBERT and WavLM at multiple scales.Whisper encoders range from tiny to large-v2, with their decoders discarded; HuBERT and WavLM base models use 960 hours, HuBERT large uses 60,000 hours, and WavLM large uses 94,000 hours. HuBERT also includes an X-Large model; model details are referenced in Section 4.3.Source: §3.1.1 · p. 4
3.1.2 LLM
- The LLM sweep includes raw pretrained TinyLLaMA (1B) and LLaMA-2 (7B), alongside chat models TinyLLaMA-Chat (1B), Phi-2 (2B), LLaMA-2-Chat (7B), and Vicuna (7B).The paper distinguishes raw pretrained LLMs from chat LLMs with supervised fine-tuning, and RLHF where conducted.Source: §3.1.2 · p. 4
3.1.3 Projector
- The projector downsamples 50 Hz encoder outputs by k = 5 to 10 Hz and uses a 2048-dimensional hidden layer, with input and output dimensions varying by model.The projector serves as the adaptor aligning speech features with the LLM.Source: §3.1.3 · p. 4
3.2 Dataset
- The evaluation uses standard LibriSpeech with 960 hours of training data, no augmentation or splicing, devother for validation, and 10-hour test-clean and test-other sets.Source: §3.2 · p. 4
3.3 Training Detail
- Training computes loss only on transcribed text, uses AdamW with a 1 × 10−4 maximum learning rate and no weight decay, and stops by validation-loss early stopping or 100,000 steps.The schedule warms up for 1,000 steps, then holds the maximum learning rate. Whisper inputs are padded to 30 seconds with batch size 4; other encoders retain input length with batch size 6.Source: §3.3 · pp. 4–5
3.4 Inference Detail
- At inference, LLMs generate autoregressively, and beam search with beam = 4 is used for deterministic speech-recognition hypotheses.Source: §3.4 · p. 5
4 Exploration
The exploration finds that chat models and stronger speech encoders improve LLM-based ASR, then presents SLAM-ASR as a simple system that performs strongly against LLM-based and conventional ASR models. Its only trainable component is a linear projector.
4.1 A Basic Benchmark
- Across the preliminary Whisper and LLM combinations, larger speech encoders improve ASR with diminishing returns, and chat models outperform pretrained LLMs regardless of model size.The authors suggest chat-model SFT may activate a translation-like treatment of speech embeddings as “language.”Source: §4.1 · p. 5
4.2 Exploration in LLMs
- With Whisper-large fixed, Phi-2 Chat at 2.78B parameters has comparable test-other WER to 6.74B LLaMA-2, while Vicuna performs better than LLaMA-2 and LLaMA-2-Chat.These comparisons support the finding that chat models work well in LLM-based ASR.Source: §4.2 · p. 5
4.3 Exploration on Speech Encoders
- With Vicuna fixed, supervised speech-encoder performance improves as encoder size increases; at 0.3B, WavLM Large outperforms all listed supervised encoders, while similarly sized HuBERT Base and WavLM Base perform similarly to Whisper-small.HuBERT Base to HuBERT Large yields no obvious improvement. Fine-tuning HuBERT Large on 960 hours of Librispeech training data before using it as the speech encoder to train the projector yields WERs of 2.30% on test-clean and 4.53% on test-other, exceeding WavLM.Source: §4.3 · p. 5
- With Librispeech-960 fine-tuned HuBERT X-Large, the model reaches 1.94% WER on test-clean and 3.81% on test-other, reducing WER versus Whisper-large by 24.8% and 41.1%, respectively.Directly feeding 80-dimensional FBank features to the projector performs far worse than using a well-trained speech encoder.Source: §4.3 · p. 7
4.4 SLAM-ASR
- SLAM-ASR uses HuBERT X-Large, Vicuna-7B, and only a trainable linear projector; it reduces WER by 17.4/26.9% versus Yu et al. (2024) on test-clean/other with the same 960 hours, and beats audio-universal models despite their larger training data.SLAM-ASR also outperforms the contemporary system when that system is trained on over 4,000 hours of speech.Source: §4.4 · p. 7
- SLAM-ASR matches or exceeds leading Librispeech-960 specialist ASR models without their complex engineering, and outperforms Whisper-large-v2 and OWSM-v3.1 among general-purpose models.The specialist comparison includes systems with data augmentation, model averaging, and in-domain language-model fusion or rescoring.Source: §4.4 · p. 7
5 Capability Emergence
LLM-based ASR exhibits capability emergence during training, with alignment timing affected by the speech encoder, LLM size, and whether the encoder is frozen or fine-tuned.
5 Capability Emergence
- Next-token accuracy rises rapidly, then slowly, before suddenly spiking within one epoch (around 12k steps), indicating capability emergence during training.The paper describes the spike as if the ability is suddenly learned.Source: §5 · p. 7
- With Vicuna-7B fixed, stronger speech encoders, including Whisper Large and WavLM Large, emerge earlier, apparently because their representations are easier for the projector to align.This is presented as a possible explanation for the earlier emergence.Source: §5 · pp. 7–8, Figure 2 · p. 8
- With Whisper Large fixed, TinyLLaMA-Chat and Phi-2 emerge earlier than LLaMA-2-7B-Chat and Vicuna-7B, but are less effective; larger LLMs are harder to align.Source: §5 · p. 8, Figure 3 · p. 8
- With TinyLLaMA-1.1B-Chat, freezing the speech encoder reaches alignment in 1k steps versus 25K when trainable, and yields much better WER on Librispeech.Both settings quickly reach around 40% training accuracy early on; the authors conclude that 1k hours of speech is insufficient to train a task-specific encoder, favoring freezing and modal alignment.Source: §5 · p. 8, Table 7 · p. 8, Figure 4 · p. 8
6 Conclusion
SLAM-ASR combines a speech encoder and LLM using only a trainable linear projector, and outperforms other LLM-based and previous NN-based ASR models on Librispeech. The study also identifies stronger performance from supervised-finetuned LLMs and self-supervised speech encoders fine-tuned for the task, and reports capability emergence during alignment.
6 Conclusion
- SLAM-ASR outperforms other LLM-based ASR models and previous NN-based ASR models on the Librispeech benchmark.The framework uses only a trainable linear projector to align the speech encoder and LLM.Source: §6 · p. 8
- LLMs that undergo supervised fine-tuning exhibit improved performance and robustness.Source: §6 · p. 8
- Speech encoders fine-tuned from self-supervised models demonstrate superior capabilities.Source: §6 · p. 8
- Exploratory experiments show capability emergence in LLM-based ASR systems.Source: §6 · p. 8
A Appendix: More Exploration · A.1 Text Perplexity
The appendix tests whether chat models’ ASR advantage reflects transcript-domain agreement by comparing LLM text perplexity and WER on Librispeech test-clean and test-other. LLaMA-2 without SFT has the lowest perplexity but the worst WER, so the reported chat-model advantage is not attributed to domain agreement with the transcripts.
A.1 Text Perplexity
- LLaMA-2 without SFT has the lowest text perplexity by a large margin but the worst WER, while Vicuna has the best WER among the listed models.The comparison covers Librispeech test-clean and test-other; the authors conclude chat models’ better results are not due to domain agreement with the transcripts.Source: §A.1 · p. 11, Table 8 · p. 11
A.2 Prompt Engineering
Prompt design materially affects LLM-based ASR: a short prompt performs better than a long, complex prompt, while removing the prompt or varying complex prompts substantially degrades performance. The authors suggest prompts support task-specific in-context learning, but excessive complexity makes learning harder.
A.2 Prompt Engineering
- The authors suggest prompts enable task-specific in-context learning, whereas overly complex prompts increase learning difficulty and can produce a suboptimal solution.They frame this as a possible explanation for the observed prompt effects.Source: §A.2 · p. 11
- A short prompt performs better than a long, complex prompt, while removing the prompt causes performance to drop on Librispeech.The comparison covers test-clean and test-other; the passages do not provide the specific scores.Source: Table 10 · p. 11, §A.2 · p. 11
- Randomly drawing among 10 complex prompts at training and testing leads to a big performance drop.The prompts were generated from the ASR seed prompt used in SpeechGPT (Zhang et al., 2023).Source: Table 10 · p. 11, §A.2 · p. 11