Source-linked AI summary
Efficient Streaming LLM for Speech Recognition
Junteng Jia, Gil Keren, Wei Zhou, Egor Lakomkin, Xiaohui Zhang, Chunyang Wu, Frank Seide, Jay Mahadeokar, Ozlem Kalinli
TL;DR
Existing SpeechLLMs extrapolate poorly beyond training audio lengths, have quadratic total inference cost, and delay transcription until a full utterance has arrived. SpeechLLM-XL processes audio chunk by chunk, generates each chunk’s transcript autoregressively until EOS, and uses CTC forced alignment when reference alignment is unavailable. SpeechLLM-XL gives accuracy comparable to non-streaming SpeechLLM and extrapolates perfectly beyond training length, with slightly lower WER on 2x/10x concatenated test-other utterances.
Problem
Existing SpeechLLMs extrapolate poorly beyond training audio lengths, have quadratic total inference cost, and delay transcription until a full utterance has arrived.
Source: §I · p. 1Method
SpeechLLM-XL processes audio chunk by chunk, generates each chunk’s transcript autoregressively until EOS, and uses CTC forced alignment when reference alignment is unavailable.
Source: §I · p. 1, §III · p. 2Results
SpeechLLM-XL gives accuracy comparable to non-streaming SpeechLLM and extrapolates perfectly beyond training length, with slightly lower WER on 2x/10x concatenated test-other utterances.
Source: §I · p. 2, D. Length Extrapolation · p. 4Takeaways & Limitations
A small LLM attention context can reduce inference scaling from quadratic to linear without considerable quality degradation, while removing all prior chunks from context degrades quality.
Source: B. LLM Context · p. 3
Abstract
from arXiv · showhide
Recent works have shown that prompting large language models with audio encodings can unlock speech recognition capabilities. However, existing techniques do not scale efficiently, especially while handling long form streaming audio inputs -- not only do they extrapolate poorly beyond the audio length seen during training, but they are also computationally inefficient due to the quadratic cost of attention. In this work, we introduce SpeechLLM-XL, a linear scaling decoder-only model for streaming speech recognition. We process audios in configurable chunks using limited attention window for reduced computation, and the text tokens for each audio chunk are generated auto-regressively until an EOS is predicted. During training, the transcript is segmented into chunks, using a CTC forced alignment estimated from encoder output. SpeechLLM-XL with 1.28 seconds chunk size achieves 2.7%/6.7% WER on LibriSpeech test clean/other, and it shows no quality degradation on long form utterances 10x longer than the training utterances.
I. INTRODUCTION
SpeechLLMs face length-extrapolation, computational-cost, and latency challenges on long-form audio. SpeechLLM-XL processes audio in chunks with limited attention and reports comparable accuracy at a 1.28-second chunk size, no quality degradation on long concatenated utterances, and favorable comparisons with streaming baselines.
I. INTRODUCTION
- SpeechLLMs extrapolate poorly beyond training audio lengths, incur quadratic total inference cost, and produce higher user-perceived latency by waiting for a full utterance before transcription.The paper links early termination on long utterances to the models’ tendency to predict EOS after transcript lengths exceeding those seen in training.Source: §I · p. 1
- SpeechLLM-XL processes audio in static-length chunks, generating each chunk’s transcript autoregressively until EOS before using the next chunk.Previous audio chunks and decoded tokens provide context for later chunks; chunk size and LLM context size trade off latency, accuracy, and computation.Source: §I · p. 1
- When reference audio-text alignment is unavailable, an auxiliary CTC loss and forced aligner align encoder outputs with transcript tokens for training.The authors report that training with CTC forced alignment gives similar accuracy to training with reference alignment.Source: §I · p. 1
- The model interleaves chunked audio encodings and aligned text embeddings for next-token prediction, with EOS marking each chunk’s end.Its limited attention window lets each chunk attend to the current and a bounded number of prior chunks.Source: Figure 1 · p. 2
- SpeechLLM-XL with 1.28-second chunks gives accuracy comparable to non-streaming SpeechLLM, and compares favorably with streaming baselines at similar latency.The study also reports no quality degradation on concatenated long-form utterances.Source: §I · pp. 1–2
II. BACKGROUND ON SPEECHLLM
SpeechLLM maps an audio feature sequence to a text-token transcription by estimating the transcription’s conditional distribution given the audio. Its training objective covers the transcript extended with an EOS token, and inference generates the extended transcript autoregressively.
II. BACKGROUND ON SPEECHLLM
- Given audio features X and a text-token sequence y, SpeechLLM estimates the conditional distribution of the transcription given the audio.The input is a sequence of Log-Mel features, and the output is a sequence of text tokens.Source: §II · p. 2
- The transcript is extended with an EOS token before its conditional probability is rewritten.EOS marks the end of the generated transcript.Source: §II · p. 2
- During inference, the extended transcript is generated autoregressively using beam search, and EOS is removed from the final transcript.Source: §II · p. 2
III. SPEECHLLM-XL
SpeechLLM-XL segments audio and transcripts into aligned chunks, then predicts each chunk using its audio and a bounded history of earlier chunks. CTC forced alignment supplies token end times when reference alignment is unavailable.
III. SPEECHLLM-XL
- Audio and transcript are segmented into chunks of size c, with each transcript chunk corresponding to an audio chunk under known alignment.The transcript is represented as chunk text followed by an EOS marker.Source: §III · p. 2
- The decoder predicts each chunk using its audio and cached attention values from the previous b chunks, limiting the attention window to b × c.A small b reduces attention computation on long utterances.Source: §III · p. 2
- When reference audio-text alignment is unavailable, CTC forced alignment estimates token end times and yields similar accuracy to reference alignment.Source: §III · p. 2
- Beam search keeps all hypotheses at the same token count, including EOS, at each decoding step.Source: §III · p. 2
IV. EXPERIMENTS
Experiments benchmark SpeechLLM-XL and its ablations on LibriSpeech while keeping the encoder, decoder, and output vocabulary fixed. The setup combines a streaming Emformer encoder with a downsized Llama2 decoder.
IV. EXPERIMENTS
- The benchmark uses LibriSpeech’s 960 hours of English audiobook training data and 80-channel log-filterbank features with a 25 ms window and 10 ms frame shift.Source: §IV · p. 2
- Across experiments, the encoder, decoder, and output vocabulary are held constant to focus comparisons on SpeechLLM-XL architecture and hyperparameters.The encoder is a 20-layer streaming Emformer; the decoder is a 12-layer LLM downsized from Llama2.Source: §IV · pp. 2–3
- The encoder and decoder are pretrained separately, then trained together on LibriSpeech with cross-entropy and auxiliary CTC loss weighted 0.5.The CTC forced aligner estimates audio-text alignment unless otherwise specified.Source: §IV · p. 3
A. Chunk Size
Chunk size trades latency against accuracy: smaller chunks reduce latency but increase WER, and the study selects 1.28 seconds for subsequent experiments. Streaming SpeechLLM-XL also improves over non-streaming SpeechLLM on the 50 longest dev-clean utterances.
A. Chunk Size
- The chunk-size study fixes LLM context at 5.12 seconds and compares chunk sizes from 2.56 to 0.32 seconds against non-streaming SpeechLLM.Source: A. Chunk Size · p. 3
- On the 50 longest dev-clean utterances, SpeechLLM has total WER 16.4 and deletion rate 13.9, versus 2.2 and 0.1 for streaming model 1.Model 1 uses a 2.56-second chunk and also gives better WER on dev/test-clean than non-streaming SpeechLLM.Source: A. Chunk Size · p. 3
- Reducing chunk size lowers latency but increases WER across evaluation datasets, so 1.28 seconds is chosen as the latency–accuracy trade-off for the remaining paper.Source: A. Chunk Size · p. 3
B. LLM Context
With audio chunks fixed at 1.28 seconds, reducing LLM context from full context to 1.28 seconds causes little accuracy degradation, while removing prior chunks degrades quality. The comparison frames small context as a way to reduce attention cost without materially affecting accuracy.
B. LLM Context
- The non-streaming SpeechLLM baseline uses a 110M-parameter Conformer encoder and the same 92M-parameter LLM decoder as SpeechLLM-XL.Source: B. LLM Context · p. 3
- Reducing LLM context from full context to 1.28 seconds causes little accuracy degradation, but removing all previous chunks causes notable quality degradation.The study fixes audio chunk size at 1.28 seconds.Source: B. LLM Context · p. 3, Table II · p. 3
- The context ablation compares left-context sizes from infinity to zero, showing that some previous-chunk context remains necessary for quality.Source: B. LLM Context · p. 3
C. CTC Forced Alignment
The section evaluates CTC forced alignment against reference hybrid alignment on LibriSpeech dev-clean/other and examines its effect on ASR quality. CTC alignment is close to the reference, though hybrid-aligned training performs slightly better.
C. CTC Forced Alignment
- CTC forced alignment is compared with reference hybrid alignment using averaged token-end-time delay and absolute alignment delta on LibriSpeech dev-clean/other.The study also replicates training with reference hybrid alignment to assess the downstream ASR impact.Source: C. CTC Forced Alignment · p. 3
- CTC-aligned training slightly under-performs hybrid-aligned training, likely because CTC token ends are early by an average of 52ms.The alignments have a 63ms alignment delta.Source: C. CTC Forced Alignment · pp. 3–4, Table III · p. 4
D. Length Extrapolation
On concatenated LibriSpeech test utterances, SpeechLLM-XL extrapolates beyond training length without quality degradation, unlike the SpeechLLM baseline. The authors note that this could support short-utterance training while retaining long-form quality.
D. Length Extrapolation
- SpeechLLM-XL shows no quality degradation through 10x training length, while SpeechLLM’s WER more than triples at 2x training length.SpeechLLM-XL also yields slightly lower WER on 2x/10x concatenated test-other utterances.Source: D. Length Extrapolation · p. 4
- The extrapolation experiment repeats and concatenates test-clean/other utterances and evaluates models trained on regular LibriSpeech utterances.SpeechLLM-XL and SpeechLLM are both tested on the concatenated utterances.Source: D. Length Extrapolation · p. 4
- The authors connect length extrapolation to production robustness and to training on segmented short utterances for better batching and training efficiency.They state that model quality can be maintained on long-form utterances.Source: D. Length Extrapolation · p. 4
V. ADDITIONAL BASELINES AND RELATED WORKS
SpeechLLM-XL is compared with non-streaming and streaming ASR baselines, as well as related chunked and decoder-only methods. The comparisons report stronger results than several baselines and similar WER to Chunked AED under similar settings.
V. ADDITIONAL BASELINES AND RELATED WORKS
- The authors exclude baselines using self-supervised pretraining, pseudo labeling, or LM shallow fusion because pursuing LibriSpeech state-of-the-art WER is not this work’s purpose.Source: §V · p. 4
- SpeechLLM-XL performs notably better than the streaming CTC and Transducer models, and outperforms a model with a BiLSTM encoder and RNN decoder despite being streaming and smaller in size.The CTC model has 246M parameters and the Transducer model has 256M parameters.Source: §V · p. 4
- SpeechLLM-XL has similar WER to Chunked AED with similar decoder chunk size and encoder lookahead, while its decoder-only architecture is easier to integrate with a pretrained LLM.Chunked AED is described as a related streaming ASR model using a similar chunking idea.Source: §V · p. 4
- SpeechLLM-XL performs notably better than ReaLLM despite using a smaller encoder lookahead and decoder chunk size, and beats CTC-prompt-LM by a considerable margin.ReaLLM scales quadratically with audio length and adds one blank prediction per frame; CTC-prompt-LM uses joint CTC and decoder score fusion to determine chunk ends.Source: §V · p. 4