Source-linked AI summary
SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translation
Zhehuai Chen, He Huang, Andrei Andrusenko, Oleksii Hrinchuk, Krishna C. Puvvada, Jason Li, Subhankar Ghosh, Jagadeesh Balam, Boris Ginsburg
TL;DR
SALM addresses the challenge of bringing LLM multitask and in-context learning to speech tasks. It combines speech inputs with a frozen LLM and task instructions, achieving parity with bespoke ASR and AST baselines while demonstrating zero-shot keyword boosting.
Problem
The work targets bringing LLM multitask and in-context learning capabilities to speech tasks, including speech understanding.
Source: §1 · p. 1, §3.3 · p. 2Method
SALM combines an audio encoder and modality adapter with a frozen text LLM and LoRA layers, conditioning on speech prompts, text context, and instructions.
Source: §3.1 · p. 2Results
Takeaways & Limitations
The authors open-source the implementation via NeMo to accelerate research on speech-LLM modeling.
Source: §6 · p. 4SALM remains prone to hallucination, long-form deletion, and early stopping.
Source: §5.1 · p. 3, §5.2 · p. 4
Abstract
from arXiv · showhide
We present a novel Speech Augmented Language Model (SALM) with {\em multitask} and {\em in-context} learning capabilities. SALM comprises a frozen text LLM, a audio encoder, a modality adapter module, and LoRA layers to accommodate speech input and associated task instructions. The unified SALM not only achieves performance on par with task-specific Conformer baselines for Automatic Speech Recognition (ASR) and Speech Translation (AST), but also exhibits zero-shot in-context learning capabilities, demonstrated through keyword-boosting task for ASR and AST. Moreover, {\em speech supervised in-context training} is proposed to bridge the gap between LLM training and downstream speech tasks, which further boosts the in-context learning ability of speech-to-text models. Proposed model is open-sourced via NeMo toolkit.
1. INTRODUCTION
The paper applies LLM multitask and in-context learning capabilities to speech tasks. It presents SALM as a unified model for ASR and AST, with zero-shot keyword boosting and speech-supervised in-context training.
1. INTRODUCTION
- SALM unifies speech-language modeling, performing on par with bespoke Conformer baselines in ASR and AST.The implementation is open-sourced via NeMo [15].Source: §1 · p. 1
- The work equips speech-to-text models with zero-shot in-context learning, demonstrated through keyword boosting for ASR and AST.Source: §1 · p. 1
- Speech-supervised in-context training is proposed to further boost speech models' in-context learning ability.Source: §1 · p. 1
2. RELATED WORK
The paper situates SALM among speech–LLM interfaces and contextual speech-recognition approaches, emphasizing speech-to-text in-context learning as its focus.
2. RELATED WORK
- Prior speech–LLM approaches use text interfaces, which may lose speaker information, emotion, and accents; Speech-LLaMA [11] [12] prepends continuous audio embeddings to text embeddings before a decoder-only LLM.Source: §2 · p. 1
- SALM extends speech-to-text models with in-context learning, an under-explored direction compared with prior speech in-context work focused on text-to-speech [17, 18].The implementation is to be open-sourced to accelerate research in this area.Source: §2 · p. 1
- Keyword boosting has used external keyword LMs and WFSTs or contextual information integrated into end-to-end modeling (CLAS) [20].SALM studies LLM in-context learning for keyword boosting without external biasing graphs or explicit boosted-word weights.Source: §2 · p. 1
3. SALM - SPEECH AUGMENTED LANGUAGE MODEL
SALM uses supervised speech instruction tuning to make an LLM condition on speech, text context, and task instructions, with separate training strategies for multitask learning and in-context learning.
3.1. Speech and text prompts
- A modality adapter of two Conformer layers with 4X subsampling creates speech prompts, which are projected and prepended to text context and instructions.The LLM is frozen while LoRA layers and the remaining components are trained.Source: §3.1 · p. 2
3.2. Multitask supervised speech instruction tuning
- Multitask speech instruction tuning uses paired ASR and AST data with diverse, randomly prepended task instructions to train the shared adapter and encourage instruction following.Source: §3.2 · p. 2
3.3. In-context learning for speech-to-text tasks
- For speech-to-text in-context learning, SALM learns boosted words from optional text context without back-propagation.Keywords are placed before the text instruction; prior approaches may require explicit weights or external biasing graphs.Source: §3.3 · p. 2
3.4. Speech supervised in-context training
- Speech ICT augments supervised speech instruction tuning with optional text context sampled from words in the current and other utterances.The training data includes that context with a 5% probability, and the sampling uses a positive ratio of ground-truth words.Source: §3.4 · p. 2, Figure 3 · p. 2
4. DATA AND EXPERIMENTAL SETUP
The experiments evaluate SALM on LibriSpeech ASR, MuST-C speech translation, and keyword boosting using an internal GTC-talks test set.
4. DATA AND EXPERIMENTAL SETUP
- The SALM setup uses a 2B-parameter Megatron LLM and a pretrained audio encoder, with the modality adapter randomly initialized.The LLM was trained on 1.1T tokens and then fine-tuned on public instruction-following datasets.Source: §4 · p. 2
- ASR training uses LibriSpeech, selects checkpoints by dev-set WER, and evaluates on test-clean and test-other.The baseline uses an NGC ASR-pretrained Fast Conformer-large encoder and transducer decoder.Source: §4 · p. 3
- Speech translation training uses 2.7M segments, or 4.8K audio hours, with pseudo-generated German and Japanese translations; evaluation uses MuST-C v2 tst-COMMON.The baseline uses a Fast Conformer-large encoder and a 6-layer Transformer decoder.Source: §4 · p. 3
- Keyword boosting is evaluated on an 8-hour aligned and segmented internal set of NVIDIA GTC talks, with 64 default keywords.The set contains many acronyms, product names, and technical terms; evaluation reports keyword precision, recall, and F-score.Source: §4 · p. 3
5. RESULTS AND ANALYSIS
SALM combines ASR and AST, with the unified model outperforming the two task-specific baselines and AST-only SALM, though it trails ASR-only SALM on ASR. Keyword prompting supports zero-shot boosting in ASR and AST, while Speech ICT and dictionary-guided translation are further explored.
5.1. Unified model for ASR and AST
- The unified SALM performs better than the two task-specific baselines and AST-only SALM, but worse than ASR-only SALM on ASR.The best ASR/AST SALM variant with nucleus sampling also beats the from-scratch baseline but remains behind the stronger pretrained baseline.Source: §5.1 · p. 3
5.2. Zero-shot in-context Learning for keyword boosting
- SALM with prompted keywords improves on unprompted SALM, with nucleus sampling further boosting ASR keyword performance and gains similar to baseline boosting.The SALM approach does not require external biasing graphs or explicit learned biasing embeddings.Source: §5.2 · pp. 3–4
- Speech ICT improves keyword-boosting performance, while a larger positive ratio trades precision for recall; the authors use 6%.Without Speech ICT, SALM follows prompts ineffectively and yields limited improvement.Source: §5.2 · p. 4, Table 4 · p. 4
- As the number of boosted words grows, SALM and baseline boosting both lose precision while recall stays almost unchanged.The passage attributes this to more false accepts as candidate words increase and suggests better long-context handling as a possible remedy.Source: §5.2 · p. 4, Figure 4 · p. 4
- SALM tends to help shorter and compound words and text normalization, while FC-T boosting handles longer words and phrases better.SALM also has hallucination and early-stopping problems seldom seen in the baseline.Source: §5.2 · p. 4, Table 5 · p. 4
5.3. In-context learning for dictionary-guided translation
- In a German AST keyword-boosting study, overall F-score gains are moderate, though examples show natural translations that correctly use boosted words.The study selects 40 German words from MuST-C EN-DE dev references that occur less often in hypotheses.Source: §5.3 · p. 4, Figure 5 · p. 4, Table 6 · p. 4
6. CONCLUSION
SALM uses a unified LLM framework for speech tasks and adds in-context learning to speech-to-text models. It performs on par with bespoke ASR and AST baselines, and speech-supervised in-context training further improves in-context learning.
6. CONCLUSION
- SALM performs on par with bespoke ASR and AST baselines while providing a unified model for multiple speech tasks.Source: §6 · p. 4
- The work defines and studies in-context learning for speech-to-text models and reports improving it through speech-supervised in-context training.Source: §6 · p. 4
- The implementation is open-sourced, with future work targeting hallucination, deletion, and long-context issues in LLM-based SALM.Source: §6 · p. 4