Source-linked AI summary
SALMONN: Towards Generic Hearing Abilities for Large Language Models
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Chao Zhang
TL;DR
General hearing for LLMs requires understanding speech, audio events, and music rather than only text or a single audio modality. SALMONN integrates a text LLM with speech and audio encoders, achieves competitive trained-task performance, and exhibits unseen cross-modal abilities after activation tuning, while task over-fitting remains a documented limitation.
Problem
The paper addresses the need for LLMs to perceive and understand general auditory information spanning speech, audio events, and music.
Method
SALMONN integrates a pre-trained text-based LLM with Whisper and BEATs speech/audio encoders, then uses activation tuning to recover cross-modal emergent abilities.
Results
SALMONN achieves competitive performance on trained speech, audio, and music tasks and demonstrates untrained abilities including slot filling, storytelling, and speech audio co-reasoning.
Takeaways & Limitations
SALMONN demonstrates a promising direction toward generic hearing abilities for LLMs across trained and untrained auditory understanding tasks.
Abstract
from arXiv · showhide
Hearing is arguably an essential ability of artificial intelligence (AI) agents in the physical world, which refers to the perception and understanding of general auditory information consisting of at least three types of sounds: speech, audio events, and music. In this paper, we propose SALMONN, a speech audio language music open neural network, built by integrating a pre-trained text-based large language model (LLM) with speech and audio encoders into a single multimodal model. SALMONN enables the LLM to directly process and understand general audio inputs and achieve competitive performances on a number of speech and audio tasks used in training, such as automatic speech recognition and translation, auditory-information-based question answering, emotion recognition, speaker verification, and music and audio captioning etc. SALMONN also has a diverse set of emergent abilities unseen in the training, which includes but is not limited to speech translation to untrained languages, speech-based slot filling, spoken-query-based question answering, audio-based storytelling, and speech audio co-reasoning etc. The presence of cross-modal emergent abilities is studied, and a novel few-shot activation tuning approach is proposed to activate such abilities. To our knowledge, SALMONN is the first model of its type and can be regarded as a step towards AI with generic hearing abilities. The source code, model checkpoints and data are available at https://github.com/bytedance/SALMONN.
1 INTRODUCTION
SALMONN connects a text-based LLM with speech and audio encoders to understand speech, audio events, and music in one multimodal model. It targets trained audio tasks while addressing task over-fitting that suppresses cross-modal abilities on unseen tasks.
- SALMONN is a single audio-text multimodal LLM designed to perceive and understand speech, audio events, and music.
- Its architecture combines a Whisper speech encoder, a BEATs audio encoder, a window-level Q-Former, and LoRA adaptation of Vicuna.The Q-Former converts variable-length encoder outputs into augmented audio tokens aligned with the LLM input space.
- Instruction tuning can produce competitive performance on trained speech, audio, and music tasks but cause task over-fitting on unseen cross-modal abilities.
- The evaluation spans three levels, from eight instruction-tuned tasks to five untrained speech-based NLP tasks and multimodal tasks such as storytelling and co-reasoning.
- Activation tuning is proposed to activate cross-modal emergent abilities and alleviate catastrophic forgetting of tasks seen during training.
2 RELATED WORK
Prior multimodal systems commonly connect LLMs to individual modalities or use task-defined pipelines for audio. SALMONN instead provides an end-to-end audio-text model with cross-modal emergent abilities for open-ended tasks.
- Prior work extends LLMs with encoders for individual modalities, including speech, audio events, images, and video, or combines multiple input types.
- Audio-event systems often process fixed-sized spectrogram images and do not explicitly model temporal correlations, limiting their ability to handle speech.
- AudioGPT supports speech, audio events, and music through a pipeline of other models and predefined tasks.
- Unlike AudioGPT, SALMONN is an end-to-end model with cross-modal emergent abilities for open-ended tasks.
3 METHODOLOGY
SALMONN combines complementary speech and non-speech audio encoders with a frozen language model through window-level alignment and trainable adapters. Its three-stage training procedure addresses task over-fitting by adding activation tuning to recover untrained cross-modal abilities while retaining trained-task performance.
- Model Architecture: SALMONN feeds the aligned audio tokens alongside a text instruction prompt into Vicuna, while LoRA is trainable and the Vicuna LLM and encoders remain frozen.LoRA adapts Vicuna’s query and value weight matrices to align the augmented input space with the output space.
- Model Architecture: Whisper and BEATs provide complementary speech and non-speech audio features that are synchronised and concatenated frame by frame.Both encoders operate at the same 50Hz output frame rate, producing a combined sequence for downstream processing.
- Model Architecture: A window-level Q-Former converts variable-length encoder outputs into augmented audio tokens with high temporal resolution for the LLM.The audio sequence is divided into windows, each processed with fixed trainable queries, yielding ⌈T/L⌉×N textual tokens and a variable overall token count.
- Training Method: Training proceeds through pre-training, instruction tuning, and activation tuning across speech, audio-event, and music tasks.Pre-training initializes the connection module and adapter using speech recognition and audio captioning; instruction tuning then adds supervised multimodal tasks.
- Training Method: Instruction tuning can cause task over-fitting, producing competitive trained-task results but limited untrained cross-modal abilities and irrelevant responses to novel prompts.The paper attributes this bias partly to simpler prompts, less diverse responses, and deterministic outputs in tasks such as speech recognition and audio captioning.
- Training Method: Activation tuning regularizes the intrinsic conditional language model by using longer, more diverse responses generated with a reduced LoRA scaling factor.The resulting few-shot self-supervised fine-tuning is intended to restore question-answering and storytelling abilities while retaining competitive trained-task performance.
4 EXPERIMENTAL SETUP
SALMONN is evaluated through staged training and a three-level benchmark design spanning trained capabilities, untrained speech-based NLP tasks, and newly proposed multimodal reasoning tasks. The setup combines broad audio instruction tuning with few-shot activation tuning and reports task-specific metrics alongside instruction-following and story-diversity measures.
- Model and training: SALMONN uses Whisper-Large-v2, BEATs, and Vicuna-13B, with Q-Former and LoRA connecting audio representations to the language model.The Q-Former produces 88 textual tokens for a 30-second audio input under the stated configuration.
- Model and training: Instruction tuning covers ASR, speech translation, captioning, phone and emotion recognition, music captioning, overlapped speech recognition, speaker verification, gender recognition, and audio question answering.Questions for speech, audio, and music question answering are generated from text caption labels, and answers must use the general audio input and prompt.
- Model and training: The final activation stage uses twelve audio-based stories and 12 teacher-forcing cross-entropy steps with one story sample per step.The stories are written by SALMONN from audio clips using a reduced LoRA.
- Task specifications: Speech audio co-reasoning requires understanding a spoken question, finding evidence in background audio or music, and reasoning from it to answer.Storytelling and speech audio co-reasoning are introduced as new tasks requiring end-to-end auditory perception and reasoning.
- Metrics and references: Following rate measures instruction compliance on selected complex tasks, while Story diversity counts distinct words and reflects richness rather than story quality.Table 2 also identifies Whisper, Whisper + Vicuna, and other state-of-the-art results as reference sources; SQQA and KE speech data are synthesized with commercial text-to-speech.
5 EXPERIMENTAL RESULTS
SALMONN performs competitively on trained tasks, but task over-fitting limits untrained cross-modal abilities until activation tuning or LoRA scaling-factor discounting is applied. Activation tuning improves emergent-task performance while preserving trained-task results and reduces the model’s bias toward dominant tasks.
- Full results: SALMONN produces competitive results on all level 1 tasks, whereas the untuned model barely performs level 2 and level 3 tasks.The untuned model particularly struggles with SQQA, Story, and SAC, which require multimodal interactions.
- Full results: Activation tuning considerably improves the FRs of SQQA, SF, Story, and SAC.These tasks represent untrained or emergent abilities evaluated beyond the instruction-tuning tasks.
- Activation tuning: Activation tuning needs only a few training samples and steps to activate emergent abilities while leaving ASR and PR results almost unchanged.SQQA, Story, and SAC show an emergent performance trend during tuning.
- LoRA scaling-factor discounting: When the LoRA scaling factor decreases to around 2.0, the model suddenly exhibits cross-modal reasoning abilities together with drops in %PER.This test-time intervention is presented as evidence for an intrinsic conditional LM embedded in LoRA.
- Task over-fitting analysis: During activation tuning, PPL gaps between AAC and Story/SAC responses are mitigated, and Story/SAC PPL eventually becomes lower than AAC PPL.The model can eventually perform the instructed task as its bias toward the dominant AAC task is reduced.
- Activation data: Story or QA with long answers can activate the model, but Story-based activation outperforms QA-based activation and yields a lower repeat rate.ASR with long labels and text-only Story training do not activate the model; the paper attributes QA’s weaker result possibly to less diverse answers.
6 CONCLUSION
SALMONN combines dual auditory encoders with a language model to perform trained speech, audio, and music tasks while generalising to several untrained understanding tasks. The authors present it as a step toward generic hearing abilities for LLMs.
- Conclusion: SALMONN achieves competitive performance on trained tasks including speech recognition, audio captioning, and speech translation.The conclusion also describes trained-task performance across related speech, audio, and music capabilities.
- Conclusion: SALMONN generalises to untrained tasks including slot filling, speech translation for untrained languages, and keyword extracting.These tasks extend the model beyond the tasks used in instruction tuning.
- Conclusion: The paper presents SALMONN as a step toward generic hearing abilities for LLMs.This conclusion follows the model’s reported coverage of trained and untrained auditory tasks.
7 REPRODUCIBILITY STATEMENT
The paper provides implementation materials and illustrates SALMONN across a broad range of auditory tasks. The examples span speech recognition and translation, audio captioning, reasoning, recognition, question answering, storytelling, and music-related generation and understanding.
- Reproducibility statement: The authors release training data, benchmark details, source code, and model checkpoints for reproducibility.The source code, checkpoints, and data are available through the SALMONN project GitHub page.
- Illustrative examples: The illustrated tasks include speech recognition, speech translation, phone recognition, audio captioning, coreasoning, event detection, storytelling, speaker and emotion recognition, spoken-query question answering, keyword extraction, slot filling, and music captioning.Additional music examples include music emotion recognition, lyrics writing, and titling music.
B PERFORMANCE ANALYSIS
With activation tuning2, SALMONN performs near state of the art on trained tasks, generalises across speech-grounded NLP tasks, and handles tasks unavailable to existing models, while retaining task-specific limitations.
- SALMONN performs close to state-of-the-art results on trained ASR, En2Zh, AAC, and MC tasks.
- SALMONN learns PR and OSR, which are difficult for the cascaded Vicuna + Whisper approach.
- SALMONN generalises across speech-grounded NLP tasks, outperforming Whisper+Vicuna on En2De and En2Ja.The passage attributes this advantage to avoiding error propagation and loss of non-linguistic information such as prosody.
- SALMONN tackles audio-based storytelling and SAC, tasks existing models cannot handle to the authors’ knowledge.
- Phoneme recognition remains limited because Whisper provides less fine-grained pronunciation modelling than specialised PR systems.
- OSR has relatively high WER because the Whisper ASR encoder could not perform OSR.
- SQQA performance drops when LoRA cross-modal adaptation causes the LLM to forget some text-based commonsense knowledge.
C STATISTICS OF TASK OVER-FITTING
Perplexity changes across training stages reveal that activation tuning supports untrained storytelling and SAC while preserving performance on trained tasks.
- ASR and AAC perplexity becomes very small after pre-training, revealing learned cross-modal alignment.
- PR perplexity drops after instruction tuning because LoRA learns phonemes as a new output language.
- Story and SAC perplexity decreases after instruction tuning but remains insufficient for successful task performance without LoRA removal or activation tuning.
- Activation tuning leaves ASR, AAC, and PR perplexities almost unchanged, unlike removing LoRA.This pattern is presented as evidence for activation tuning’s advantage in preserving trained-task behaviour.
D CHANGES IN PERPLEXITY DURING TRAINING
Figure 22 visualizes perplexity changes across SALMONN’s three training stages for six speech, audio, and music-related tasks.
- Figure 22 shows perplexity changes during S1 cross-modal pre-training, S2 instruction tuning, and each step S3-t of activation tuning.
- The visualization covers ASR, AAC, AST (En2Zh), OSR, Story, and SAC.
E CALCULATION OF FOLLOWING RATE (FR) OR ACCURACY (ACC) FOR SQQA, SF, STORY AND SAC TASKS
The evaluation defines task-specific following-rate, accuracy, diversity, and correctness procedures, including length thresholds, word counts, and GPT-3.5 judgments.
- For SQQA and SF, outputs with WER below 30% against the spoken question are treated as ASR rather than instruction following.
- Story outputs are capped at 200 tokens, and responses shorter than 50 words count as instruction-disobeying.
- Story diversity is measured by the number of different words in the generated story.
- For SAC, GPT-3.5 judges whether the model follows the instruction or answers correctly using the background audio caption and question.
- SALMONN generates responses from variable-length Q-Former auditory tokens paired with a text prompt in the Vicuna training template.
- GPT-3.5 is used both to generate some data and to automatically evaluate model-output quality through task-specific prompts.
- The hearing-focused model could potentially be extended to speech generation because speech production is related to auditory perception.