PaperlayerView exact revision on arXiv

Source-linked AI summary

SLAM-LLM: A Modular, Open-Source Multimodal Large Language Model Framework and Best Practice for Speech, Language, Audio and Music Processing

Ziyang Ma, Guanrou Yang, Wenxi Chen, Zhifu Gao, Yexing Du, Xiquan Li, Zhisheng Zheng, Haina Zhu, Jianheng Zhuo, Zheshu Song, Ruiyang Xu, Tiranrui Wang, Yifan Yang, Yanqiao Zhu, Zhikang Niu, Liumeng Xue, Yinghao Ma, Ruibin Yuan, Shiliang Zhang, Kai Yu, Eng Siong Chng, Xie Chen

arXiv:2601.09385v1cs.SDcs.CLcs.MM

TL;DR

Existing multimodal frameworks provide limited native support for speech, audio, and music, leaving researchers to adapt vision- or text-oriented systems. SLAM-LLM offers a modular open-source framework with configurable encoders, projectors, LLMs, PEFT methods, recipes, and checkpoints, with reported competitive or state-of-the-art results across audio-related tasks.

  • Problem

    Most open-source multimodal frameworks are vision-centric and provide limited support for speech, audio, and music, constraining native development of audio-language models.

  • Method

    SLAM-LLM uses a modular encoder–projector–LLM framework with configurable components, PEFT strategies, training recipes, inference recipes, and pretrained checkpoints for audio-related tasks.

  • Results

    Recipes achieve competitive or state-of-the-art performance across several benchmarks, particularly for speech recognition and audio captioning, while experiments provide empirical insights.

  • Takeaways & Limitations

    The open-source framework provides a baseline and configurable foundation for developing and comparing LLM-based speech, audio, and music systems.

Abstract

from arXiv · show

The recent surge in open-source Multimodal Large Language Models (MLLM) frameworks, such as LLaVA, provides a convenient kickoff for artificial intelligence developers and researchers. However, most of the MLLM frameworks take vision as the main input modality, and provide limited in-depth support for the modality of speech, audio, and music. This situation hinders the development of audio-language models, and forces researchers to spend a lot of effort on code writing and hyperparameter tuning. We present SLAM-LLM, an open-source deep learning framework designed to train customized MLLMs, focused on speech, language, audio, and music processing. SLAM-LLM provides a modular configuration of different encoders, projectors, LLMs, and parameter-efficient fine-tuning plugins. SLAM-LLM also includes detailed training and inference recipes for mainstream tasks, along with high-performance checkpoints like LLM-based Automatic Speech Recognition (ASR), Automated Audio Captioning (AAC), and Music Captioning (MC). Some of these recipes have already reached or are nearing state-of-the-art performance, and some relevant techniques have also been accepted by academic papers. We hope SLAM-LLM will accelerate iteration, development, data engineering, and model training for researchers. We are committed to continually pushing forward audio-based MLLMs through this open-source framework, and call on the community to contribute to the LLM-based speech, audio and music processing.

I. INTRODUCTION

Existing open-source multimodal frameworks are largely vision-centric and provide limited native support for speech, audio, and music. SLAM-LLM addresses this gap with an open-source, modular framework, recipes, checkpoints, and broad component support for audio-related tasks.

  • Vision-centric multimodal frameworks offer limited support for speech, audio, and music, forcing auditory researchers to adapt systems through inefficient, fragmented workflows.
  • SLAM-LLM provides a modular encoder–projector–LLM architecture that supports customizable pretrained encoders, projection modules, LLM backbones, and PEFT strategies.Supported components include Whisper, HuBERT, BEATs, MERT, MLPs, CNNs, Q-Former, LLaMA, Vicuna, Qwen, LoRA, and prefix-tuning.
  • The framework supplies training recipes and pretrained checkpoints for ASR, S2TT, SEC, AAC, and MC.
  • SLAM-LLM contributes a unified, extensible framework and curated recipes across speech, audio, and music tasks, with reported state-of-the-art benchmark results particularly in speech recognition and audio captioning.
  • Its open-source implementation is intended to encourage community collaboration and accelerate development of audio-language models.

III. DESIGN OF SLAM-LLM

SLAM-LLM configures customized multimodal systems through YAML specifications for model, training, and data components. Its workflow illustrates an LLM-based Southeast Asian ASR setup, while its architecture separates Encoder, Projector, and LLM modules.

  • Users specify model, training, and data configurations in a YAML file to tailor SLAM-LLM training and inference to customized requirements.
  • A Southeast Asian ASR example combines Whisper Large-v3, a linear projector, Sailor-7B, GigaSpeech 2, and LoRA fine-tuning.
  • SLAM-LLM decomposes multimodal models into independently configurable Encoder, Projector, and LLM components.

B. Modular Design

SLAM-LLM uses a decoupled Encoder–Projector–LLM architecture that supports configurable components, tuning strategies, and audio-language tasks. Its recipes and experiments provide reusable baselines across speech recognition, low-resource ASR, and code-switching ASR.

  • Modular architecture: SLAM-LLM decomposes multimodal models into independently configurable Encoder, Projector, and LLM components.Encoders perceive signals, projectors align features with the LLM input space, and LLMs generate outputs.
  • Component flexibility: The framework supports variable- and fixed-length projectors, including linear projectors and Q-Former, alongside diverse speech, language, audio, and music encoders and LLMs.Components can be individually selected and tuned through training configurations.
  • Recipes and baselines: SLAM-LLM provides task-specific recipes and checkpoints intended to guide optimization and serve as baseline references for future research.The framework emphasizes mainstream components and large-scale experiments across classic tasks.
  • ASR results: 1.84% test-clean WER and 3.39% test-other WER are achieved with a fine-tuned HuBERT X-Large encoder in LLM-based ASR.These correspond to relative reductions of 28.7% and 47.4% over Whisper-large.
  • ASR results: The LLM-based model outperforms specialist LibriSpeech systems without heavy engineering, but SSL systems with larger pretraining and in-domain language models remain stronger on noisy test-other.The comparison identifies language-model integration as a major source of those SSL systems’ gains.
  • ASR results: 20.2% relative MER reduction is achieved by the LLM-based model over Whisper large-v3 on Mandarin-English code-switching ASR.The system uses Whisper large-v3, Qwen2-7B, a linear projector, and LoRA adapters.

1) Task setup:

Visual Contextual ASR uses slide-derived textual context to improve transcription of conference speech. The SlideSpeech setup pairs speech with synchronized slides, OCR results, and keywords, then evaluates contextual prompting with an LLM-based ASR architecture.

  • Task setup: Visual Contextual ASR uses OCR-extracted slide keywords paired with each speech segment to improve conference-content transcription.The related figure presents visual contextual ASR as an LLM-based SLAM-LLM task.
  • Task setup: SlideSpeech provides synchronized audio, transcripts, slides, OCR results, and segment keywords from YouTube conference videos.Training uses either 473 hours in L95 or 161 hours in S95.
  • Task setup: The model combines WavLM Large, Vicuna-7B, and a lightweight linear projector for contextual ASR.The projector downsamples 50Hz speech features to 10Hz before alignment with the LLM input space.
  • Results: 9.0%/11.2% WER is obtained with keyword prompts on L95/S95, improving over baseline values of 9.4%/11.7%.Keyword prompting also improves biased-word recognition while leaving unbiased WER stable.

3) Contextual Biasing Speech Recognition:

Contextual Biasing ASR supplies candidate vocabulary lists to help recognize rare or domain-specific words. On LibriSpeech, the LLM-based system benefits most from short biasing lists but retains improvements as distractors increase.

  • Model setup: The model uses a fine-tuned WavLM Large encoder, Vicuna-7B decoder, and lightweight linear projector.WavLM is fine-tuned with CTC on the 960-hour LibriSpeech training set.
  • Results: 1.27%/3.67% WER/B-WER on test-clean and 2.72%/8.02% on test-other are achieved with 100-word biasing lists.These correspond to relative WER/B-WER reductions of 39.81%/63.37% and 35.24%/61.37% over baseline, while U-WER remains stable.
  • Results: Performance degrades slightly as biasing lists grow, but 2,000-word lists still improve over baseline and demonstrate effective filtering of distractors.The model achieves 1.38%/4.41% on test-clean and 3.20%/10.02% on test-other with 2,000-word lists.
  • Comparison: The LLM-based contextual ASR method significantly outperforms traditional neural-network contextual ASR approaches with 1,000-word biasing lists.The comparison uses artificial biasing lists on LibriSpeech test sets.

D. Speech-to-Text Translation (S2TT)

SLAM-LLM’s S2TT model aligns frozen Whisper speech features to Qwen2-7B through a trainable Q-Former. A multimodal chain-of-thought formulation decomposes speech translation into recognition followed by translation and achieves strong supervised and zero-shot results.

  • Model setup: The S2TT model uses a frozen Whisper encoder, a trainable Q-Former projector, and Qwen2-7B to generate text from speech and textual embeddings.The Q-Former compresses and aligns speech features with the LLM input space.
  • Evaluation: Training uses CoVoST2, while MuST-C provides zero-shot evaluation.The setup evaluates speech translation across supervised and cross-dataset settings.
  • Method: The SRT formulation decomposes speech translation into sequential ASR and multimodal machine translation stages.Minimal prompts distinguish ASR, MMT, and SRT during instruction fine-tuning.
  • Results: The model achieves state-of-the-art performance on CoVoST-2 and outperforms existing methods for zero-shot English-to-Chinese translation on MuST-C.The training procedure uses three-stage supervised fine-tuning with curriculum learning before zero-shot evaluation.

E. Speech Emotion Captioning (SEC)

The SEC experiments adapt SLAM-LLM to generate nuanced natural-language descriptions of speech emotion, using curated emotional speech data and a simplified model pipeline. The model reaches a SIM of 71.10% without auxiliary objectives, while the broader ASR experiments identify scaling, model, encoder, and input-length effects.

  • Task setup: SEC describes speech emotions with natural-language captions, offering more nuanced affective descriptions than fixed-category speech emotion recognition.
  • Datasets: Approximately 40 hours of in-house emotional speech data were annotated with 1 to 3 emotion-related captions per segment.
  • Model setup: The model replaces SECap's complex auxiliary-learning strategy to simplify training while enhancing intrinsic emotion perception.
  • Experiments: 71.10% SIM was achieved on 600 EMOSpeech test sentences without auxiliary objectives, outperforming vanilla SECap.
  • ASR takeaways: Larger encoders and LLMs, chat LLMs, sufficiently large self-supervised encoders, and domain-specialized LLMs improve ASR performance.
  • ASR takeaways: Whisper's 30-second padding and truncation can degrade performance while increasing computational demands during LLM post-training.

V. LLM-BASED AUDIO AND MUSIC PROCESSING

SLAM-LLM supports supervised and zero-shot automated audio captioning through modular encoders, projectors, LLMs, and parameter-efficient training. Across Clotho and AudioCaps, the vanilla system improves over prior systems, while the zero-shot model leads in-domain and cross-domain comparisons.

  • Task setup: AAC generates fine-grained natural-language descriptions from audio in either fully supervised or zero-shot settings.The vanilla setting uses paired audio-text training, whereas zero-shot training uses text data and captions audio at inference.
  • Datasets and metrics: The experiments use Clotho, AudioCaps, WavCaps, and MACS with METEOR, CIDEr, SPICE, SPIDEr, SPIDEr-FL, and FENSE metrics.
  • Vanilla AAC: The vanilla system combines frozen EAT features, a downsampling linear projector, and Vicuna decoding with efficient fine-tuning.
  • Vanilla AAC: On Clotho, the vanilla system achieves the highest scores across all AAC metrics, including 54.0% FENSE.
  • Vanilla AAC: On AudioCaps, the system reaches 84.1% CIDEr versus LOAE's 81.6% and achieves 66.8% FENSE, outperforming all other models.
  • Ablations: Ablations show that fine-tuned encoders, pre-training, LoRA, RAG, and projection-based decoding improve captioning quality or efficiency.
  • Zero-shot AAC: The zero-shot model surpasses competitive systems by a large margin in-domain and achieves state-of-the-art results across cross-domain metrics.

B. Music Captioning (MC)

The music-captioning experiments train frozen-encoder, frozen-LLM systems with only a linear projector updated, using a small music-captioning dataset. Despite this restricted training data, the models achieve results comparable to systems pretrained on a larger dataset.

  • Datasets: Training uses the LP-MusicCaps-MC subset because raw LP-MusicCaps-MSD audio is difficult to access.
  • Results: Models trained only on LP-MusicCaps-MC achieve results comparable to models pretrained on LP-MusicCaps-MSD.
  • Model setup: The music encoder and Vicuna-7b-v1.5 remain frozen, while a linear projector is the only trainable component.
  • Model setup: Frame-wise encoders are downsampled to 0.5 Hz, whereas sequence-wise encoders provide one projected token directly to the LLM.
  • Experiments: Among frame-wise encoders, MuQ outperforms MusicFM and MERT; sequence-wise encoders generally perform better despite using only one token.

C. Takeaways

The paper's experiments yield practical design guidance for LLM-based speech, audio, and music systems. The reported patterns concern model scale, encoder training, parameter-efficient adaptation, retrieval, projection decoding, token compression, and task-dependent projector choice.

  • Audio and music tasks: LLM-based models outperform non-LLM baselines and achieve state-of-the-art results on both Clotho and AudioCaps.
  • Audio and music tasks: Fine-tuned EAT audio encoders consistently outperform pretrained or less optimized counterparts.
  • Audio and music tasks: Pre-training combined with LoRA produces better captions at lower training cost.
  • Audio and music tasks: RAG improves descriptions of unseen soundscapes by providing access to external knowledge.
  • Audio and music tasks: Projection-based decoding narrows the CLAP modality gap, whereas directly using audio-encoder representations performs unsatisfactorily in zero-shot settings.
  • Audio and music tasks: A strong MuQ encoder can condense sparse music information into a single token for the LLM decoder.
  • Projector selection: Linear projectors suit strict temporal alignment, while Q-Former bridges suit global semantic perception and cross-modal context.

VI. CONCLUSION

SLAM-LLM provides a modular, flexible, open-source framework tailored to speech, language, audio, and music processing. Its recipes achieve competitive or state-of-the-art benchmark performance while offering insights for audio-language research.

  • SLAM-LLM provides a modular, flexible, open-source framework specifically tailored to speech, language, audio, and music processing.
  • Its encoder–projector–LLM architecture enables customization and deployment across tasks including ASR, SEC, AAC, and other audio-processing applications.
  • Recipes in SLAM-LLM achieve competitive or state-of-the-art performance on several benchmarks and provide insights for the LLM-based audio-processing community.
  • By lowering the entry barrier and promoting community collaboration, SLAM-LLM is positioned to accelerate research and innovation in audio-language modeling.
Loading 2601.09385v1…