MiDashengLM: Efficient Audio Understanding with General Audio Captions

1 Introduction

• Separate processing of transcripts, audio captions, and music captions can limit auditory-scene analysis, while existing captions may omit spoken semantics, room acoustics, and signal quality.The proposed unified caption fuses speech transcripts, audio captions, and music captions into a more detailed textual representation.Source: §1 · p. 2

1.1 Motivation

• ASR-based pretraining can discard non-speech audio, rely on simple monotonic speech-text alignment, and omit speaker attributes and acoustic characteristics.The paper argues that general captioning can use noisy audio and capture a wider range of acoustic information.Source: §1.1 · pp. 2–3

1.2 Audio caption and speech summarization

• Existing public caption datasets often neglect spoken language and reuse underlying audio sources, limiting data diversity; prior work also paraphrases captions or adds video context.The paper identifies general audio captions as a fusion of speech summarization, music captions, and audio captions.Source: §1.2 · p. 3

2 Framework

• The framework maps audio-encoder features into an LLM embedding space through an MLP layer in a prefix-based language model.Figure 2 depicts the proposed framework and its training stages.Source: §2 · p. 4
• Training, supervised fine-tuning, and instruction tuning use only publicly available audio-text data.The data sources are listed in Tables 14–18.Source: §2 · p. 4
• MiDashengLM aligns audio and text through general captions and uses one general audio encoder for speech, sound, and music, without ASR or sound-event models.The paper describes this as its first audio-text alignment approach via general captions.Source: §2 · p. 4
• Most training samples are 1–10 seconds; variable-length inputs reduce padding, and downsampling to 5 Hz shortens sequences for training and inference.The paper says shorter sequences benefit decoder computation, where most compute is performed.Source: §2 · p. 5, Figure 3 · p. 5

3.1 ACAVCaps and Multi-Expert Chain for Audio Tasks (MECAT)

• ACAV100M was selected for its public availability, multilingual speech, varied music and complex audio environments, and limited overlap with prior caption datasets.The source lacked audio-caption labels, prompting a dedicated curation pipeline.Source: §3.1 · p. 5
• The curation pipeline predicts sound events on a 2-second scale, applies specialized speech, vocal, music and environmental-acoustics analyses, then uses a reasoning LLM to generate captions.The resulting captions are split into ACAVCaps for training and MECAT for evaluation; MECAT is license-filtered and scored for audio-text consistency.Source: §3.1 · pp. 5–6
• ACAVCaps captions have a richer vocabulary than prior approaches, reflecting spoken-content summaries and multilingual transcripts retained in captions.Table 2 compares publicly available caption datasets by unique-word count and average sentence length.Source: §3.1 · p. 6, Table 2 · p. 7
• MECAT-QA pairs each clip with five question-answer pairs, covering perception, analysis and reasoning across more than 100,000 total pairs.Its subcategories include direct perception, sound characteristics, quality assessment, environment reasoning, inference and judgement, and application context.Source: §3.1 · p. 7

3.2 Training datasets and tasks

• About 1.1 million hours of public data support training, with three ACAVCaps alignment epochs followed by roughly 1.4 pretraining epochs and one SFT epoch on 352k hours.ACAVCaps provides 38,000 hours of general captions; approximately 90% of the training data originates from public ASR datasets.Source: §3.2 · pp. 7–8

4 Experimental Setup

• MiDashengLM is a Transformer encoder-decoder using Dasheng-0.6B, a frame-level ViT pretrained with a masked-autoencoder objective, and a text decoder initialized from Qwen2.5-Omni-7B.Inputs are resampled to 16 kHz and converted to 64-dimensional mel-spectrograms; the encoder extracts 32 ms frame features at a 10 ms stride.Source: §4 · pp. 8–9
• Dasheng downsamples features fourfold and accepts up to 1008 frames; longer audio is processed in non-overlapping sliding windows whose frame features are concatenated.The full pipeline required roughly 19,200 GPU hours, or 10 days on 80 GPUs.Source: §4 · p. 8
• Audio-text alignment fine-tunes both encoder and decoder end-to-end on ACAVCaps, then uses the trained encoder to initialize pretraining and SFT.Frozen-LLM integration and LoRA alternatives yielded unsatisfactory audio-encoder performance.Source: §4 · p. 8
• The decoder minimizes cross-entropy to predict each current text token from prior text tokens conditioned on audio features, while LoRA supports parameter-efficient pretraining and SFT.The decoder is initialized from the publicly available Qwen2.5-Omni-7B model.Source: §4 · p. 9
• Training uses a 1,000-iteration linear warm-up followed by cosine decay to 10% of the maximum learning rate; SFT uses a lower rate and expands LoRA trainable parameters.“All-linear” LoRA modifies all decoder projection layers, whereas “q,v” adapts query and value matrices.Source: §4 · p. 9

5.1 Audio encoder performance

• On X-Ares, the Dasheng-based encoder outperforms Whisper-Large v3 on 18 of 22 tasks, while Whisper leads on the remaining four, all speech-related.Largest reported gains are +195% on VoxCeleb1 speaker recognition, +137% on DESED, and +87% on Clotho audio-text retrieval.Source: §5.1 · p. 10, Table 5 · p. 10

5.2.1 Audio captioning results

• MiDashengLM consistently outperforms baseline models on traditional music and sound captioning datasets, with larger gains on general audio than music-specific benchmarks.FENSE is the primary caption metric.Source: §5.2.1 · p. 11, Table 7 · p. 11

5.2.2 MECAT

• On MECAT’s nine captioning domains, MiDashengLM surpasses the baselines by a significant margin, while Kimi-Audio-Instruct performs poorly.The domains include short and long captions, pure and mixed speech, sound and music, and environmental captions.Source: §5.2.2 · p. 11, Table 8 · p. 11

5.2.3 Audio and paralinguistic classification

• Across ten tested tasks, MiDashengLM outperforms baselines on speaker verification, language identification, sound classification, and music classification.The reported tasks include VoxCeleb1, VoxLingua107, VGGSound, FSD50k, NSynth, and FMA.Source: §5.2.3 · p. 12, Table 9 · p. 12

5.2.4 Automatic speech recognition

• MiDashengLM trails baselines on traditional LibriSpeech ASR, but outperforms Qwen2.5-Omni on the larger People’s Speech test set and supports ASR in Indonesian, Vietnamese, and Thai.The authors note that higher audio-token framerates improve ASR performance at the expense of computational efficiency.Source: §5.2.4 · p. 12, Table 10 · p. 12

5.3 Question answering results

• MiDashengLM leads the baselines on both closed QA benchmarks, with 71.35% accuracy on MMAU and 66.30% on MuChoMusic, and maintains its lead on open QA tasks.The open tasks are MusicQA and AudioCaps-QA.Source: §5.3 · p. 13, Table 11 · p. 13

5.3.1 MECAT-QA

• On the open MECAT-QA dataset, MiDashengLM outperforms the baselines by a significant margin.The dataset is part of the publicly available MECAT benchmark; results are reported using FENSE.Source: §5.3.1 · p. 13, Table 12 · p. 13

5.4 Inference speed

• Up to 4× lower TTFT (160ms vs. 40ms) and 20.2× potential throughput speedup at larger batches distinguish MiDashengLM from Qwen2.5-Omni-7B.At comparable batch sizes, the reported throughput speedup is 3.2×; the comparison uses the same text-decoder backbone.Source: §5.4 · p. 13, Figure 7 · p. 14, Table 13 · p. 14

• MiDashengLM uses general audio captioning, ACAVCaps, and a Dasheng-based encoder, and is trained exclusively on publicly available audio-text data.Source: §6 · p. 14
• The Dasheng-based encoder outperforms Whisper-Large V3 on 18 of 22 X-Ares tasks, while MiDashengLM competes favorably with Qwen2.5-Omni and Kimi-Audio across multiple audio domains.The conclusion names audio captioning, closed and open QA, sound event detection, and paralinguistic tasks.Source: §6 · p. 14
• Against the baselines, MiDashengLM delivers up to 20.2× faster inference and up to 4× reduced time-to-first-token latency.Source: §6 · p. 14

• The table lists the speech training data.Source: Table 14 · p. 23
• A dagger marks datasets for which the authors used Whisper to generate automatic transcripts.Source: Table 14 · p. 23
• The “SFT ?” column marks supervised-finetuning use; by default, listed data is used for pretraining.Source: Table 14 · p. 23

• The table covers general sound and audio datasets.Source: Table 15 · p. 24
• ACAVCaps is used for audio-text alignment.Source: Table 15 · p. 24
• The “SFT ?” column marks supervised-finetuning use; by default, listed data is used for pretraining.Source: Table 15 · p. 24

• The table catalogs speech and sound paralinguistic datasets.Source: Table 16 · p. 25
• Its “SFT ?” column indicates whether each dataset is used for supervised finetuning.Source: Table 16 · p. 25
• By default, all listed data is used for pretraining.Source: Table 16 · p. 25

• The table provides an overview of music-related datasets.Source: Table 17 · p. 26
• Its “SFT ?” column indicates whether each dataset is used for supervised finetuning.Source: Table 17 · p. 26
• By default, all listed data is used for pretraining.Source: Table 17 · p. 26

A.5 Question Answering Datasets

• † marks datasets modified by using an LLM to convert captions into question-answer pairs.Source: Table 18 · p. 26
• The table reports each dataset’s number of questions and answers as # QA.Source: Table 18 · p. 26
• Only AVQA, MusicQA, and ClothoAQA are used for pretraining by default; “SFT ?” indicates supervised-finetuning use.Source: Table 18 · p. 26

B Contributors

• The listed contributors are presented in alphabetical order.Source: B Contributors · p. 26