OSUM: Advancing Open Speech Understanding Models with Limited Resources in Academia

1 Introduction

• Speech understanding requires semantic content and paralinguistic information such as emotion, speaking style, speaker gender, and age.The passage also notes that multi-label speech generation could reduce the computational burden of labeling tasks separately.Source: §1 · p. 1
• OSUM performs better than Qwen2-Audio on most tasks despite using significantly fewer computational resources and training data.Figure 1 summarizes average results on public and internal test sets.Source: Figure 1 · p. 2, §1 · p. 3
• Advanced SULMs often rely on millions of training hours and massive GPU resources, while key data and training details are rarely disclosed.These constraints are described as beyond academia’s reach and as hindering further research.Source: §1 · p. 2
• The OWSM series replicates Whisper-style training using open tools and publicly available data to advance research on speech understanding models.The cited passage presents OWSM as an example of the open-science movement in Speech LLM research.Source: §1 · p. 3
• OSUM combines a Whisper encoder with Qwen2, supports eight speech tasks, and uses 50,500 hours of data with an ASR+X training strategy.The data processing pipeline and training details are publicly available.Source: §1 · p. 3

2.1 Model Architecture

• The architecture uses a Whisper speech encoder and adaptor with updated parameters, while Qwen2-7B-Instruct is fine-tuned with LoRA and receives speech plus descriptive task prompts.OSUM currently produces text-based responses; the adaptor bridges the encoder output to the LLM input.Source: §2.1 · p. 4

2.2 Multitask Supervised Training

• Training first fine-tunes Whisper on multi-task data, then integrates it with Qwen2 for further supervised training on a larger dataset.The initial stage is intended to support faster convergence and verify multi-task data reliability.Source: §2.2 · p. 4
• ASR+X trains ASR alongside a target task, then combines the transcription with acoustic features for that task within the LLM’s autoregressive framework.The method adjusts predicted labels without changing the model architecture or loss functions.Source: §2.2 · pp. 4–5

2.3 Training Data

• The multi-task training data total 50,500 hours, including open-source and processed datasets; the ASR component alone uses 24,000 hours.The model is designed to share learning across tasks for speech understanding in conversational scenarios.Source: §2.4 · p. 5, §2.3 · p. 5, Table 2 · p. 6

2.4 Data Processing Pipe-line

• SRWT timestamps come from GMM-HMM forced alignment, while vocal-event scarcity is addressed by inserting voice-converted events into speech audio.The authors report that voice conversion mitigates overfitting caused by scarce vocal-event training data.Source: §2.4 · pp. 6–7
• SER labels combine Emotion2Vec audio annotations with GLM text annotations, while SSR retains style labels agreed on by Qwen2.5 and GLM.Both pipelines use intersected annotations to produce training labels.Source: §2.4 · pp. 6–7
• For SGC and SAP, the authors use manually labeled data after automated labeling performed poorly; noisy augmentation is used for both tasks.The SAP data are described as reliable speaker-age labels.Source: §2.4 · p. 7
• STTC training combines human-recorded audio questions, generated speech-text dialogue pairs, and response sentences selected from Wenetspeech.Qwen2.5-7B is used to select suitable response sentences and guide generation of text answers.Source: §2.4 · p. 7

3.1 Training Setup

• The two-stage setup fine-tunes Whisper-Medium on multitask data, then trains OSUM on 24 Huawei Ascend NPUs for 704,000 steps over 10 days.The first stage uses 8 Nvidia A6000 GPUs for 150,000 steps over approximately 15.3 days.Source: §3.1 · pp. 7–8

3.2 Internal Test Sets

• Internal Chinese multi-task test sets complement public English evaluations, and transcripts are included to support assessment of ASR+X convergence.Internal ASR metrics are for reference and will not be publicly reported; the sets are planned for future release.Source: §3.2 · p. 8

3.3 Main Results

• OSUM is comparable to or outperforms Qwen-Audio, Qwen2-Audio, Whisper, and SenseVoice across the reported tasks.The results are presented in Table 5 and the ASR evaluation in Table 4.Source: §3.3 · p. 9, Table 4 · p. 9, Table 5 · p. 11
• On Chinese ASR test sets, OSUM consistently leads on WenetSpeech test-meeting, three AISHELL-2 subsets, and four internal SpeechIO sets, while matching SenseVoice-S on English.On ASRU code-switching, MER/CER/WER are 3.89%/2.41%/19.30%; the authors attribute reduced training-resource needs and improved ASR convergence to ASR+X.Source: §3.3 · p. 9, Table 4 · p. 9
• OSUM beats Whisper-Large-v3 by a relative 36.70% and also surpasses Qwen-Audio on SRWT, slightly exceeding the GMM-HMM timestamp model.The authors say end-to-end timestamp prediction also simplifies integration with tasks such as speaker diarization.Source: §3.3 · pp. 9–10
• VED results are qualified by dataset mismatch: OSUM shows balanced public and internal performance, while vocal-event augmentation is reported to mitigate overfitting; SER is strong on MER2023 and comparable to EmoBox internally, but disgust and fear remain difficult.On MELD, OSUM ranks just below SenseVoice-L, which had additional large-scale emotion training data; scarcity of disgust and fear examples partly explains their difficulty.Source: §3.3 · p. 10
• Acoustic-text SSR outperforms GLM-4-9B-Chat's text-only modality, while SGC reaches 100% on AISHELL-1 but may be overfit and falls short of Qwen2-Audio internally.OSUM slightly exceeds Qwen2-Audio on Kaggle for SGC, but Qwen2-Audio is stronger on the internal set.Source: §3.3 · p. 10

AISHELL-1

• Age prediction groups speakers as child, adult, or old because teenagers and adults are acoustically difficult to distinguish.OSUM reaches 76.52% accuracy on Kaggle and outperforms Qwen2-Audio there and on the proprietary test set, despite a slight proprietary-set decline.Source: AISHELL-1 · pp. 11–12
• OSUM's age accuracy is lower on its proprietary test set than on Kaggle, but it still exceeds Qwen2-Audio on that set.The passage describes the proprietary-set accuracy as slightly declined without giving its numerical value.Source: AISHELL-1 · p. 12
• OSUM scores below Qwen2-Audio on AirBench's official speech sub-test, but outperforms it on the internal Chinese conversational set.The authors attribute the English shortfall primarily to the absence of English conversational training data and say the Chinese result indicates ASR-before-chat is beneficial.Source: AISHELL-1 · p. 12

4 Future Works

• Planned functionality additions include language and accent identification for multilingual and diverse speech scenarios.These are described as extensions to OSUM's current capabilities.Source: §4 · p. 12
• The authors aim to enable emotion, age, gender, and speaking-style recognition in one inference pass and use this capability for audio-data labeling.The stated goal is to streamline audio-data processing pipelines.Source: §4 · p. 12
• OSUM is planned to gain full-duplex voice interaction for context-aware responses that can match a questioner's emotion or speaking style.The passage gives a child's speaking style as an example.Source: §4 · p. 12
• The authors plan to continue sharing training methodologies, data pipelines, and model updates to support academic collaboration.They frame this as an open-science contribution to access and research.Source: §4 · p. 12

• OSUM combines a Whisper encoder with a Qwen2 LLM and supports eight speech tasks.Source: §5 · p. 12
• ASR+X training simultaneously optimizes ASR alongside target tasks for efficient and stable multi-task training.Source: §5 · p. 13
• Openly accessible data-preparation and training methodologies provide practical guidance for the academic community.Source: §5 · p. 13

• Author ranking proceeds from top to bottom, then from left to right.Source: §7 · p. 13
• The first listed authors are Xuelong Geng, Kun Wei, Qijie Shao, and Shuiyun Liu, whose asterisk marks equal contribution.Source: §7 · p. 13
• The next listed authors are Chengyou Wang, Tianyi Xu, Yuhang Dai, Xinfa Zhu, Yue Li, Li Zhang, and Lei Xie, marked as corresponding author.Source: §7 · p. 13
• The remaining listed authors are Peikun Chen, Yangze Li, Pengcheng Guo, Mingchen Shao, Shuiyuan Wang, and Yuang Cao.Source: §7 · p. 13