Source-linked AI summary
Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, Jingren Zhou
TL;DR
Existing audio-language models support limited audio types and tasks, motivating a broader pre-trained model for audio interaction. Qwen-Audio scales audio-language training across diverse tasks and audio types with hierarchical tags to manage label variation and interference, achieving strong benchmark performance without task-specific fine-tuning; Qwen-Audio-Chat further supports multilingual, multi-turn audio-text dialogue.
Problem
Existing audio-language models lack pre-training that covers diverse audio types and tasks, limiting the range of supported audio interactions.
Method
Qwen-Audio scales audio-language pre-training to over 30 tasks, eight languages, and varied audio types, using hierarchical decoder tags to share knowledge and mitigate label interference.
Results
Qwen-Audio outperforms previous works across diverse benchmarks without task-specific fine-tuning, while Qwen-Audio-Chat supports multilingual, multi-turn dialogue from audio and text inputs.
Takeaways & Limitations
The Qwen-Audio series provides universal audio understanding and audio-text interaction across diverse audio-oriented scenarios.
Abstract
from arXiv · showhide
Recently, instruction-following audio-language models have received broad attention for audio interaction with humans. However, the absence of pre-trained audio models capable of handling diverse audio types and tasks has hindered progress in this field. Consequently, most existing works have only been able to support a limited range of interaction capabilities. In this paper, we develop the Qwen-Audio model and address this limitation by scaling up audio-language pre-training to cover over 30 tasks and various audio types, such as human speech, natural sounds, music, and songs, to facilitate universal audio understanding abilities. However, directly co-training all tasks and datasets can lead to interference issues, as the textual labels associated with different datasets exhibit considerable variations due to differences in task focus, language, granularity of annotation, and text structure. To overcome the one-to-many interference, we carefully design a multi-task training framework by conditioning on a sequence of hierarchical tags to the decoder for encouraging knowledge sharing and avoiding interference through shared and specified tags respectively. Remarkably, Qwen-Audio achieves impressive performance across diverse benchmark tasks without requiring any task-specific fine-tuning, surpassing its counterparts. Building upon the capabilities of Qwen-Audio, we further develop Qwen-Audio-Chat, which allows for input from various audios and text inputs, enabling multi-turn dialogues and supporting various audio-central scenarios.
1 Introduction
Qwen-Audio addresses limited audio interaction capabilities by unifying diverse audio types and tasks in large-scale audio-language pre-training. Its hierarchical-tag framework supports knowledge sharing while reducing interference, and the resulting models perform strongly across benchmarks without task-specific fine-tuning.
- Motivation: Existing audio-language models were limited by the lack of pre-trained models handling diverse audio types and tasks.Prior systems often focused on human speech or natural sounds rather than broad audio interaction.
- Contribution: Qwen-Audio scales training across over 30 tasks, eight languages, and various audio types to advance universal audio understanding.The model covers speech, natural sounds, music, and songs through a large-scale audio-language setup.
- Method: Hierarchical decoder tags address variation in task labels by encouraging shared knowledge while mitigating one-to-many interference.The variation reflects differences in task objectives, languages, annotation granularity, and text structure.
- Contribution: Qwen-Audio-Chat extends the model to multi-turn dialogues and diverse audio-oriented scenarios through instruction fine-tuning.The system supports various audio and text inputs.
- Results: SRWT training improves grounding, grounding-based question answering beyond speech signals, and ASR performance.The contribution identifies SRWT as an often-overlooked task in audio multimodal research.
- Results: Qwen-Audio achieves impressive performance across diverse benchmark tasks without task-specific fine-tuning and surpasses counterparts.The evaluation spans 12 datasets covering ASR, S2TT, AAC, ASC, SER, AQA, VSC, and MNA.
2 Related Work
Prior multimodal audio systems either use external tools, specialize in particular audio capabilities, or adapt existing language models with targeted supervision. Qwen-Audio instead aims at unified audio-text multi-task learning across varied audio inputs and labels.
- Multi-task Audio-Text Learning: Multi-task audio learning is difficult because human speech, natural sounds, music, and songs use substantially different labeling formats.SpeechNet and SpeechT5 organize human-speech tasks into speech/text input and output formats.
- Interact with LLMs through Multiple Modality: Visual multimodal systems extend language models to image understanding or generation through different integration methods.The related work positions these systems as broader multimodal precedents rather than audio-specific solutions.
- Interact with LLMs through Multiple Modality: AudioGPT and HuggingGPT use language models as interfaces to well-trained audio tools, but tool pipelines can omit prosody and sentiment in speech.These approaches may transcribe speech or generate commands for external tools before language-model processing.
- Interact with LLMs through Multiple Modality: LTU improves sound perception and reasoning through audio-question-answering supervision, while SALMONN combines text and speech encoders with a Q-former-style connection.These systems use targeted adaptation of audio modules or language-model components.
3 Methodology
Qwen-Audio combines an audio encoder with a large language model and trains them through multitask pretraining across diverse audio datasets. Its hierarchical tagging framework organizes task, language, timestamp, and output-format information to promote sharing among related tasks while reducing interference, followed by instruction fine-tuning for multi-turn audio dialogue.
- 3.1 Model Architecture: Qwen-Audio combines an audio encoder with a large language model to model paired audio and text sequences.The training objective maximizes next-token probabilities conditioned on audio representations and preceding text.
- 3.1 Model Architecture: The model uses a single audio encoder to process diverse audio types and a Qwen-7B-derived language model as its foundational component.The encoder is initialized from Whisper-large-v2, while the language model uses pretrained Qwen-7B weights.
- 3.2 Multitask Pretraining: The framework aims to share knowledge among similar tasks while distinguishing different output formats to reduce one-to-many interference.This design is presented as the guiding principle of the multitask format.
- 3.2 Multitask Pretraining: Directly mixing datasets can cause interference because labels differ in task focus, language, annotation granularity, and text structure.The paper frames this as a one-to-many mapping problem during co-training.
- 3.2 Multitask Pretraining: Qwen-Audio organizes outputs with hierarchical tags for transcription mode, audio language, task, text language, timestamps, and output instructions.The tags distinguish tasks and formats while enabling shared representations among related tasks.
- 3.3 Supervised Fine-tuning: Qwen-Audio-Chat is produced through instruction fine-tuning with audio-centric and pure-text data for multi-turn, multi-audio interaction.The instruction-tuning dataset contains 20k examples and uses Audio id labels for multiple audio inputs.
4 Experiments
Qwen-Audio is evaluated across diverse audio tasks and datasets without task-specific fine-tuning. Results indicate strong performance in speech, audio analysis, timestamp-aware recognition, and conversational scenarios.
- Evaluation covers ASR, S2TT, AAC, SRWT, ASC, SER, AQA, VSC, and MNA across 12 datasets.
- Speech recognition and translation: Qwen-Audio achieves 2.0% and 4.2% WER on LibriSpeech test-clean and test-other, respectively, and state-of-the-art results on Aishell1 dev and test sets.
- Speech recognition and translation: Qwen-Audio outperforms baselines across all seven CoVoST2 translation directions.
- Audio analysis: Across audio analysis tasks, Qwen-Audio consistently outperforms baselines and achieves state-of-the-art results on CochlScene, ClothoAQA, and VocalSound.
- Interactive chat: Qwen-Audio-Chat demonstrates conversational capabilities through illustrative multi-turn audio-interaction cases.
- Word-level timestamps: SRWT trains the model to generate transcripts with word-level timestamps, supporting fine-grained audio alignment and grounding-based question answering.
- Word-level timestamps: Removing SRWT training reduces performance in ASR and audio question answering, including natural-sounds and music QA, while leaving audio-dataset coverage unchanged.
5 Conclusion
The Qwen-Audio series targets universal audio understanding through unified multi-task learning across varied audio inputs and text formats. Without task-specific fine-tuning, Qwen-Audio outperforms prior work across diverse benchmarks, while Qwen-Audio-Chat supports multilingual, multi-turn audio-text dialogue.
- Qwen-Audio uses unified multi-task learning to share knowledge across audio tasks while avoiding interference from different text formats.
- Without task-specific fine-tuning, Qwen-Audio outperforms previous works across diverse benchmarks.
- Supervised instruction fine-tuning gives Qwen-Audio-Chat multilingual, multi-turn dialogue capabilities from audio and text inputs.
A Hyperparameters
The paper reports Qwen-Audio’s training hyperparameters in Table 6.
- Training hyperparameter settings for Qwen-Audio are reported in Table 6.