Source-linked AI summary
Qwen2-Audio Technical Report
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou
TL;DR
Qwen2-Audio addresses the challenge of improving audio-language-model instruction following across diverse audio inputs and tasks. It expands training data, uses natural-language prompts, jointly supports audio analysis and voice chat, and applies instruction tuning and DPO; evaluations report strong audio understanding and dialogue performance, including state-of-the-art results on several benchmarks.
Problem
Qwen2-Audio targets improved instruction following and comprehensive understanding of diverse audio signals in large audio-language models.
Method
The model expands training data, replaces hierarchical tags with natural-language prompts, jointly trains audio analysis and voice chat, and uses instruction tuning plus DPO.
Results
Qwen2-Audio outperforms previous LALMs across diverse tasks without task-specific fine-tuning and achieves state-of-the-art performance on Aishell2, FLUERS-zh, VocalSound, and AIR-Bench chat.
Takeaways & Limitations
Qwen2-Audio provides audio understanding and dialogue capabilities with seamless voice and text interactions across its supported modes.
Abstract
from arXiv · showhide
We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community.
1 Introduction
Qwen2-Audio is a large audio-language model designed to improve instruction following across diverse audio tasks through expanded training, natural-language prompts, and aligned interaction modes. It supports audio analysis and voice chat, and outperforms previous LALMs across varied evaluations without task-specific fine-tuning.
- Model and training: Qwen2-Audio processes audio and text inputs to generate textual outputs, with training focused on improved instruction-following capability.The model scales up training data and replaces hierarchical tags with natural-language prompts for different data and tasks.
- Evaluation: Qwen2-Audio is evaluated across 10 datasets spanning ASR, S2TT, SER, VSC, and audio-centric instruction following.AIR-Bench chat scores cover speech, sound, music, and mixed dimensions, while ASR results use 1 - WER%.
- Interaction modes: Audio Analysis accepts audio or text commands for analyzing speech, sounds, music, and mixed audio, while Voice Chat supports unrestricted conversational interaction.The model autonomously identifies command segments in audio and allows users to switch between audio and text interaction during voice chat.
- Interaction modes: The two interaction modes are jointly trained and integrated without requiring users to distinguish between them or switch modes with system prompts.
- Results: Without task-specific fine-tuning, Qwen2-Audio outperforms previous LALMs across diverse tasks and achieves state-of-the-art results on Aishell2, FLUERS-zh, VocalSound, and AIR-Bench chat.
2 Methodology
Qwen2-Audio combines an audio encoder with a large language model and is trained to process paired audio-text inputs. Its methodology emphasizes natural-language prompting, instruction tuning, preference optimization, and seamless audio-analysis and voice-chat interaction.
- Model Architecture: Qwen2-Audio uses an audio encoder and a large language model to model text-token probabilities from paired audio and text sequences.The model conditions on audio representations and previous text sequences during training.
- Pre-training: Natural-language prompts replace hierarchical tags during pre-training to improve generalization and instruction following.The report presents this change as a way to reduce the gap between pre-training and post-training.
- Supervised Fine-tuning: Instruction-based fine-tuning uses curated, high-quality data to align the model with human intent and produce an interactive chat model.The authors emphasize the importance of SFT-data quality and complexity.
- Interaction Modes: Qwen2-Audio supports audio analysis and voice chat, accepting audio or text instructions for analysis and voice interaction without separate system-prompt switching.The two modes are jointly trained and seamlessly integrated during use.
- Direct Preference Optimization: Direct Preference Optimization trains the model against human-annotated good and bad responses to improve alignment with human preferences.The optimization uses input audio with preferred and dispreferred response triplets.
3 Experiments
The experiments evaluate Qwen2-Audio across speech, sound, translation, emotion, classification, and chat tasks without task-specific fine-tuning. Results report strong performance across conventional benchmarks and state-of-the-art instruction following on AIR-Bench chat.
- Evaluation Setup: Evaluation covers diverse audio tasks without task-specific fine-tuning, including ASR, speech translation, emotion recognition, sound classification, and chat.The evaluation is intended to assess universal audio understanding across multiple task types.
- Benchmark Motivation: AIR-Bench scores are presented as more closely aligned with actual user interaction experience than many previous test datasets.The authors criticize some existing SLU and SER datasets as limited for real-world evaluation.
- Speech Recognition: 1.6% WER and 3.6% WER are achieved on LibriSpeech test-clean and test-other, respectively.The report also compares Qwen2-Audio with Whisper-large-v3 on the Fleurs zh subset.
- Speech Translation: Qwen2-Audio outperforms baselines across all seven CoVoST2 speech-translation directions.The report describes the margin as substantial across the evaluated directions.
- AIR-Bench Chat: Qwen2-Audio achieves state-of-the-art instruction following across AIR-Bench speech, sound, music, and mixed-audio subsets.The model substantially improves over Qwen-Audio and significantly outperforms other LALMs.
4 Cases
The cases section presents examples of Qwen2-Audio’s audio-based interaction, including free chat around speech. The report directs readers to the project repository for additional examples and resources.
- Examples: The report includes selected cases and refers readers to the Qwen2-Audio GitHub repository for more impressive examples.The repository is also identified as the location for code, demos, and models.
- Free Chat: Figure 4 illustrates Qwen2-Audio’s capability in free chat around speech.The figure is presented as an example of audio-based interaction capability.
5 Conclusion
Qwen2-Audio extends audio analysis with voice interaction, using expanded data and staged alignment to improve responses. The paper reports proficiency in audio understanding, dialogue, and flexible voice interaction across diverse benchmarks and examples.
- 5 Conclusion: Qwen2-Audio combines analysis of speech, sound, music, and mixed audio with voice interaction capabilities.The paper presents these as complementary audio-understanding and dialogue abilities.
- 5 Conclusion: Natural-language prompts and expanded pre-training data support training across different audio data and tasks.The approach replaces complex hierarchical tags during pre-training.
- 5 Conclusion: SFT increases the quantity, quality, and complexity of interaction data, while DPO improves response quality.Together, these stages target alignment with human interaction and preferred behavior.
- 5 Conclusion: Objective metrics on diverse benchmarks demonstrate proficiency in audio understanding and dialogue capabilities.The paper also presents cases illustrating fluent and flexible voice interaction.