Source-linked AI summary
Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models
Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, Chunyan Miao
TL;DR
Audio reasoning has lacked large-scale chain-of-thought training and datasets suited to complex queries. The paper builds CoTA through annotation refinement, question synthesis, and structured CoT generation, then trains Audio-Reasoner on it. Audio-Reasoner achieves state-of-the-art results across several audio benchmarks, including a 25.4% MMAU-mini gain over Qwen2-Audio-Instruct.
Problem
Audio chain-of-thought reasoning is underexplored, and existing audio datasets with simple labels limit progress on complex queries.
Method
The paper creates CoTA, a 1.2-million-sample reasoning dataset, and fine-tunes Audio-Reasoner with structured chain-of-thought training and inference scaling.
Results
Audio-Reasoner achieves state-of-the-art performance across key benchmarks, including a +25.4% improvement on MMAU-mini over Qwen2-Audio-Instruct.
Takeaways & Limitations
The findings underscore the role of reasoning-rich datasets and structured reasoning in improving audio reasoning, especially on complex tasks.
Abstract
from arXiv · showhide
Recent advancements in multimodal reasoning have largely overlooked the audio modality. We introduce Audio-Reasoner, a large-scale audio language model for deep reasoning in audio tasks. We meticulously curated a large-scale and diverse multi-task audio dataset with simple annotations. Then, we leverage closed-source models to conduct secondary labeling, QA generation, along with structured COT process. These datasets together form a high-quality reasoning dataset with 1.2 million reasoning-rich samples, which we name CoTA. Following inference scaling principles, we train Audio-Reasoner on CoTA, enabling it to achieve great logical capabilities in audio reasoning. Experiments show state-of-the-art performance across key benchmarks, including MMAU-mini (+25.42%), AIR-Bench chat/foundation(+14.57%/+10.13%), and MELD (+8.01%). Our findings stress the core of structured CoT training in advancing audio reasoning.
1 Introduction
Audio-Reasoner addresses the underexplored problem of large-scale chain-of-thought reasoning in audio by combining structured reasoning training with CoTA, a diverse 1.2-million-sample dataset. It achieves strong results across audio reasoning and related tasks, including a 25.4% gain on MMAU-mini over Qwen2-Audio-Instruct.
- Motivation: Audio chain-of-thought reasoning remains underexplored, while existing audio datasets often provide short, simple labels that limit complex reasoning.Existing zero-shot audio reasoning methods do not significantly improve performance on complex queries.
- Contribution: Audio-Reasoner uses structured chain-of-thought training for long-context audio reasoning, organizing generation into planning, captioning, reasoning, and summary stages.The model is trained with a 4K-token context window and can generate reasoning exceeding 1K tokens in real-world tasks.
- Results: The paper evaluates Audio-Reasoner across sound, speech, music, translation, and emotion-classification benchmarks.MMAU-mini covers sound, speech, and music; AIR-Bench covers speech, natural sounds, and music.
- Contribution: CoTA contains 1.2 million high-quality captions and QA pairs spanning multiple audio datasets and tasks.The dataset undergoes filtering for diversity and quality.
2 Related Work
Related work establishes chain-of-thought reasoning in language and multimodal models, while large audio language models have mainly focused on audio understanding and dialogue. These lines of work provide context for extending structured reasoning to audio.
- Chain-of-Thought Reasoning: Chain-of-thought methods include tree-based, manual few-shot, and automatically generated reasoning approaches for language models.Prior studies also examine CoT’s necessity, theoretical foundations, and effectiveness across tasks.
- Multimodal Chain-of-Thought: Multimodal chain-of-thought models use visual tools, recaptioning, scaled datasets, and reflective methods to improve reasoning.Examples include Visual-COT, LLaVA-Reasoner, LLaVA-CoT, MAmmoTH-VL, and Mulberry.
- Large Audio Language Models: Large audio language models broadly target audio understanding or real-time dialogue, commonly combining an encoder, connector, and LLM.Representative systems include Mu-LLaMA, LTU, EmoBox, GAMA, LTU-AS, SALMONN, and Qwen2-Audio.
3 Audio-Reasoner
Audio-Reasoner trains large audio language models to produce explicit chain-of-thought reasoning before final answers. Its CoTA dataset and structured data-generation pipeline support reasoning across diverse audio tasks and complexities.
- 3.1 Model Training with Audio Reasoning: Audio-Reasoner generates a chain-of-thought process followed by a final response, unlike conventional single-response language models.The model learns the concatenation of structured reasoning C and final response R.
- 3.1 Model Training with Audio Reasoning: The training objective maximizes the likelihood of both reasoning and final-response generation.This explicitly encourages the model to reason first and then provide an answer.
- 3.1 Model Training with Audio Reasoning: At inference, the model follows four sequential stages: planning, captioning, reasoning, and summary.These stages analyze the query, extract relevant multimodal content, perform step-by-step reasoning, and produce a concise result.
- 3.2 Systematic Data Preparation for Audio Reasoning: CoTA contains 1.2 million samples spanning audio, speech, and music, created through annotation enrichment, question synthesis, reasoning-chain construction, and validation.The pipeline is designed to transform simple human annotations and raw audio into structured reasoning tasks.
- 3.2.2 Task Taxonomy: Categories and Reasoning Methodologies: CoTA combines real-world data with 14.15% synthetic Multi-Speaker and Complex Audio samples to support complex multi-step inference.The dataset unifies ten task categories across multiple audio domains.
- 3.3 CoTA Dataset Analysis: Most CoTA responses contain 300–500 words, while Multi-Speaker responses reach 1,500 words and simpler S2TT responses typically span 100–200 words.The distribution reflects task-dependent response lengths intended to balance detailed reasoning with concise answers.
4 Experiments
Audio-Reasoner is evaluated against diverse closed-source, open-source, and cascade baselines across audio reasoning, translation, and emotion tasks. It achieves state-of-the-art results across the reported benchmarks, with structured reasoning supporting broad performance across sound, music, speech, and mixed audio.
- Experimental Setup: Audio-Reasoner is compared with closed-source, open-source, cascade, and multimodal baselines across MMAU-mini, AIR-Bench, CoVoST 2, and MELD.The evaluation uses multiple benchmarks spanning reasoning, speech translation, and emotion recognition.
- Main Results: 1.01 points: Audio-Reasoner’s AIR-Bench chat score of 7.94 surpasses Qwen2-Audio’s 6.93 across sound, music, speech, and mixed audio.Its domain scores include 7.68 in sound, 8.05 in music, and 8.19 in speech.
- Main Results: 6.0 points: Audio-Reasoner’s AIR-Bench foundation score of 65.2 exceeds Qwen-Audio-Turbo’s 59.2.It leads across sound, music, and speech subdomains, including SER, SIC, and SNV.
- Main Results: 50.87 average BLEU: Audio-Reasoner exceeds Gemini-1.5-pro by 4.63 points and Qwen2-Audio-Instruct by 13.80 points on EN-ZN CoVoST 2 translation.For ZN-EN, it achieves an average BLEU score of 29.13, also exceeding both baselines.
- Main Results: 53.9 unweighted accuracy on MELD is 4.0 percentage points above Qwen2-Audio-Instruct’s 49.9.This is the highest score among the reported baseline models for speech emotion recognition.
- Case Study: The case study shows Audio-Reasoner combining guitar-riff analysis, commentary interpretation, and purpose inference in a structured response.The system considers whether the interaction suggests an informal instructional or demonstration piece.
5 Conclusion
The paper introduces Audio-Reasoner and CoTA, a large structured reasoning dataset, to advance deep reasoning in audio tasks. Experiments report state-of-the-art performance on several benchmarks and motivate continued development of structured CoT methods for audio language models.
- Conclusion: Audio-Reasoner is a large audio language model trained for deep audio reasoning using inference scaling and structured chain-of-thought reasoning.The model is built around CoTA, which contains around 1.2 million structured reasoning samples.
- Conclusion: CoTA is generated through annotation refinement, question synthesis, and chain-of-thought generation in a systematic data pipeline.The dataset provides the central training resource for the proposed model.
- Conclusion: State-of-the-art results are reported on MMAU-mini (+25.04%), CoVoST 2 (+8.31%), and MELD (+8.01%).The reported improvements cover audio reasoning, speech translation, and emotion recognition benchmarks.
- Conclusion: The findings underscore the role of reasoning-rich datasets and inference scaling in multimodal learning for complex audio tasks.Future directions include multi-turn dialogue reasoning, cross-modal knowledge integration, and real-world speech-driven AI applications.
A Prompt Details
The prompts specify a staged process for turning audio, descriptions, questions, and answers into detailed, structured reasoning data. They require progressively challenging questions, explicit reasoning stages, strict output formatting, and validation against hallucinations.
- A Prompt Details: The prompt combines a clear task definition, structured examples, and precise format specifications to reduce ambiguity in audio-data generation.It requests detailed descriptions, progressively challenging questions, and a fixed response format.
- A.1 Prompt of Stage 1 when Processing Data (Sample from AudioSet): Stage 1 expands simple audio descriptions into detailed captions and generates three open-ended or single-choice questions with answers.Questions should remain grounded in the audio, increase in difficulty, and include four options for single-choice items.
- A.1 Prompt of Stage 1 when Processing Data (Sample from AudioSet): Stage 1 enforces XML-like tags separating captions, questions, and answers, with a sample format that the model must follow strictly.The sample illustrates three question blocks and answer fields.
- A.2 Prompt of Stage 2 when Processing Data (Sample from AudioSet): Stage 2 constructs audio chain-of-thought data by dividing reasoning into planning, captioning, reasoning, and summarizing before producing a response.The think section is hidden from users, while the response is based on the reasoning process.
- A.2 Prompt of Stage 2 when Processing Data (Sample from AudioSet): The Stage 2 sample demonstrates how observed audio characteristics are progressively analyzed and condensed into a final answer.The example moves from identifying static noise to analyzing intensity changes and summarizing dynamic behavior.
- A.2 Prompt of Stage 2 when Processing Data (Sample from AudioSet): Stage 2 uses the answer during data construction to ensure chain-of-thought correctness, while instructing the generated reasoning not to reveal that reference.The reasoning must appear to derive the answer from the question and audio content.
- A.3 Prompt of Stage 3 when Processing Data (Sample from AudioSet): Stage 3 asks data reviewers to check whether the audio description, questions, answers, and thought process are correct and free of hallucinations.The reviewer returns True when no issues are found and False when errors occur.
B.1 Synthetic Data Introduction
The synthetic data introduction describes Multi-Speaker and Complex Audio datasets designed to expose Audio-Reasoner to challenging speaker interactions and intricate temporal or concurrent sound compositions.
- Multi-Speaker Dataset: The Multi-Speaker dataset uses commercial models and text-to-speech synthesis to create diverse multi-turn conversations among multiple speakers.It is intended to train navigation of intricate speaker interactions in natural conversations.
- Complex Audio Dataset: The Complex Audio dataset combines materials from AudioSet, AudioCaps, and an internal library to represent temporally complex and concurrent audio.Its tasks include sorting or counting sequential sounds and distinguishing multiple simultaneous sound sources.
- Synthetic Data Introduction: Examples for the Multi-Speaker and Complex Audio datasets are provided in separate sections to illustrate their respective tasks.
B.2 Sample from Complex Audio Dataset
The Complex Audio example shows a structured reasoning trace for identifying multiple sounds and ordering them before selecting a matching answer option.
- B.2 Sample from Complex Audio Dataset: The reasoning begins by repeatedly listening for sound identities and comparing the resulting sequence with the answer choices.The analysis identifies glass breaking, a cow, a cat, and a chainsaw as candidate sounds.
- B.2 Sample from Complex Audio Dataset: The trace assigns a definite identity and position to each of four sounds before evaluating the answer options.It places glass breaking first, cow second, cat third, and chainsaw fourth.
- B.2 Sample from Complex Audio Dataset: The summarized sequence is glass break, cow, cat, chainsaw, which matches option (a).
B.3 Sample from Multi-Speaker Dataset
The Multi-Speaker example reasons from dialogue content and vocal differences to determine how many speakers participate in an audio conversation.
- B.3 Sample from Multi-Speaker Dataset: The analysis identifies two hikers and plans to distinguish their individual lines and vocal characteristics.
- B.3 Sample from Multi-Speaker Dataset: The caption differentiates the speakers through their stances, utterances, and vocal tones within a hiking-related dialogue.One speaker favors continuing toward the summit, while the other expresses caution about conditions.
- B.3 Sample from Multi-Speaker Dataset: The reasoning confirms two speakers by noting consistent vocal tones, alternating dialogue, and no extra voices or interruptions.
- B.3 Sample from Multi-Speaker Dataset: The final response reports two distinct speakers and selects answer B.
C Further Dataset Analysis
CoTA spans varied reasoning complexities and audio durations across tasks, from concise translation and emotion-recognition contexts to intricate multi-speaker and music-understanding scenarios.
- 327–1,454 tokens span CoTA’s reasoning processes, covering concise CoVoST2 examples through highly intricate Multi-Speaker tasks.The reported range runs from 327 tokens for CoVoST2 to 1,454 for Multi-Speaker.
- 192.96 average tokens characterize complex-audio reasoning, whose maximum reaches 423 tokens.The complex-audio task is described as requiring deep logical inference and having a balanced token distribution.
- 164.48–481.57 average token lengths align with task difficulty, with longer chains in sound description and music understanding.AudioSet sound description averages 395.26 tokens, while MusicBench music understanding averages 463.89.
- 2.85–26.34-second audio durations support contexts ranging from concise MELD reasoning to extended Multi-Speaker multi-turn inference.MELD averages 2.84 seconds, whereas Multi-Speaker averages 26.34 seconds.
D More Case Studies
The case studies present Audio-Reasoner responses to music, environmental sound, multi-speaker speech, complex audio, and a buzzing-plastic identification question.
- A music prompt asks how vocals, synthesizers, and percussion shape texture and dynamic progression from the intro to the main rhythm.
- A music-based case study illustrates Audio-Reasoner answering a music question.
- Another example asks listeners to infer a dog’s likely environment and activities from the sounds present in the audio.
- A sound-based case study illustrates Audio-Reasoner answering a sound question.
- A multi-speaker speech case study illustrates Audio-Reasoner answering a question involving multiple speakers.
- One example asks which background sound accompanies buzzing plastic, with options including a washing machine, rain, a toilet, or waves.
- A complex-audio case study illustrates Audio-Reasoner answering a complex-audio question.