PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Reasoner: Improving Reasoning Capability in Large Audio Language Models

Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, Chunyan Miao

arXiv:2503.02318v2cs.SDcs.AIcs.CLcs.LGcs.MMeess.AS

TL;DR

Chain-of-thought reasoning in audio remains underexplored in large audio language models. Audio-Reasoner trains on CoTA, a dataset of around 1.2 million structured reasoning samples, and achieves +25.4% over Qwen2-Audio-Instruct on MMAU-mini.

  • Problem

    Audio chain-of-thought remains underexplored: existing large audio language models lack CoT reasoning at scale, and reported zero-shot methods have not significantly improved complex-query performance.

    Source: §1 · p. 2

  • Method

    Audio-Reasoner uses commercial models to build CoTA, a 1.2-million-pair dataset with structured CoT annotations, then fine-tunes a large audio language model on it.

    Source: §1 · pp. 2–3

  • Results

    +25.4% over Qwen2-Audio-Instruct on MMAU-mini, with reported improvements across reasoning, translation, and emotion recognition tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that reasoning-rich datasets and inference scaling have a critical role in multimodal learning, particularly for audio-based tasks.

    Source: §5 · p. 13

Abstract

from arXiv · show

Recent advancements in multimodal reasoning have largely overlooked the audio modality. We introduce Audio-Reasoner, a large-scale audio language model for deep reasoning in audio tasks. We meticulously curated a large-scale and diverse multi-task audio dataset with simple annotations. Then, we leverage closed-source models to conduct secondary labeling, QA generation, along with structured COT process. These datasets together form a high-quality reasoning dataset with 1.2 million reasoning-rich samples, which we name CoTA. Following inference scaling principles, we train Audio-Reasoner on CoTA, enabling it to achieve great logical capabilities in audio reasoning. Experiments show state-of-the-art performance across key benchmarks, including MMAU-mini (+25.42%), AIR-Bench chat/foundation(+14.57%/+10.13%), and MELD (+8.01%). Our findings stress the core of structured CoT training in advancing audio reasoning.

1 Introduction

The introduction presents Audio-Reasoner and the 1.2-million-sample CoTA dataset as an approach to audio reasoning, where CoT has been underexplored and simple datasets limit complex reasoning. It reports improvements over Qwen2-Audio-Instruct across audio reasoning, translation, and emotion-classification benchmarks.

  • Audio CoT remains underexplored because short, simple dataset labels limit complex and long-form reasoning, while zero-shot methods can fail on complex queries and cause hallucinations or degraded performance.Source: §1 · p. 2
  • Audio-Reasoner is designed for long-context audio reasoning, trained with a 4K-token context window and structured CoT reasoning that can exceed 1K tokens.Its reasoning framework proceeds through planning, captioning, step-by-step reasoning, and a concise, grounded summary.Source: §1 · p. 2
  • CoTA contains 1.2 million refined captions and question-answer pairs spanning multiple audio datasets and tasks, with filtering for diversity and quality.The dataset is intended to support structured reasoning and audio-language pretraining.Source: §1 · pp. 2–3
  • +25.4% on MMAU-mini, +14.6% on AIR-Bench chat, +30.6% on CoVoST2(ZN/EN subset average BLEU), and +8.01% on MELD versus Qwen2-Audio-Instruct.MMAU-mini reasoning-subtask gains were +44.4%, +26.1%, and +9.3%.Source: §1 · pp. 2–3
  • A scalable generation pipeline uses commercial models to create complex reasoning-based QA pairs and structured CoT annotations for training.Source: §1 · pp. 2–3

2 Related Work

Prior work has developed CoT across LLMs and multimodal models, but audio CoT remains nascent; LALMs primarily target audio understanding and real-time dialogue, with limited exploration of reasoning.

Chain-of-Thought Reasoning

Multimodal Chain-of-Thought

Large Audio Language Models

3 Audio-Reasoner

Audio-Reasoner is trained to produce structured reasoning before an answer, using CoTA, a large-scale dataset built through annotation, reasoning-chain construction, and validation.

3.1 Model Training with Audio Reasoning

  • Audio-Reasoner generates a chain of thought followed by a final response, rather than a single output.Its training objective maximizes the likelihood of both components; at inference, it plans, extracts relevant audio content, reasons step by step, and summarizes. The authors describe improved interpretability and reduced hallucinations versus the direct-response counterpart [Chu et al., 2024].Source: §3.1 · pp. 4–5

3.2 Systematic Data Preparation for Audio Reasoning

  • CoTA contains 1.2 million reasoning-focused samples spanning audio, speech, and music.The dataset was designed for complex reasoning-based question answering and is built from simple human-labeled annotations.Source: §3.2 · p. 5

3.2.1 Multistage Data Generation Pipeline

  • CoTA’s generation pipeline enriches annotations, creates questions and answers, constructs structured reasoning chains, and validates the resulting data.External closed-source models generate evidence-based descriptions and diverse questions; explicit markers such as <THINK> and <REASONING> support multi-step chains, which are checked against the audio and earlier outputs for accuracy and coherence.Source: §3.2.1 · p. 6

3.2.2 Task Taxonomy: Categories and Reasoning Methodologies

  • CoTA covers sound-based and speech-based question answering, emotion recognition, speech-to-text translation, and music question answering.The described reasoning includes analyzing sound characteristics, speaker timbre and speech content, emotion and translation, and—in music—attributes such as tonality, tempo, and emotion before deeper inference.Source: §3.2.2 · pp. 6–7

3.2.3 Conclusion and Next Steps

  • The authors summarize the pipeline as enriching audio annotations, generating questions, building explicit reasoning chains, and validating the data.They point to analysis of the final CoTA dataset and its reasoning capabilities in the following section.Source: §3.2.3 · p. 7

3.3 CoTA Dataset Analysis

  • The authors frame CoTA’s analysis around broad audio coverage and scalable reasoning complexity to address gaps in audio-language pretraining.The dataset spans real-world and synthetic scenarios and aligns task difficulty with structured inference patterns.Source: §3.3 · p. 7
  • CoTA combines speech (38.33%), music (14.12%), and environmental sounds (47.55%) across 10 task categories.Synthetic Multi-Speaker and Complex Audio samples make up 14.15% of the dataset, while most samples come from real-world sources; tasks range from classification to translation and irony detection.Source: §3.3 · p. 7
  • CoTA’s response lengths vary with task complexity, from concise S2TT answers to extended reasoning for demanding tasks.Most responses are 300–500 words, Multi-Speaker responses can reach 1,500 words, and S2TT responses are typically 100–200 words. Figure 4 shows response-length distributions across nine CoTA sub-datasets.Source: §3.3 · pp. 7–8, Figure 4 · p. 8

4 Experiments

Audio-Reasoner is evaluated across audio reasoning, conversation, translation, and emotion tasks, and reported to outperform prior systems. The paper attributes its results to CoTA’s broad task coverage and structured contextual inference.

4.1 Experimental Setup

  • Audio-Reasoner uses 8.4 billion parameters and was fine-tuned for one epoch on CoTA; evaluation spans closed-form, conversational, translation, and emotion tasks against closed- and open-source baselines.The evaluation also includes speech-to-text translation on CoVoST 2 and speech emotion recognition on MELD; Qwen2-Audio-Instruct is the base model and a comparison baseline.Source: §4.1 · p. 8

4.2 Main Results

  • 61.71% on MMAU-mini and 7.94 on AIR-Bench chat made Audio-Reasoner the highest-scoring model, ahead of GPT-4o (57.30%) and Gemini-1.5-Pro (5.70), respectively.The paper attributes its performance to CoTA’s balanced sound, speech, and music coverage and its structured logical and contextual inference.Source: Table 4 · p. 9, §4.2 · pp. 9, 10, 12, Table 5 · p. 10
  • 64.30% in MMAU-mini music exceeded GPT-4o (60.77%) and Gemini-1.5-Pro (49.40%); Audio-Reasoner also led open-source models overall by 12.51 percentage points.The next-best open-source model, Qwen2-Audio-Instruct, scored 49.20% overall.Source: §4.2 · pp. 9–10, Table 4 · p. 9
  • 7.94 overall on AIR-Bench chat topped Gemini-1.5-Pro (5.70) and Whisper+GPT-4 (7.54), while music and speech scored 8.05 and 8.19.The mixed-audio score was 6.65, and the overall score exceeded Qwen2-Audio’s 6.93 by 1.01 points.Source: Table 5 · p. 10, §4.2 · p. 10
  • 65.2 overall on AIR-Bench foundation topped Qwen-Audio-Turbo (59.2), while Audio-Reasoner led SER (60.5), SIC (88.1), and SNV (56.3).The benchmark covers sound, music, and speech, including three speech subdomains.Source: Table 6 · p. 10, §4.2 · p. 11
  • 50.87 average BLEU on EN-ZN beat Gemini-1.5-pro (46.24) and Qwen2-Audio-Instruct (37.07); 29.13 on ZN-EN exceeded them at 26.39 and 24.18, respectively.Audio-Reasoner also scored 33.95 BLEU-4 on EN-ZN and 11.62 BLEU-4 on ZN-EN.Source: Table 7 · p. 11, §4.2 · p. 11
  • 53.9 unweighted accuracy on MELD surpassed the previous state-of-the-art Qwen2-Audio-Instruct (49.9) by 4.0 percentage points.The task evaluates speech emotion recognition.Source: Table 8 · p. 11, §4.2 · p. 11

4.3 Case Study

  • In the case study, Audio-Reasoner interprets a guitar riff alongside spoken commentary as an informal instructional or demonstrative context rather than a formal performance.It analyzes musical features, commentary tone and intent, and the relationship between them.Source: Figure 5 · p. 12, §4.3 · p. 12

5 Conclusion

Audio-Reasoner combines inference scaling and structured chain-of-thought reasoning with CoTA, a dataset of around 1.2 million structured reasoning samples. It achieves state-of-the-art results on three reported benchmarks.

5 Conclusion

  • Audio-Reasoner uses inference scaling and structured chain-of-thought reasoning, supported by CoTA, a dataset of around 1.2 million structured reasoning samples.CoTA was generated through annotation refinement, question synthesis, and chain-of-thought generation.Source: §5 · p. 13
  • MMAU-mini (+25.04%), CoVoST 2 (+8.31%), and MELD (+8.01%) achieved state-of-the-art performance.The conclusion attributes these results to the effectiveness of structured reasoning in audio.Source: §5 · p. 13
  • Future research directions include multi-turn dialogue reasoning, cross-modal knowledge integration, and real-world speech-driven AI applications; the authors plan to publicly release the model, dataset, and code.The authors aim for the release to foster further innovation and collaboration in audio reasoning.Source: §5 · p. 13

A Prompt Details

The appendix specifies a staged prompting workflow for generating audio descriptions, questions, and structured reasoning, then reviewing the resulting data for errors and hallucinations. Prompts emphasize explicit instructions, fixed output formats, and grounding in the audio.

A Prompt Details

  • Prompts use a clear task definition, a structured example, and precise formatting rules to make outputs consistent and reproducible.The examples use delimiters such as <caption> and <question1>.Source: A Prompt Details · p. 18

A.1 Prompt of Stage 1 when Processing Data (Sample from AudioSet)

  • Stage 1 asks for a detailed, accurate audio description followed by three progressively challenging questions with answers.Questions may be open-ended or single-choice; single-choice questions require four options and an analyzed answer in the specified format.Source: §A.1 · p. 18
  • A Stage 1 sample demonstrates the requested caption-and-question structure, with separate tags for the caption and each question-answer pair.The sample illustrates describing static noise and asking questions about its characteristics and changes over time.Source: §A.1 · p. 18

A.2 Prompt of Stage 2 when Processing Data (Sample from AudioSet)

  • Stage 2 requires a <think> process with planning, captioning, reasoning, and summarizing, followed by a user-facing <response> based on that process.The prompt specifies that <think> is invisible to the user and requires strict adherence to the tagged format.Source: §A.2 · p. 19
  • Stage 2 instructs the model to use the supplied answer only to check reasoning correctness, not to assume prior knowledge or reveal the answer was provided.The response should rely on the audio and question, avoid mentioning the supplied description or answer, and provide detailed, flexible logic.Source: §A.2 · p. 19

A.3 Prompt of Stage 3 when Processing Data (Sample from AudioSet)

  • Stage 3 asks reviewers to judge whether the audio, description, questions, answers, and reasoning are correct, especially checking for hallucinations.Reviewers return <True> when there are no issues and <False> when errors are present.Source: §A.3 · p. 20

B Synthetic Data Generation Pipeline · B.1 Synthetic Data Introduction

The synthetic-data pipeline introduces Multi-Speaker and Complex Audio datasets to train the model on multi-turn speaker interactions and challenging audio compositions. The datasets use synthesized multi-speaker conversations, temporal ordering tasks, and blended audio tracks for sound extraction.

B.1 Synthetic Data Introduction

  • The Multi-Speaker dataset combines diverse conversation text, synthesized speech, and distinct timbres to train navigation of multi-speaker interactions.Commercial models generate conversation text; CosyVoice2 [Du et al., 2024] synthesizes speech using LibriSpeech timbres as prompts.Source: §B.1 · p. 20
  • The Complex Audio dataset draws on AudioSet, AudioCaps, and internal materials to train handling of temporal patterns and concurrent sounds.It includes temporally ordered short clips for sorting or counting, and blended long-form tracks for extracting target sounds.Source: §B.1 · p. 20

B.2 Sample from Complex Audio Dataset

The sample demonstrates structured audio reasoning through repeated listening, sound identification, and sequence matching. The analysis identifies glass breaking, a cow, a cat, and a chainsaw in that order, matching option (a).

B.2 Sample from Complex Audio Dataset

  • The analysis calls for listening multiple times, noting sounds, and comparing their sequence with the answer options.Initial listening suggests shattering, an animal sound, a cat, and mechanical tools; relistening is used to clarify the sounds and their order.Source: §B.2 · p. 21
  • The resolved sound sequence is glass breaking, a cow mooing, a cat meowing, and a chainsaw.The analysis reaches these identifications after relistening.Source: §B.2 · p. 21
  • The identified sequence matches option (a).Source: §B.2 · p. 21

B.3 Sample from Multi-Speaker Dataset

This sample reasons about speaker count in a hiking conversation by distinguishing voices and dialogue, concluding that two speakers are present.

B.3 Sample from Multi-Speaker Dataset

  • One speaker urges continuing toward the summit, while the other recommends turning back because of darkening skies and fading daylight.Their differing positions help distinguish the speakers in the dialogue.Source: §B.3 · p. 22
  • The clip contains two distinct speakers, identified as hikers in a back-and-forth conversation.The sample’s final answer is B.Source: §B.3 · p. 22
  • The voices remain distinct in tone, and the smooth exchange contains no extra voices or interruptions.The eager speaker is described as energetic and determined; the cautious speaker as measured and worried.Source: §B.3 · p. 22

C Further Dataset Analysis

CoTA varies reasoning length and audio duration across tasks, covering concise and intricate reasoning as well as short and extended audio contexts.

C Further Dataset Analysis

  • 327 (CoVoST2) to 1,454 (Multi-Speaker) token lengths span the dataset, while complex audio answers have max = 423 and avg = 192.96 words.The passage describes the complex-audio token distribution as balanced, supporting structured multi-step reasoning without unnecessary redundancy.Source: Figure 6 · p. 23, C Further Dataset Analysis · p. 23
  • Average token lengths range from 164.48 to 481.57, with sound description (AudioSet: 395.26) and music understanding (MusicBench: 463.89) among the longer-reasoning tasks.The passage says average token lengths align with task difficulty.Source: C Further Dataset Analysis · p. 23
  • Audio durations range from 2.85s to 26.34s, including MELD clips at 2.84s and Multi-Speaker sequences at 26.34s.The passage associates shorter clips with concise context-dependent reasoning and extended sequences with complex multi-turn inference.Source: C Further Dataset Analysis · p. 23

D More Case Studies

The case studies present Audio-Reasoner answering questions about music, sound, multi-speaker speech, and complex audio.

  • Figure 7 presents an example of Audio-Reasoner answering a music-based question.Source: Figure 7 · p. 24
  • Figure 8 presents an example of Audio-Reasoner answering a sound-based question.Source: Figure 8 · p. 25
  • Figure 9 presents an example of Audio-Reasoner answering a multi-speaker speech-based question.Source: Figure 9 · p. 26
  • Figure 10 presents an example of Audio-Reasoner answering a complex-audio-based question.Source: Figure 10 · p. 27

Related papers

Loading 2503.02318v2…