Source-linked AI summary
Audio-Maestro: Enhancing Large Audio-Language Models with Tool-Augmented Reasoning
Kuan-Yi Lee, Tsung-En Lin, Hung-Yi Lee
TL;DR
End-to-end audio models can struggle with tasks requiring structured reasoning and precise low-level acoustic analysis. Audio-Maestro lets an LALM invoke external tools and integrate their timestamped outputs; on MMAU-Test, it improves average accuracy over the strongest baseline across three backbones.
Problem
End-to-end audio-language models can struggle with precise low-level computations and tasks requiring structured reasoning, such as chord estimation.
Source: §2.1 · p. 1, §1 · p. 1Method
Audio-Maestro lets an LALM decide whether to answer directly or invoke specialized tools, then integrates their structured outputs with the audio and query.
Source: §1 · p. 1, §3.1 · p. 2Results
Audio-Maestro raises MMAU-Test average accuracy over the strongest baseline across all three backbones, including Gemini-2.5-flash from 67.39% to 71.05% (+4.66%).
Source: §5.1 · p. 3Takeaways & Limitations
Experiments on MMAU show improved reasoning accuracy across multiple state-of-the-art models, while error analysis identifies inaccurate tool outputs as a source of many failures.
Source: §6 · p. 4External tools increase inference time, which may limit real-time applications, and tool errors can propagate to the final output.
Source: §7 · p. 4
Abstract
from arXiv · showhide
Recent advancements in large multimodal models (LMMs) have shown strong capabilities in audio understanding. However, most systems rely solely on end-to-end reasoning, limiting interpretability and accuracy for tasks that require structured knowledge or specialized signal analysis. In this work, we present Audio-Maestro -- a tool-augmented audio reasoning framework that enables audio-language models to autonomously call external tools and integrate their timestamped outputs into the reasoning process. This design allows the model to analyze, transform, and interpret audio signals through specialized tools rather than relying solely on end-to-end inference. Experiments show that Audio-Maestro consistently improves general audio reasoning performance: Gemini-2.5-flash's average accuracy on MMAU-Test rises from 67.4% to 72.1%, DeSTA-2.5 from 58.3% to 62.8%, and GPT-4o from 60.8% to 63.9%. To our knowledge, Audio-Maestro is the first framework to integrate structured tool output into the large audio language model reasoning process.
1. Introduction
Audio understanding combines low-level acoustic analysis with high-level semantic understanding, but end-to-end models can struggle with domain-specific tasks requiring structured reasoning and symbolic precision. Audio-Maestro addresses this gap by letting a large audio-language model invoke specialized tools and integrate their timestamped outputs.
1. Introduction
- Audio understanding requires both low-level acoustic analysis and high-level semantic understanding, including symbolic precision for tasks such as chord estimation.The paper contrasts these requirements with end-to-end models’ difficulties on domain-specific computations.Source: §1 · p. 1
- Audio-Maestro lets a large audio-language model decompose complex queries, decide whether to invoke specialized tools, and integrate their structured, timestamped outputs.The framework combines high-level semantic understanding with tools’ precise low-level acoustic analysis.Source: §1 · p. 1
2. Related Work
Prior work has advanced audio-language reasoning and tool use, but many audio models remain end-to-end and many tool interfaces remain text-oriented. Audio-Maestro extends autonomous tool use to audio analysis, with tools tailored to speech, music, and sound.
2.1. Audio Language Model
- Prior audio reasoning work includes Mellow, Audio-Reasoner, R1-AQA, Omni-R1, and Audio-Thinker, which use lightweight architectures, multiphase inference, or reinforcement learning.The cited approaches target reasoning performance, consistency, or sample efficiency.Source: §2.1 · p. 1
- End-to-end audio-language models often struggle with precise low-level computations such as pitch extraction and can require substantial computational resources.These limitations motivate exploration of tool-augmented reasoning for audio.Source: §2.1 · p. 1
2.2. Tool-Augmented Reasoning
- Prior tool-augmented approaches in language and vision demonstrate that structured tool calls can improve accuracy and compositional reasoning.Examples include ART [10], Toolformer, and visual chain-of-thought approaches.Source: §2.2 · p. 1
- Audio-domain systems have used retrieval or text-based tool interfaces, but these approaches may not capture the temporal and acoustic structure needed for precise audio reasoning.Step-Audio 2 [13] focuses on retrieval, while Speech-Copilot and ToolLLM primarily use text-based interfaces.Source: §2.2 · p. 1
- Audio-Maestro enables an LALM to autonomously invoke a toolkit tailored to speech, music, and sound analysis while leveraging low-level acoustic details.The framework pairs this analysis with high-level semantic understanding.Source: §2.2 · p. 1
3. Method
Audio-Maestro uses a two-phase pipeline: an LALM decides whether to answer directly or call tools, then integrates selected tools’ structured outputs with the audio and query. Its implementation uses zero-shot prompting and returns tool outputs in structured form.
3.1. Overall Pipeline
- The two-phase pipeline first decides whether to answer directly or call tools, then integrates their structured results into the final response.The overview depicts tool outputs as timestamped and combined with the audio and query for final inference.Source: Figure 1 · p. 2, §3.1 · p. 2
- The pipeline is intended to support context-aware, explainable reasoning that bridges low-level perception and high-level symbolic analysis.The model’s decision to answer or call tools can reflect semantic understanding and acoustic cues, including emotion shifts or overlapping speakers.Source: §3.1 · p. 2, §3.2 · p. 2
3.2. Phase 1: Decision-Making
- Given the audio, query, and available tools, the LALM decides whether to answer directly or invoke one or more external tools.The direct-answer option is denoted Ans, while C denotes calling one or more tools.Source: §3.2 · p. 2
3.3. Phase 2: Execution and Integration
- When tools are invoked, selected tools process the same audio and return structured, timestamped outputs that are combined with the audio and query for the LALM’s response.These outputs can represent interpretable features such as emotion trajectories, sound-event durations, or chord progressions.Source: §3.3 · p. 2
- The integration phase is intended to let the model use explicit acoustic evidence alongside symbolic reasoning in complex auditory scenarios.The authors describe this as improving interpretability and robustness.Source: §3.3 · p. 2
3.4. Implementation Details
- The zero-shot implementation prompts the LALM with its audio-expert role, tool descriptions, and the user’s audio file and text query.Invoked tools execute externally and return structured JSON output for the model’s final response.Source: §3.4 · p. 2
4. Experiments Setup
The experiments compare three audio-language backbones on MMAU using accuracy as the metric, and evaluate a toolkit generated with Speech-Copilot. The benchmark covers speech, environmental sounds, and music tasks requiring audio knowledge and reasoning.
4.1. Evaluated Backbones
- The evaluated backbones are DeSTA-2.5, Gemini-2.5-flash [1], and GPT-4o [17], selected for reasoning and tool-invocation capabilities.Source: §4.1 · p. 2
4.2. Benchmark
- MMAU [18] evaluates multiple-choice audio understanding and reasoning across speech, environmental sounds, and music, with accuracy as the reported metric.Its tasks require expert-level audio knowledge and complex reasoning beyond simple classification or transcription.Source: §4.2 · p. 2
4.3. Toolkit
- The toolkit was generated with Speech-Copilot [14], and its 12 tools cover tasks including diarization and chord recognition.The tools return structured, timestamped outputs; Table 1 describes the decomposed tasks and selected models.Source: §4.3 · p. 2, Table 1 · p. 3
- Figure 2 segments the three models’ MMAU results into eight categories and distinguishes MMAU-Test (T) from MMAU-Test-Mini (Tm).Source: Figure 2 · p. 3
5. Result
Across controlled comparisons, Audio-Maestro outperforms baselines both when adding native audio perception and when adding tools to native audio; sampled failures most often stem from tool outputs.
5.1. Main Result
- Audio-Maestro raises accuracy over the strongest baseline on MMAU-Test: 67.39%→71.05% on Gemini-2.5-flash, 60.19%→63.06% on DeSTA-2.5, and 60.82%→63.85% on GPT-4o.The comparison includes Audio Without Tool, Text Only + Tool, and Audio-Maestro, testing native audio perception alongside tool access.Source: §5.1 · p. 3, Table 2 · p. 4
5.2. Native Perception vs. Text-only Tooling
- Compared with Text Only + Tool, Audio-Maestro consistently outperforms across backbones: Gemini-2.5-flash, 58.24%→71.09%; DeSTA-2.5, 52.61%→63.06% (10.45% absolute gain); and GPT-4o, 56.80%→63.85% (7.05% absolute gain).Source: §5.2 · p. 3
5.3. Tool Invocation Contribution
- With raw audio fixed, adding tools improves MMAU-Test accuracy by 4.66% on Gemini-2.5-flash, 2.87% on DeSTA-2.5, and 3.03% on GPT-4o.Tools offload specialized operations, complementing rather than substituting for native audio perception.Source: §5.3 · p. 3
5.4. Analysis of Error Cases
- Across models, most sampled failures were Tool Output Errors; Gemini’s share was 90.0%, in 30 randomly sampled errors per model.The authors identify external-tool reliability as the framework’s primary bottleneck.Source: §5.4 · p. 4, Table 3 · p. 4
6. Conclusion
Audio-Maestro delegates specialized signal analysis to external tools and improves MMAU reasoning accuracy across models; error analysis identifies inaccurate tool outputs as a prominent failure source.
6. Conclusion
- Audio-Maestro delegates specialized signal-analysis tasks to external tools within a modular framework for large audio-language models.The framework combines high-level semantic reasoning with low-level signal operations.Source: §6 · p. 4
- Experiments on MMAU show that the modular design substantially improves reasoning accuracy across multiple state-of-the-art models.Source: §6 · p. 4
- Error analysis indicates that many failures arise from inaccurate tool outputs, making tool robustness a key improvement direction.Source: §6 · p. 4
7. Limitations
The authors identify inference time and tool accuracy as limitations of Audio-Maestro.
7. Limitations
- Integrating external tools increases inference time, which may limit real-time applications.Source: §7 · p. 4
- Framework performance depends on tool accuracy, and external-tool errors can propagate to final outputs.Source: §7 · p. 4
- The authors identify optimizing tool invocation and enhancing tool robustness as future directions.Source: §7 · p. 4
8. Generative AI Use Disclosure
The authors disclose using generative AI tools only for manuscript editing, language polishing, and LaTeX formatting.
8. Generative AI Use Disclosure
- Generative AI tools assisted with editing, language polishing, and LaTeX formatting during manuscript preparation.Source: §8 · p. 4
- The authors state that generative AI tools did not produce any significant conceptual or technical part of the manuscript.Source: §8 · p. 4
- The human authors assume responsibility for the final content, results, and scientific integrity, and consented to submission.Source: §8 · p. 4