PaperlayerView exact revision on arXiv

Source-linked AI summary

AuTAgent: A Reinforcement Learning Framework for Tool-Augmented Audio Reasoning

Siqian Tong, Xuan Li, Yiwei Wang, Baolong Bi, Yujun Cai, Shenghua Liu, Yuchen He, Chengpeng Hao

arXiv:2602.13685v1cs.SDcs.AI

TL;DR

LALMs struggle with complex audio reasoning that requires precise acoustic measurements, and their encoders can discard fine-grained features. AuTAgent learns when and which tools to invoke through reinforcement learning with a baseline-subtracted differential reward. It improves accuracy across MMAU Test-mini and MMAR for open- and closed-source backbones.

  • Problem

    LALMs struggle with complex audio reasoning requiring precise acoustic measurements, while their encoders can discard fine-grained and mid-level acoustic features.

    Source: §1 · p. 1

  • Method

    AuTAgent trains an audio tool agent with GRPO and baseline-subtracted differential rewards, using backend reasoner feedback to favor tool-assisted results that improve on direct reasoning.

    Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5

  • Results

    AuTAgent improves accuracy on MMAU Test-mini and MMAR by 4.2% and 6.2% for Qwen2-Audio-7B-Instruct, and by 9.8% and 8.0% for GPT-4o Audio, respectively.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    The learned policy transfers as a plug-and-play module to diverse reasoning backbones without additional training.

    Source: §7 · p. 9

    The reported low-resource training setting uses 2,025 samples randomly subsampled from the AVQA training split.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Large Audio Language Models (LALMs) excel at perception but struggle with complex reasoning requiring precise acoustic measurements. While external tools can extract fine-grained features like exact tempo or pitch, effective integration remains challenging: naively using all tools causes information overload, while prompt-based selection fails to assess context-dependent utility. To address this, we propose AuTAgent (Audio Tool Agent), a reinforcement learning framework that learns when and which tools to invoke. By employing a sparse-feedback training strategy with a novel Differential Reward mechanism, the agent learns to filter out irrelevant tools and invokes external assistance only when it yields a net performance gain over the base model. Experimental results confirm that AuTAgent complements the representation bottleneck of LALMs by providing verifiable acoustic evidence. It improves accuracy by 4.20% / 6.20% and 9.80% / 8.00% for open-source and closed-source backbones on the MMAU Test-mini and the MMAR benchmarks, respectively. In addition, further experiments demonstrate exceptional transferability. We highlight the complementary role of external tools in augmenting audio model reasoning.

1. Introduction

LALMs can perceive general audio content but struggle with reasoning that requires precise acoustic measurements, partly because their encoders discard fine-grained features. AuTAgent learns context-aware tool selection with differential rewards and improves benchmark accuracy across open- and closed-source backbones.

1. Introduction

  • LALMs struggle with complex audio reasoning requiring multi-step deduction and precise acoustic measurement, while their encoders can discard fine-grained and mid-level audio features.The paper identifies this representation bottleneck as a reason models may produce generic descriptions or rely on textual priors rather than auditory facts.Source: §1 · pp. 1–2
  • Naive all-tool invocation and prompt-based selection can underperform the no-tool baseline because irrelevant outputs add noise and tool utility varies with audio complexity and user intent.The paper argues that tool use requires a learned policy for when to invoke which tools.Source: §1 · p. 2
  • AuTAgent uses GRPO and backend reasoner feedback, rewarding tool-assisted results only when they improve on the base model's direct reasoning.This baseline-subtracted differential reward is intended to suppress redundant operations and focus tool use on verifiable net gains.Source: §1 · p. 2
  • AuTAgent improves Qwen2-Audio-7B-Instruct accuracy by 4.2% and 6.2%, and GPT-4o Audio accuracy by 9.8% and 8.0%, on MMAU Test-mini and MMAR, respectively.The framework is also reported to reach 66.80% on MMAU Test-mini and 62.80% on MMAR with GPT-4o Audio as a plug-and-play module.Source: §1 · p. 2

2. Preliminary Analysis: Potential and Pitfalls of Tool Augmentation

Exploratory experiments on MMAU Test-mini compare no-tool, single-tool, all-tool, and theoretical upper-bound settings. The results show substantial tool potential, but indiscriminate invocation can reduce accuracy below the no-tool baseline.

2. Preliminary Analysis: Potential and Pitfalls of Tool Augmentation

  • The comparison evaluates six tools individually, all six together, the frozen base model without tools, and a post-hoc upper bound counting success by any single tool.The six tools are denoted T1 through T6.Source: §2 · p. 3
  • The Oracle Upper Bound reaches 66.1% versus 50.0% for the No-Tool Baseline, while All-Tools falls to 40.9%.The paper interprets the gap as complementary tool knowledge that requires precise invocation to realize.Source: §2 · p. 3
  • Single tools vary in effect: T4 reaches 52.2% and T1 reaches 51.5%, while T2, T3, and T5 degrade performance; the best single tool trails the 66.1% Oracle by about 14%.The authors conclude that no single tool is universally applicable because the optimal choice varies by audio-query pair.Source: §2 · p. 3
  • The authors conclude that external tools hold potential, but indiscriminate integration is detrimental and tool selection should depend on the specific audio-query context.Source: §2 · p. 3

3. Method

AuTAgent trains an audio tool-selection policy through trial-and-error while keeping the reasoning backbone frozen. It compares tool-augmented predictions with a raw-input baseline and uses the resulting performance differences to guide policy updates.

3. Method

  • AuTAgent uses reinforcement learning to train an audio tool agent to select tools and provide structured audio evidence to a frozen Reasoner.The Reasoner processes the audio, query, and tool-augmented output; keeping it frozen allows the learned policy to transfer across reasoning backbones without additional training.Source: §3 · p. 3, Figure 3 · p. 4

3.1. Problem Definition

  • The framework pairs a trainable policy with a frozen Reasoner, which predicts answers from raw audio and a query, optionally augmented with selected tools' outputs.The agent discovers tool combinations without explicit ground-truth labels for tool selection.Source: §3.1 · p. 3

3.2. Policy Optimization via Differential Reward Mechanism

  • The Baseline-Subtracted Differential Reward measures whether tool use improves answer correctness over the Reasoner's raw-input baseline.It rewards correcting a baseline error, gives zero when the baseline is already correct, and penalizes tool use that turns a correct baseline answer incorrect.Source: §3.2 · p. 4
  • GRPO updates the policy using relative advantages across sampled tool-selection candidates, while clipping and a KL term constrain optimization.The group comparison helps distinguish which tool combinations perform better for a specific query.Source: §3.2 · pp. 4–5

4. Experimental Setup

The experiments evaluate AuTAgent using a small AVQA-derived training set and compare it on two audio-reasoning benchmarks. The setup uses a frozen Qwen2-Audio backbone, GRPO training, and a six-tool library spanning speech, music, and environmental audio.

4. Experimental Setup

  • The experimental setup is designed to evaluate AuTAgent’s effectiveness and generalization in audio reasoning tasks.Source: §4 · p. 5

4.1. Dataset Construction

  • AuTAgent is trained on 2,025 randomly sampled AVQA training examples, a micro-training set intended to test learning under low-resource conditions.The source questions are adapted from AVQA by extracting video audio tracks and replacing references to “video” with “audio.”Source: §4.1 · p. 5

4.2. Implementation Details

  • The tool agent uses Qwen2-Audio-7B-Instruct as its policy backbone and is trained with GRPO for 200 steps.Training uses TRL on eight NVIDIA A100 GPUs, with a learning rate of 1 × 10^-6 and six sampled generations per query.Source: §4.2 · p. 5

4.3. Evaluation Benchmarks

  • The evaluation measures multiple-choice accuracy on MMAU Test-mini and MMAR, covering diverse audio domains and complex mixed-audio reasoning.MMAU Test-mini spans speech, sounds, and music; MMAR emphasizes complex mixed-audio scenarios.Source: §4.3 · p. 5

4.4. Modular Tool Library

  • The tool library comprises six tools across speech, music, and environmental audio, converting audio signals into structured evidence for reasoning.Its tools cover transcription, emotion, chords, tempo, pitch, and sound classification.Source: §4.4 · pp. 5–6, Table 1 · p. 5

5. Results and Analysis

AuTAgent outperforms comparison methods across open- and closed-source backbones and transfers effectively between them. Its learned policy shifts tool calls toward useful acoustic evidence, while the differential reward reduces redundant calls; case studies illustrate tool use across audio domains.

5.1. Main Results

  • 54.20% on MMAU Test-mini and 41.30% on MMAR with Qwen2-Audio-7B-Instruct; 66.80% on MMAU Test-mini and 63.00% on MMAR with GPT-4o Audio.The paper reports AuTAgent as the best-performing method for both backbones; Table 2 compares it with four baselines.Source: §5.1 · p. 6

5.2. Transferability Analysis

  • 9.4% on MMAU Test-mini and 7.8% on MMAR are the reported Plugin improvements for GPT-4o Audio over the Vanilla baseline, with Plugin-to-Self gaps of 0.2%–0.8%.The Plugin policy achieves 66.80% on MMAU Test-mini and 62.80% on MMAR, versus Self scores of 67.20% and 63.00%, respectively.Source: §5.2 · pp. 6–7, Table 3 · p. 7

5.3. Policy Evolution and Distributional Shift

  • No-tool calls fall from 234 to 41, T4 calls rise from 149 to 570, and suboptimal T2 calls fall from 413 to 90 after RL training.Figure 4 contrasts the description-based agent’s tool-call distribution with AuTAgent’s.Source: §5.3 · p. 7, Figure 4 · p. 7

5.4. The Necessity of Differential Reward

  • Binary Reward yields marginal accuracy gains over Vanilla but is surpassed by AuTAgent in accuracy and efficiency, with average tool calls of 1.62 versus 1.30.Differential reward assigns zero reward when the baseline prediction is already correct, penalizing zero-gain tool use.Source: §5.4 · p. 7, Table 4 · p. 7

5.5. Case Study

  • Case studies show AuTAgent using chord recognition for music, ASR to correct a speech count, and sound classification for overlapping or low-level acoustic details.The examples illustrate specialized tools supplying domain-specific information for audio-question pairs.Source: §5.5 · pp. 7–8, Table 5 · p. 8

6. Related Work

Prior LALMs perform strongly on perception but can lose fine-grained acoustic details needed for complex reasoning. Existing audio tool agents largely use heuristic selection, while this work applies reinforcement learning to the external tool-action space.

Related Work

  • LALMs such as SALMONN (Tang et al., 2023) and Qwen-Audio (Chu et al., 2023) use audio-text training for perception, but encoders can discard fine-grained acoustic details.The cited examples include pitch and chord structures; this loss is associated with errors on tasks requiring precise measurement or complex reasoning.Source: §6 · p. 8
  • AudioGPT (Huang et al., 2024a) and HuggingGPT (Shen et al., 2023) use training-free tool scheduling, while indiscriminate or prompt-based selection can add noise or negative transfer.The passage contrasts these heuristic approaches with the need to learn dynamic trade-offs for complex acoustic scenarios.Source: §6 · p. 8
  • Unlike RL work focused on internal reasoning chains, AuTAgent applies GRPO to external tool actions to learn tool-composition paths despite uncertain tool gain and scarce supervised data.The comparison cites DeepSeek-R1 (Guo et al., 2025), R1-AQA (Li et al., 2025), and Audio-Thinker (Wu et al., 2025).Source: §6 · p. 8

7. Conclusion

AuTAgent uses a baseline-subtracted differential reward within a GRPO framework to learn context-aware tool combinations through trial and error. It improves accuracy across the reported benchmarks and transfers to diverse reasoning backbones without additional training.

  • AuTAgent uses a baseline-subtracted differential reward mechanism within a GRPO framework to address representation bottlenecks and noise from improper tool integration.The design enables discovery of effective, context-aware tool combinations through trial-and-error interaction.Source: §7 · pp. 8–9
  • Accuracy gains were 4.2% and 6.2% on MMAU Test-mini and MMAR for open-source backbones, and 9.8% and 8.0% for closed-source models such as GPT-4o Audio.Source: §7 · p. 9
  • The decoupled design makes the learned policy transferable as a plug-and-play module for diverse reasoning backbones without additional training.Source: §7 · p. 9

A. Prompt Details

The prompt designs specify the agent’s role, available information, and required output format across tool selection and reasoning. The appendix also describes a modular structure intended to generalize across open-source and proprietary audio-language models.

A. Prompt Details

  • The prompts combine role-specific task definitions, modular information integration, and structural constraints, with a format intended to generalize across open-source and proprietary audio-language models.Source: A. Prompt Details · p. 11

A.1. Tools w/Desc. Prompt

  • The tool-selection prompt offers six independent tools for speech recognition, emotion, chords, tempo and beats, pitch, and audio-event detection.Source: §A.1 · p. 11
  • The selection instruction asks the agent to analyze the audio and query, choose helpful tool indices, and return only a comma-separated list in answer tags.Source: §A.1 · p. 11

A.2. AuTAgent Prompt

  • The AuTAgent selection prompt substitutes six type labels for tool names while retaining the instruction to return selected indices in answer tags.Source: §A.2 · p. 12

A.3. Tool-Augmented Reasoner Prompt

  • The tool-augmented reasoner receives extra information from audio analysis and must answer using the audio and that information within answer tags.Source: §A.3 · p. 12

A.4. Baseline Reasoner Prompt

  • The baseline reasoner prompt asks for an answer based on the audio, constrained to answer tags.Source: §A.4 · p. 12

B. Tool Output Json Format

The appendix illustrates tool outputs as compact JSON-like records, with values tailored to each audio-analysis function. The examples include text, labels, timestamped musical or pitch values, and a scalar tempo estimate.

B.1. Automatic Speech Recognition Tool Output Format

  • The speech-recognition output example stores a transcript as the value of an output field paired with the automatic-speech-recognition tool name.Source: §B.1 · p. 12

B.2. Emotion Recognition Tool Output Format

  • The emotion-recognition example returns the label “sad” in the output field.Source: §B.2 · p. 13

B.3. Chord Recognition Output Format

  • The chord-recognition example records chord values with timestamp intervals, including A:maj, G:maj, and D:maj.Source: §B.3 · p. 13

B.4. Tempo Estimation Tool Output Format

  • The tempo-estimation example returns a single numeric tempo value, 117.19.Source: §B.4 · p. 13

B.5. Pitch Tracking Tool Output Format

  • The pitch-tracking example pairs timestamps with fundamental-frequency values in hertz, including 520.91 Hz at 0.0 seconds.Source: §B.5 · p. 14

B.6. Sound Classification Tool Output Format

  • The sound-classification example returns event labels including basketball bounce, walking, footsteps, and speech.Source: §B.6 · p. 14

Related papers

Loading 2602.13685v1…