Source-linked AI summary
Derivation Depth Exposes Reasoning Failures in Tool-Augmented Music Analysis
Rupali Rajesh, Sethuraman T V, Savya Khosla, Srinidhi Sunkara, Aditya Shanmugham, Rohan Maheshwari, Derek Hoiem
TL;DR
Audio-language models struggle to recover musical properties from audio, and MIR tools can provide measurements for higher-level questions. The paper varies how many reasoning steps separate a tool report from the answer, finding that models often follow stated conclusions but struggle to derive them from measurements.
Problem
Audio-language models recover basic musical properties from audio only near chance, while evaluations of tool-augmented systems can conflate tool errors with failures to reason over correct outputs.
Source: §1 · p. 1Method
The authors hold musical information fixed while varying derivation depth, testing five models with controlled tasks, real tool outputs, and real recordings.
Source: §1 · p. 1, §2.1 · p. 2Results
Models were near-perfect when reports stated conclusions but followed deliberately inverted conclusions; accuracy declined as more reasoning from measurements was required.
Source: §3 · p. 3Takeaways & Limitations
Tool-augmented systems should be evaluated for how much derivation they leave to models and whether models reliably interpret and reason over returned outputs.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Audio-language models remain unreliable at extracting basic musical properties from audio, motivating pipelines that place a music information retrieval (MIR) tool between the audio and the model. The tool reports measurements such as keys, chords, and structural similarity, and the model is expected to derive the answer. We test whether it can. Holding the musical information fixed, we vary derivation depth: the number of reasoning steps between the tool output and the answer. When the report states the conclusion, all five models score near 100%, and they follow it even when it is wrong. One step away, accuracy drops. On modulation, open models fall to 57-81%, although they answer that single step correctly when asked directly, and reach 0% at greater depths. Frontier models degrade less, but still degrade. Adding the audio does not recover the loss. The bottleneck in tool-augmented music analysis is not access to musical information but reasoning from it, so tool design must account for how much derivation it leaves to the model.
1. INTRODUCTION
Audio-language models struggle to recover musical properties from audio, motivating the use of MIR measurements for higher-level questions. This paper isolates whether models can derive answers from those measurements and finds failures even when the needed musical knowledge is available.
1. INTRODUCTION
- MIR tools can supply keys, chords, or structural similarity so models can answer higher-level questions without recognizing every property directly from audio.Audio-language models achieve near-chance performance on basic musical-property recovery from audio [1] [2].Source: §1 · p. 1
- The study isolates reasoning over tool output because existing tool-augmented evaluations conflate tool errors with failures to derive answers from correct measurements.A tool may report a new key, while the question asks whether the modulation is to a major or minor key.Source: §1 · p. 1
- Derivation depth measures the number of inferential steps from a tool measurement to the answer, from a stated conclusion to raw measurements.The evaluation spans five models and five music-analysis tasks, using controlled stimuli and real recordings.Source: §1 · p. 1
- Models often fail at the first derivation step: three open models fell from 100% to 57–81% on modulation despite answering the underlying major/minor question correctly when asked directly.Providing the chord reduced accuracy to 0–26%, and providing only its pitches reduced it to 0%.Source: §1 · p. 1
- Writing out an intermediate step can help when the network forms the needed information, but it does not rescue deeper derivations and can discard information needed for the final answer.Source: §1 · p. 2
- The results identify reliable reasoning over tool outputs as a missing capability; improving the MIR tool alone is insufficient when models cannot transform measurements into requested conclusions.With audio access, modulation answers never become decodable inside the network, while structure answers emerge midnetwork but fade before output.Source: §1 · pp. 1–2
2. METHOD AND EXPERIMENTS
The experiments hold musical information fixed while varying its format, the presence of audio, and the reasoning distance to the answer. They combine controlled task pairs with real-tool and real-recording tests, then probe model knowledge, internal representations, and a two-call mitigation.
2.1. Setup
- Each task uses minimal pairs differing in one musical property, and the battery covers cadence, modulation, scale, motif, and structure.Synthetic scores are rendered as piano audio with exact ground truth; each task has six compositions and up to 30 pairs, or 23 for scale.Source: §2.1 · p. 2
- Six formats express the same musical information, from stated conclusions to increasingly cluttered or structured tool output; a flipped conclusion tests whether models follow an incorrect report.Source: §2.1 · p. 2
- The derivation ladder reports the same fact at four depths, from a stated conclusion to the pitches composing its evidence; real-tool formats and GuitarSet recordings extend the evaluation beyond synthetic inputs.The ladder uses 72 items per task, while the GuitarSet test uses 60 recordings with expert key annotations.Source: §2.1 · p. 2
- The evaluation tests measurement-only and audio-crossed inputs, scores matched pairs, and uses knowledge and mechanistic probes to separate missing facts from failures to combine or use them.Pair accuracy requires both alternatives to be correct; chance is 25%, and automatic keyword scoring was validated against manual scoring.Source: §2.1 · p. 2
- A two-call mitigation asks the model to interpret the measurement first, then answer using only its own statement with the original measurement withheld.The approach tests whether forcing an intermediate step into the transcript helps when derivation is not performed.Source: §2.1 · p. 3
3. RESULTS AND DISCUSSION
Accuracy declines as the tool output requires more derivation, even when the needed musical facts are known or audio is available. Models also follow incorrect stated conclusions, while probes reveal that derivations may never form or may fade before output.
Finding 1
- When tool outputs state a conclusion, models are near-perfect, but flipping that conclusion drops every model to 0% accuracy, even with audio.The models follow the incorrect verdict rather than correcting it from the audio.Source: §3 · p. 3
Finding 2
- Across the three open models, modulation accuracy is 100% when the report states the conclusion (d0), falls as low as 57% when only the new key is given (d1), and drops to 0% with only its pitches (d3).Gemini 2.5 Flash scores 15% at d2, while Gemini 3.6 Flash scores 86% and 82% at d2 and d3, respectively.Source: §3 · p. 3
Finding 3
- All three open models identify a named key’s major/minor quality with 100% accuracy alone, yet score as low as 57% when applying it to modulation.At deeper depths, both missing theory knowledge and difficulty combining known facts appear.Source: §3 · p. 3, Table 3 · p. 4
Finding 4
- Audio does not recover failures at deeper derivation depths; measurement-style outputs from a running analyzer reduce open-model accuracy by 54–73 points.On GuitarSet, every model follows a wrong key annotation 33–100% of the time.Source: §3 · pp. 3–4, Table 4 · p. 4
Finding 5
- For modulation, the answer is not reliably decodable at any layer; for structure, it becomes decodable on 93% of items midnetwork, then weakens toward output.Qwen2.5-Omni reaches 90% on structure at d2, where its internal representation is stronger.Source: §3 · p. 4, Figure 3 · p. 4
Finding 6
- Two-call reasoning helps some cases but does not extend reasoning depth: modulation d3 stays at 0%, while structure d1 falls from 100% to 62%.Structure d2 also drops from 90% to 38% when the intermediate summary omits needed score information.Source: §3 · p. 4, Table 5 · p. 4
4. CONCLUSION
Current audio-language models often copy tool-stated conclusions rather than reliably deriving conclusions from tool outputs; the authors recommend evaluating and training for correct tool reading as well as tool calling.
- Models often copy stated tool conclusions but rarely derive conclusions, with the loss growing as more reasoning steps are left to them.Source: §4 · p. 4
- Tool-augmented audio systems should be evaluated on whether they retrieve the right information and reliably perform the derivation left to the language model.Source: §4 · p. 4
- Post-training could reward correct tool reading as well as correct tool calling, using derivation-depth examples and flipped-conclusion controls as supervision or reward signals.Source: §4 · p. 4
- Future work will extend these ideas to broader model families and musical systems, including non-Western traditions.Source: §4 · p. 4
6. COMPLIANCE WITH ETHICAL STANDARDS
The authors report the data and human-baseline provenance, review status, conflicts, and generative-AI assistance for the paper.
- Non-synthetic audio comes from GuitarSet under CC BY 4.0; the other stimuli are synthetic renders of generated scores without third-party rights.Source: §6 · p. 5
- The human listening baseline came from one consenting adult musician, with no personal data recorded, and no IRB review was required.Source: §6 · p. 5
- The authors declare no conflicts of interest.Source: §6 · p. 5
- Claude assisted with figure generation and manuscript grammar and phrasing, while the authors reviewed the content and take responsibility for its correctness.Source: §6 · p. 5