PaperlayerView exact revision on arXiv

Source-linked AI summary

Listen-to-Reason: Listen with Experts, Retrieve over a Graph, Reason with LLMs

Pooneh Mousavi, Mirco Ravanelli, Cem Subakan

arXiv:2610.10749v1cs.SD

TL;DR

Large audio-language models require retraining when their audio encoder or LLM changes, and their chain-of-thought does not show what audio drove an answer. Listen-to-Reason routes audio through readable tree nodes and a transcript to a frozen text-only LLM; it outperforms audio LLMs on sound and speaker identity questions while trailing them by a relatively small gap on open reasoning, with about 1,400× fewer trained parameters than Qwen2.5-Omni-7B.

  • Problem

    Audio encoders and LLMs in LALMs are coupled through multi-stage training, so replacing either requires retraining, while their chain-of-thought is a post-hoc account rather than a trace of what the model used.

    Source: §1 · pp. 1–2

  • Method

    L2R uses small heads on frozen expert encoders to retrieve readable tree nodes for each clip chunk, then a frozen text-only LLM answers from those nodes and an ASR transcript.

    Source: §1 · p. 3, §2 · p. 4, §2.3 · p. 5

  • Results

    With a small reader, L2R outperforms audio LLMs on sound or speaker identity questions and trails them by a relatively small gap on open reasoning, while training about 1,400× fewer parameters than Qwen2.5-Omni-7B.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Replacing the deciding node with a distractor overturns 78% of correct SAKURA answers, while injecting the gold node repairs 91% of wrong answers.

    Source: §1 · p. 3

    The tree’s closed vocabulary and short-chunk descriptions cannot yet express counts, event order, musical structure, or overall scene setting, which account for most of the remaining gap.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large audio-language models (LALMs) fuse an audio encoder into a large language model (LLM) through multi-stage training. This coupling means that a new domain or a stronger LLM requires retraining, and their answers cannot be traced to what the model heard: a chain-of-thought is a post-hoc account. We propose Listen-to-Reason (L2R), an interpretable-by-design pipeline that passes audio to the LLM through an explicit, human-readable tree: small heads on frozen expert encoders map each chunk of a clip to semantically meaningful nodes on the tree (for speech, music and environmental sound), and a frozen text-only LLM answers from these nodes and an ASR transcript without hearing the clip. Every answer can therefore be traced to the nodes and transcript it read, and the nodes are causal: replacing the deciding node with a distractor overturns 78% of correct answers on SAKURA. With a 7B reader, L2R outperforms all LALMs we compare against on SAKURA and trails them by 6-12 points on MMAU and MMAR, despite training about 1,400x fewer parameters on orders of magnitude less audio data. However, because any LLM can serve as the reader, we show that a stronger reader narrows this gap without retraining any audio component. A new domain is added with one small head: with five labelled clips per species, it outperforms QLoRA fine-tuning of an LALM on the same clips by 13-26 points.

1 INTRODUCTION

Large audio-language models couple audio perception and language reasoning through costly training, while their chain-of-thought does not reveal what audio evidence drove an answer. Listen-to-Reason routes audio into a readable tree and transcript for a frozen text-only reader, making the evidence inspectable and the components modular.

1 INTRODUCTION

  • The paper identifies two limitations of LALMs: changing the encoder or LLM requires repeating training, and adapting to a new domain can degrade existing knowledge.Their chain-of-thought can also be weakly grounded in audio and vary under hallucination-inducing or adversarial inputs.Source: §1 · pp. 1–2
  • L2R uses a hierarchical tree of named audio concepts, small heads on frozen expert encoders, and a frozen text-only LLM that answers from tree nodes and an ASR transcript.Any LLM can serve as reader, and a stronger reader narrows the gap to LALMs without changing the audio side.Source: §1 · p. 3
  • L2R makes each answer traceable to the retrieved nodes and transcript, and replacing the deciding node overturns 78% of correct answers while injecting the gold node repairs 91% of wrong ones.The reader receives only the printed nodes and transcript, so the trace is the computation rather than a post-hoc account.Source: §1 · p. 3
  • With five clips per species, L2R outperforms QLoRA fine-tuning on the same clips while training 120–240× fewer parameters.Adding both new domains reduces accuracy on SAKURA, MMAU and MMAR by at most 0.6 points.Source: §1 · p. 3
  • L2R trains 7.7M parameters, about 1,400× fewer than Qwen2.5-Omni-7B, using orders of magnitude less audio data.The introduction presents L2R as a transparent alternative to end-to-end LALMs.Source: §1 · p. 3

1.1 RELATED WORK

  • Unlike separate extractor label spaces, L2R uses one hierarchical vocabulary with trained routing heads and tests whether selected nodes relate to the clip and affect the answer.The paper frames the tree as a concept bottleneck for open-ended audio question answering.Source: §1.1 · p. 4

2 METHOD

L2R organizes audio concepts in a hierarchical multimodal tree, routes clip chunks to readable nodes, and lets a frozen text-only LLM answer from that context. New domains are added through lightweight heads on frozen encoders.

2.1 LISTEN: BUILDING THE MULTIMODAL TREE

  • The four-level tree organizes speech, music and environmental sound into regions, attributes, closed classes and open-vocabulary leaves.Domain knowledge informs the attributes, while environmental-sound identities align with the AudioSet ontology.Source: §2.1 · p. 4

2.2 RETRIEVE: ROUTING AUDIO INTO THE TREE

  • For each short chunk, frozen encoders feed small trainable router heads that score tree nodes; class scores combine a node with its leaves, and masked loss trains only labeled attributes.At inference, the router includes a leaf only when confident and otherwise backs off to its class.Source: §2.2 · p. 5

2.3 REASON: ANSWERING FROM THE TREE

  • The reader receives a text context that groups retrieved nodes, adds an ASR transcript for speech or singing, and includes timed detected events for questions about their order, duration or number.Because it sees only text, any frozen LLM can answer, and its response can be traced to or changed by editing the context.Source: §2.3 · p. 5

2.4 ADAPT: ADDING A DOMAIN

  • A new domain becomes an attribute under a region, using a frozen suitable encoder and a head trained on k labeled clips per class.Training on clips without the new domain teaches the head to assign them low probability, keeping the router from entering that domain for those clips.Source: §2.4 · p. 5

3 EXPERIMENTAL DETAILS

The experiments evaluate audio reasoning on four multiple-choice benchmarks, test species identification in two new bioacoustic domains, and compare frozen text readers and audio LLM baselines.

Benchmarks

  • The four reasoning benchmarks are MMAU test-mini, MMAR, SAKURA and the single-audio multiple-choice subset of MMAU-Pro, scored by accuracy.The benchmark sets contain 1,000, 1,000, 4,000 and 4,133 items, respectively.Source: §3 · p. 6
  • The two species-identification tasks cover 48 bird species and 31 marine-mammal species, with four answer choices and 25% chance accuracy.The tasks have 938 bird test items from BirdSet and 678 marine-mammal test items from BEANS.Source: §3 · p. 6

Readers and baselines

  • The study evaluates cached tree-and-transcript contexts with 26 frozen text LLMs and compares them with three audio LLMs that hear the clip.Input ablations use the question, transcript, tree, or tree and transcript; domain adaptation is compared with three uses of the same clips by Omni-7B.Source: §3 · p. 6

4 RESULTS

With Qwen2.5-7B, L2R leads audio LLMs on SAKURA but trails them on MMAU and MMAR. Results also show reader scaling, few-shot domain adaptation, node-based interpretability, and lower training cost alongside higher inference cost.

4.1 MAIN EVALUATION ON REASONING BENCHMARKS

  • 80.1 on SAKURA beats the best audio LLM, Omni-7B, at 73.3, while L2R trails Omni-7B by 6.8 points on MMAU and 12.0 on MMAR.The comparison uses a Qwen2.5-7B reader.Source: §4.1 · p. 6
  • On SAKURA, the tree alone reaches 74.7, 28.3 points above the transcript; together, tree and transcript outperform either channel alone on every benchmark.Source: §4.1 · pp. 6–7

4.2 SCALING THE READER

  • With the same cached contexts, accuracy rises steeply with reader size up to 7B and more slowly beyond; Phi-4 scores 71.7 on MMAU, above Qwen2.5-72B’s 70.5.Source: Figure 2 · p. 7, §4.2 · p. 7

4.3 FEW-SHOT ADAPTATION TO NEW DOMAINS

  • With five clips per species, the small head reaches 79.6 on birds and 65.2 on marine mammals, versus QLoRA’s 54.1 and 52.2 while training 120–240× fewer parameters.Source: Figure 3 · p. 7, §4.3 · p. 8
  • With both domain heads added, 24 readers score 78–82 on birds and 64–68 on marine mammals; original-task scores remain close to their prior levels.For Qwen2.5-7B, SAKURA changes from 80.1 to 79.6, MMAU from 68.3 to 67.7, and MMAR from 51.6 to 51.8.Source: §4.3 · p. 8

4.4 INTERPRETABILITY

  • Audio LLM judges rate served nodes above swapped-category nodes on 97–99% of items, though judges differ in rating scale.Source: Figure 4 · p. 8, §4.4 · p. 8
  • On SAKURA single-hop items, replacing the routed node with a distractor breaks 78.3% of correct answers, while inserting the gold node repairs 90.9% of wrong ones.A case study traces one error to serving Rain instead of Waterfall after the router reached the water class.Source: §4.4 · pp. 8–9

4.5 EFFICIENCY AND RESOURCE USE

  • L2R trains 7.7M parameters, about 1,400× fewer than Omni-7B, on about 0.2M labelled clips, but inference costs 5.3 TMACs and 0.57 s per MMAU item versus 2.7 TMACs and 0.12 s.Source: §4.5 · p. 9

5 CONCLUSION

Listen-to-Reason uses an explicit audio tree and a frozen text reader, with answers traceable to the nodes and transcript provided. Its reported strengths and remaining limitations span identification, open reasoning, and domain extension.

5 CONCLUSION

  • With a small reader, L2R outperforms audio LLMs on sound or speaker identity questions and trails them by a relatively small gap on open sound and music reasoning.The tree and transcript are complementary; a stronger reader narrows the gap without changing the audio component, and a new domain can be added with one small head and a few labelled clips more effectively than by fine-tuning an audio LLM.Source: §5 · p. 9
  • Answers can be traced to the nodes they received, and audio LLMs judge those nodes as related to what is audible.The tree has a closed vocabulary and describes short chunks rather than individual events, limiting its ability to express counts, event order, musical structure, or overall scene setting; these questions fall back on the transcript and reader prior and account for most of the remaining gap.Source: §5 · p. 9
  • The tree inherits errors from its annotators and teacher classifiers, and the reader follows a wrong node rather than correcting it.Inference runs several encoders and an ASR model before the reader; domain addition has been tested only on identification tasks, leaving domains requiring reasoning beyond identity to be explored.Source: §5 · p. 10

ETHICS STATEMENT

The work uses publicly available datasets and models under their licenses and collects no new human-participant data. It notes risks involving speaker attributes and AI-derived annotations, and says healthcare applications were not evaluated.

ETHICS STATEMENT

  • The study uses publicly available datasets and models under their licenses, collects no new participant data, and draws its two bioacoustic evaluation sets from BirdSet and the Watkins collection.Source: ETHICS STATEMENT · p. 10
  • Speaker attributes such as gender, age group, and emotion can be misused for profiling, may be wrong for individuals, and reflect training-data categories that do not represent every speaker.The tree can also inherit biases from audio LLM annotations, though readable nodes allow such errors to be inspected and corrected.Source: ETHICS STATEMENT · p. 10
  • Healthcare applications motivating the work were not evaluated, and L2R is not validated for clinical use.Source: ETHICS STATEMENT · p. 10

AI USE STATEMENT

The AI-use statement describes model assistance in tree construction, evaluation, and paper preparation. The appendix outlines tree construction and implementation details, from attributes and annotation through leaf processing.

AI USE STATEMENT

  • An audio LLM annotated training chunks, a text LLM proposed leaf groups subject to lexical, acoustic, and identity checks, and three audio LLMs judged node relatedness.An AI assistant also helped edit and draft text, suggest structure, check related-work descriptions and bibliography formatting, generate figures, and prepare experiment reports; the authors reviewed the text and checked numbers and citations.Source: AI USE STATEMENT · p. 10

A TREE CONSTRUCTION DETAILS

  • The tree-construction appendix follows the steps of §2.1: defining attributes, annotating training chunks, and processing free-text leaves.Source: A TREE CONSTRUCTION DETAILS · p. 20

A.1 ATTRIBUTES

  • The attribute tables describe each region’s attributes, class and leaf counts, clip- or chunk-level scope, and expert classifier; environmental sound identity classes follow the AudioSet ontology.The environmental-sound identity heads use human labels from AudioSet, FSD50K, and ESC-50, with activity also using VGGSound classifier predictions.Source: §A.1 · p. 20, Table 2 · p. 20

A.2 ANNOTATION

  • Training chunks are labelled by expert classifiers and an audio LLM, which first identifies present regions and then fills a region-specific template with listed classes, unsure, none, and optional short leaves.The templates instruct the annotator to describe only audible content and use “unsure” when uncertain; the environmental-sound identity attributes are multiple-choice and follow AudioSet ontology.Source: §A.2 · pp. 20–21, Table 3 · p. 21

A.3 LEAF POST-PROCESSING

  • Leaf post-processing merges candidate leaves; a text LLM proposes name groups without hearing audio, and an acoustic check compares their chunks in an attribute-suited encoder space.Source: §A.3 · p. 21, Table 4 · p. 22

B IMPLEMENTATION DETAILS

  • The implementation-details section is organized around router training and inference, the reader, compute, and the QLoRA baseline.Source: B IMPLEMENTATION DETAILS · p. 22

B.1 ROUTER TRAINING

The implementation trains small router heads over frozen, cached encoder embeddings, then traverses the tree at inference and supplies selected context to a text reader. It also reports compute settings and the QLoRA baseline configuration.

B.1 ROUTER TRAINING

  • Frozen encoder embeddings are cached; class loss uses a softmax for single-choice attributes and binary cross-entropy for multiple-choice attributes, while leaf loss compares leaves only within the labelled class.Leaves require at least five training clips, or three for emotion and gender; rarer leaves train toward their class node, so the head learns when not to be specific.Source: §B.1 · p. 22

B.2 ROUTER INFERENCE

  • Inference cuts clips into 3-second chunks with a 1.5-second hop, applies region gates, and serves selected classes or leaves using attribute-specific thresholds.The transcript is added when speech or singing is heard, while an event timeline is added only for questions about event order, duration, or number.Source: §B.2 · pp. 22–23

B.3 READER

  • The reader selects an option using context-conditioned and context-free scores; the 26 readers include models from Qwen, Llama, Vicuna, Mistral, Phi, OLMo, Falcon, SmolLM, and gpt-oss.A prompt template is selected on 100 development items per benchmark and then locked for each reader.Source: §B.3 · p. 23

B.4 COMPUTE

  • Experiments ran on single NVIDIA A100 or L40S GPUs, while tree annotation and LLM cleaning used up to four A100-80GB GPUs.A router head trains in a few minutes, a new-domain head in under one minute, and a 7B reader processes all three benchmarks in about five minutes.Source: §B.4 · p. 23

B.5 QLORA BASELINE

  • The QLoRA baseline freezes the audio encoder and trains rank-16 LoRA adapters on the Omni-7B thinker, with 40.4M trainable parameters.Training uses 8, 6, 4, or 2 epochs for k = 5, 10, 20, or 50, respectively, with three epochs at k = 50 for marine mammals.Source: §B.5 · p. 23

C DATASETS

The section identifies the datasets used to build the tree and train the router, and reports dataset roles, sizes, and held-out evaluation sets.

C DATASETS

  • Table 6 lists each dataset’s role and size in tree generation or router training.The tree-generation pool contains 4,054 labelled clips; some router heads use only the pool’s labels.Source: C DATASETS · p. 23, Table 6 · p. 24
  • AudioSet test (9,042 clips) and FSD50K evaluation (10,161 clips) are held out, and no benchmark audio is used.Source: Table 6 · p. 24

D BENCHMARKS AND EVALUATION PROTOCOL

The paper specifies benchmark tasks, scoring, and baseline settings, then examines which attributes and frozen readers support performance.

D BENCHMARKS AND EVALUATION PROTOCOL

  • MMAU, MMAR, and SAKURA test distinct audio reasoning tasks, while the evaluated MMAU-Pro subset excludes unsupported multi-audio, open-ended, and instruction-following items.The evaluated MMAU-Pro items are single-audio multiple-choice; for recordings over three minutes, the router covers the first 60 seconds and the transcript the first 180 seconds.Source: D BENCHMARKS AND EVALUATION PROTOCOL · p. 24
  • Scoring uses official MMAU answers and a shared text-to-option mapping for MMAR; Omni-7B receives the same questions, options, and scoring as L2R.Three items per benchmark are in-context examples and excluded from scoring, leaving 997 MMAU, 994 MMAR, and 3,997 SAKURA items.Source: D BENCHMARKS AND EVALUATION PROTOCOL · p. 25

E WHICH ATTRIBUTES THE READER USES

  • Removing identity attributes harms SAKURA, removing music attributes harms MMAU, and no attribute group changes MMAR by more than 1.3 points.Removing speech-content attributes does not lower SAKURA or MMAU and costs 1.3 points on MMAR; the authors prune them because the transcript already carries that content.Source: E WHICH ATTRIBUTES THE READER USES · p. 25, Table 7 · p. 25

F CHOICE OF READER

  • Reader comparisons use the same cached tree-and-transcript contexts across 26 frozen readers, with prompt templates locked using development items.Table 8 reports accuracy on full sets; its caption defines B as parameters in billions and notes Qwen3-30B-A3B has 3B active parameters.Source: F CHOICE OF READER · p. 25, Table 8 · p. 26

G NEW DOMAINS: EXACT NUMBERS

The section reports few-shot species adaptation results, reader and encoder comparisons, and a boundary case in which added species nodes affect existing benchmarks.

G NEW DOMAINS: EXACT NUMBERS

  • At k=5 labelled clips per species, 24 readers from 1.5B to 72B score 78–82 on birds and 64–68 on marine mammals.The table compares one new head on a frozen encoder and frozen Qwen2.5-7B reader against Omni-7B zero-shot, which scores 32.3 on birds and 25.8 on marine mammals.Source: G NEW DOMAINS: EXACT NUMBERS · p. 26, Table 9 · p. 26
  • The encoder choice matters more than reader size in these domain results: BirdNET is used for birds, while CLAP serves marine mammals without adding an encoder.The authors attribute the reader’s limited role to the domain node carrying the answer.Source: G NEW DOMAINS: EXACT NUMBERS · pp. 26–27
  • With both domain heads added, Qwen2.5-7B scores 79.6 on SAKURA, 67.7 on MMAU, and 51.8 on MMAR.The corresponding scores without the heads are 80.1, 68.3, and 51.6, respectively.Source: G NEW DOMAINS: EXACT NUMBERS · p. 27
  • Species nodes are served on 17–21% of original-benchmark items, and all 19 SAKURA items they break are animal clips assigned a bird species node.The paper identifies a tighter abstention rule for domain heads as the fix.Source: G NEW DOMAINS: EXACT NUMBERS · p. 27

H.1 BY QUESTION TYPE

The analyses identify question types and subcategories where L2R falls behind Omni-7B, assess whether retrieved nodes match audible content, and show example reader contexts.

H.1 BY QUESTION TYPE

  • On counting, L2R scores 47.0 vs 43.0 on MMAU and 33.6 vs 27.0 on MMAR against the blind reader; its three-second chunks do not represent individual event counts.For order and timing, L2R is within three points of Omni-7B on MMAU (60.2 vs 62.8) but 11 items behind on MMAR.Source: §H.1 · p. 27
  • Identity and scene questions expose coarse nodes: the router may reach the right class without the right leaf, or serve several candidate leaves.The largest MMAU losses are 18 items each for identifying a source, instrument, or genre and for describing a scene or activity.Source: §H.1 · p. 27
  • On MMAR, the largest net loss is outside the listed question types (−61 of 447 items); the paper links these errors to speech inference and identifies a stronger reader as a repair.L2R is ahead on MMAU speaker and voice questions by 5 items but 15 items behind on MMAR; it is also ahead by 4 items on MMAR music-theory questions.Source: §H.1 · p. 27

H.2 BY BENCHMARK SUB-CATEGORY

  • Subcategory comparisons show losses in MMAU acoustic-scene reasoning (−12 items) and MMAR environmental perception (−23), alongside gains in MMAU emotion-flip detection (+5) and MMAR music-theory questions (+4).Tables 11 and 12 report the MMAU and MMAR subcategory comparisons against Omni-7B.Source: §H.1 · p. 27, §H.2 · p. 28, Table 11 · p. 28

I AUDIO LLMS AS JUDGES

  • The judge audit samples 400 items each from MMAU and MMAR and 100 multi-hop items per SAKURA track, using audio plus served nodes but not the question or transcript.The swap control draws nodes from a different-category clip, with mean overlap below 1%.Source: I AUDIO LLMS AS JUDGES · pp. 28–29
  • Three audio judges rank served nodes above swapped nodes on most items, and their standardized average preserves that ordering on every benchmark.The judges differ in score scale; Audio Flamingo 3 also rates swapped nodes above 3, while Kimi-Audio gives served nodes the lowest ratings, especially on SAKURA speech tracks.Source: I AUDIO LLMS AS JUDGES · p. 29

J EXAMPLE CONTEXTS

K COST COMPARISON

Table 13 compares the systems’ computational costs and shows that the fewest MACs do not correspond to the shortest per-item time.

K COST COMPARISON

  • 0.93 s per MMAU item is Qwen3-Omni’s highest per-item time, despite its fewest MACs and 35.3B parameters.The table reports per-item time on one A100 GPU.Source: K COST COMPARISON · p. 30, Table 13 · p. 30
  • MACs count executed operations, so the mixture-of-experts model counts only its active parameters.Qwen3-Omni runs about 3B parameters per token.Source: Table 13 · p. 30, K COST COMPARISON · p. 30
  • Table 13 compares Listen-to-Reason with audio LLMs using MACs and seconds per MMAU item on one A100 GPU.Source: Table 13 · p. 30

Related papers

Loading 2610.10749v1…