PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaLoop: Adaptive-Depth Latent Reasoning for Audio Language Models

Lee Seung-woo, Bowen Qi

arXiv:2610.06949v1cs.SDcs.LG

TL;DR

Large audio language models perform unevenly on fine-grained acoustic perception, while using fixed inference depth across questions. AdaLoop applies question-guided recurrent refinement with adaptive halting; it improves average accuracy across three backbones, with larger gains on perception-heavy categories.

  • Problem

    Audio language models understand meaning better than acoustic details, and current inference passes use the same depth regardless of question difficulty.

    Source: §1 · p. 1

  • Method

    AdaLoop iteratively refines audio representations in a question-guided recurrent block and learns when to halt for each audio–question pair.

    Source: §1 · p. 1, §2 · p. 1, §2.2 · p. 2

  • Results

    AdaLoop raises average accuracy over Base by 3.8 points on Qwen2.5-Omni, 3.4 on Kimi-Audio, and 2.9 on MiMo-Audio, leading every benchmark for each backbone.

    Source: §3.2 · p. 3

  • Takeaways & Limitations

    Gains concentrate in perception-heavy categories, while semantic categories show smaller but consistently positive gains.

    Source: §3.2 · p. 3

    AdaLoop has less headroom on tasks where a single coarse pass already suffices.

    Source: §4 · p. 4

Abstract

from arXiv · show

Large audio language models answer questions about speech, sound, and music, yet their accuracy drops sharply on tasks that need fine-grained acoustic analysis. Judging which of two speakers has the higher pitch demands iterative signal-level reasoning that a content question does not. Current models spend the same computational depth on both. We introduce AdaLoop, a lightweight recurrent module that learns how many latent refinement steps a given audio--question pair requires. A shared transformer block iterates over the audio representation, guided by the question, while a learned halting mechanism exits the loop once the representation is ready. AdaLoop adds fewer than 3\% of the base model's parameters and plugs into any audio encoder--language model pair without modifying either component. Evaluated on three architecturally distinct models across MMSU, MMAU-Pro, and MMAR, AdaLoop raises the average accuracy by 2.9 to 3.8 points, with the largest gains on perception-heavy subtasks where the model learns to apply deeper reasoning.

1. INTRODUCTION

Large audio language models lag on fine-grained acoustic perception, while inference depth remains fixed across questions. AdaLoop adapts refinement depth to each audio–question pair and improves average accuracy across three backbones.

1. INTRODUCTION

  • LALMs understand meaning better than acoustic details, limiting applications that depend on fine-grained judgments of how audio sounds.Examples include paralinguistic analysis, nonverbal vocalization understanding, and safety and privacy evaluation of spoken interactions.Source: §1 · p. 1
  • Existing inference-time approaches use fixed iterations or target single tasks, while generalist audio models process every query at fixed depth.AdaLoop instead predicts the refinement steps needed for each audio–question pair and halts early when shallow processing suffices.Source: §1 · p. 1
  • Across MMSU, MMAU-Pro, and MMAR, AdaLoop raises average accuracy by 2.9 to 3.8 points on each evaluated backbone: Qwen2.5-Omni-7B, Kimi-Audio-7B-Instruct, and MiMo-Audio-7B-Instruct.Source: §1 · p. 1

2. ADAPTIVE-DEPTH LATENT REASONING

AdaLoop replaces the audio-to-language connector with a recurrent latent block whose shared parameters refine audio features under question guidance. A learned halting mechanism adapts the number of iterations, while training combines task loss with a cost for extra computation.

2. ADAPTIVE-DEPTH LATENT REASONING

  • AdaLoop replaces the connector with a recurrent block that refines encoder features for K iterations, then passes the result to the language backbone.The learned halting mechanism determines K from the audio and question; the overview specifies question-guided cross-attention, scale-aware self-attention, and a feed-forward update.Source: §2 · p. 1, Figure 1 · p. 2

2.1. Recurrent Latent Block

  • Each iteration shares parameters but adds an iteration embedding, then applies question-guided attention, four temporal attention scales, and a feed-forward update.The pre-norm residual connects to the preceding state to stabilize training across chained iterations.Source: §2.1 · p. 2

2.2. Adaptive Halting

  • The halting unit uses the current state and question to accumulate probabilities, stopping when the cumulative signal reaches 1, up to Kmax=8.The output is a weighted mixture and is differentiable with respect to the halting parameters.Source: §2.2 · p. 2

2.3. Training

  • Training adds a ponder cost to next-token prediction and proceeds from frozen-encoder module training to joint fine-tuning of trainable components.The training set has 50,000 question–answer pairs across audio QA families; the encoder remains frozen throughout.Source: §2.3 · p. 2
  • Table 1 reports accuracy on three audio-understanding benchmarks; MMAU-Pro’s All score is not the average of its category columns.Avg. is the mean of the three All scores.Source: Table 1 · p. 3

3. EXPERIMENTS

Across three audio-language models and three benchmarks, AdaLoop improves average accuracy over the base models. Its learned depth varies with task demands, and ablations identify adaptive halting as its largest single component contribution.

3.1. Setup

  • Experiments compare Base, SFT, Fixed-4, and AdaLoop across three backbones and MMSU, MMAU-Pro, and MMAR, using MMAU-Pro’s closed nonspatial subset.MiMo-Audio reports only Base and AdaLoop because its connector does not admit a drop-in replacement for the intermediate methods.Source: §3.1 · p. 3

3.2. Main Results

  • AdaLoop leads Base on all three benchmarks for all three backbones, raising average accuracy by 3.8, 3.4, and 2.9 points on Qwen2.5-Omni, Kimi-Audio, and MiMo-Audio, respectively.On Qwen2.5-Omni, all twelve subcategories improve, with MMSU perception rising 8.6 points.Source: §3.2 · p. 3
  • On Qwen2.5-Omni, SFT gains 1.2 points, Fixed-4 adds 0.9, and AdaLoop adds a further 1.7 points.The authors identify adaptive depth as the largest step in this progression.Source: §3.2 · p. 3

3.3. Learned Reasoning Depth

  • Signal-level MMAR questions receive 5.7 iterations on average versus 2.6 for semantic questions; Kimi averages 4.2 and MiMo 3.9, reaching 5.4 on signal-level tasks.The figure reports average AdaLoop iterations by MMAR category for Qwen2.5-Omni-7B.Source: §3.3 · pp. 3–4, Figure 2 · p. 3

3.4. Ablations

  • Replacing adaptive halting with fixed depth four costs the most at −1.7 points; removing question conditioning costs 1.6, while scale-aware attention contributes 1.2.The ablations change one component at a time while keeping the training budget fixed; the base model average is 59.3%.Source: §3.4 · p. 4, Table 2 · p. 4

4. CONCLUSION

AdaLoop improves three architecturally distinct LALMs on three benchmarks, with gains concentrated on perception-heavy tasks. Its ceiling is on tasks where one coarse pass already suffices.

4. CONCLUSION

  • AdaLoop improves three architecturally distinct LALMs on three benchmarks, with gains concentrated on perception-heavy tasks.The improvements occur without modifying the models’ encoders or language backbones.Source: §4 · p. 4
  • AdaLoop is a lightweight recurrent module that learns how many latent refinement steps each audio question requires.Source: §4 · p. 4
  • AdaLoop’s ceiling is on tasks where a single coarse pass already suffices; richer training curricula and parameter-efficient adaptation are suggested next steps.Source: §4 · p. 4

Related papers

Loading 2610.06949v1…