PaperlayerView exact revision on arXiv

Source-linked AI summary

AudioRSI: Recursively Self-Improving Audio Reasoners

Jiajun Fan, Roger Ren, Jingyuan Li, Rahul Pandey, Prashanth Gurunath Shivakumar, Ankur Gandhe, Yile Gu, Ge Liu, Ivan Bulyko

arXiv:2610.36578v1cs.SD

TL;DR

Audio RL relies on rewards supplied from outside the model, leaving the supervisor fixed as the policy improves. AudioRSI makes one model both actor and critic, then refreshes the critic from the improving policy; across four out-of-distribution benchmarks, it reports strong results and develops structured reasoning behaviors.

  • Problem

    Audio RL uses outcome checks, hand-crafted process rewards, or fixed external judges, none of which lets the model judge its own reasoning as the policy improves.

    Source: §1 · p. 1, §3.2 · p. 4

  • Method

    AudioRSI uses one model as an actor generating reasoning paths and a critic comparing them for process-level reward, periodically handing the critic role to the improving policy.

    Source: §1 · pp. 1–2, §3.3 · p. 4, §3.4 · p. 5

  • Results

    Across four OOD benchmarks disjoint from AVQA training, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, while leading the 7B open-source tier on three benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings provide initial evidence that a model can supply its own process-level reward and that refreshing its critic from the improving policy sustains OOD accuracy ahead of a frozen judge.

    Source: §1 · p. 2, §5 · p. 10

    The evidence spans one backbone and training corpus, while outcome-only rewards cannot distinguish careful acoustic analysis from lucky guesses.

    Source: §3.2 · p. 4, §5 · p. 10

Abstract

from arXiv · show

Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.

1 Introduction

AudioRSI asks whether an Audio LLM can supply process-level supervision and improve that supervision as its policy improves. It makes one model both actor and critic, then hands the critic role to successive policies.

1 Introduction

  • AudioRSI makes one model generate reasoning paths and compare them pairwise as a critic, forming a process-level reward without hand-crafted rules or an external judge.Every T steps, the current policy becomes the critic for its successor.Source: §1 · pp. 1–2
  • Because the critic hears the actor’s recording, it can reject reasoning that invokes acoustic evidence the audio does not contain.The design rests on the premise that a better reasoner should also judge reasoning better.Source: §1 · p. 2
  • Across four OOD benchmarks disjoint from AVQA training, AudioRSI beats strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, and leads AVQA-trained systems there and the 7B open-source tier on the other three.The latter comparisons are among methods without external reward models; several margins are small.Source: §1 · p. 2
  • A refreshed critic keeps OOD accuracy above a frozen external judge in the second half of training, while human-label agreement in a small double-blind audit supports the model’s judgments.The paper also reports that structured reasoning emerges without a hand-written reward specifying how to reason.Source: §1 · p. 2

2 Related Work

Prior audio-reasoning work applies reinforcement learning with outcome, hand-crafted, or external-judge rewards, while related self-rewarding methods mostly concern non-audio settings. AudioRSI distinguishes itself by recursively updating the training signal within one GRPO run and conditioning its critic on the waveform.

2 Related Work

  • Audio LLM research has shifted from speech, sound, and music understanding toward reasoning over complex acoustic inputs and reading out models’ internal representations.Source: §2 · p. 2
  • Audio RL methods differ in reward source—outcome verification, hand-crafted process terms, or an external LLM judge—but those supervisors remain outside the policy.Related work also describes audio methods that regenerate rubrics or construct self-play rewards outside the model being trained.Source: §2 · p. 3
  • Related work also extends RL beyond text generation to diffusion and protein design, and studies controlling policy data distributions and behavior during learning.Other cited directions make reasoning and distillation cheaper or refine image generation through multi-round visual reasoning at inference time.Source: §2 · p. 3
  • Self-rewarding language and vision methods use text-based judgments, whereas AudioRSI recurses online within one GRPO run and conditions its critic on the waveform.The waveform lets its critic reject paths that cite acoustic events absent from the recording.Source: §2 · p. 3

3 Methodology

AudioRSI addresses the limits of outcome-only rewards by using the model to judge reasoning paths and periodically refreshing that critic from the policy. The resulting GRPO pipeline combines process-level self-critique with outcome verification.

3.1 Problem Formulation

  • The task trains an audio reasoning policy to generate both a reasoning process and a final answer, maximizing expected reward for its outputs.The structured output separates reasoning from answering, while the dominant existing design rewards only the final answer.Source: §3.1 · pp. 3–4

3.2 The Limitation of Outcome-Only Rewards

  • Outcome-only rewards treat careful acoustic analysis and lucky guesses alike, motivating process-level supervision alongside answer correctness.The binary signal gives no information about how an answer was reached, leaving the model to find effective reasoning strategies by trial and error.Source: §3.2 · p. 4

3.3 Self-Critique: When Audio Reasoners Judge Themselves

  • The policy’s copied weights act as a critic that compares reasoning paths and assigns reward according to how often each candidate wins against its group.The critic evaluates reasoning, not merely final answers, with the audio recording in context so it can check claims against what was heard.Source: §3.3 · p. 4

3.4 Periodic Critic Synchronization: The Recursive Step

  • Periodic critic synchronization keeps the critic fixed within each interval, then replaces it with the current actor only if its training-mini-batch accuracy has not regressed.Shorter intervals align critic and actor more closely but approach the unstable continuous regime; longer intervals are more stable but let the critic lag.Source: §3.4 · p. 5

3.5 AudioRSI: Recursive Self-Improvement within GRPO

  • AudioRSI integrates self-critique and outcome verification into GRPO, sampling K candidate outputs and using their combined reward to distinguish reasoning quality.The pipeline uses a group-mean baseline and omits group-standard-deviation normalization; the authors report no instability from this simplification. Figure 2 reports the largest accuracy margin over Ke-Omni-R on Sound and Hard items.Source: §3.5 · pp. 5–6, Figure 2 · p. 6

4 Experiments

The experiments evaluate AudioRSI on four out-of-distribution benchmarks and examine emergent reasoning, critic judgments, and design ablations. Results favor self-critique and refreshed critics, with small margins treated cautiously because each configuration uses one seed.

4.2 Main Results

  • AudioRSI scores highest among systems in Table 1 on MMAU test-mini and edges Omni-R1 (AVQA); the matched Ke-Omni-R comparison differs only in reward signal.The setup fine-tunes Qwen2.5-Omni-7B with GRPO on AVQA and evaluates four OOD benchmarks; other Table 1 scores are reported by their authors, not controlled comparisons.Source: §4.2 · pp. 6–7, §4.1 · p. 6
  • On MMAR, AudioRSI leads the 7B open-source tier without external reward models and leads every system on Music, but that Music lead does not transfer to MMAU or MMAU-Pro.It comes close to proprietary Gemini 2.0 Flash overall on MMAR.Source: Table 2 · p. 7, §4.2 · p. 7
  • AudioRSI ranks first overall on MMSU by a narrow margin, trading a little Perception for a larger Reasoning-split gain over matched outcome-only RL.Because configurations use a single seed, the paper treats fractional-point edges such as MMSU as comparable rather than decisively better.Source: Table 3 · p. 7, §4.2 · p. 7
  • On MMAU-Pro, AudioRSI is the strongest open-source model on average and leads every tested system on the hardest Multi-Audio split.Outcome-only Ke-Omni-R is already second on Multi-Audio, and self-critique adds to its result.Source: Table 4 · p. 7, §4.2 · p. 7

4.3 Emergent Reasoning Behaviors

  • AudioRSI traces use sequential markers tied to acoustic events, while Ke-Omni-R collapses the same event sequence into vague reasoning and selects a non-matching option.Keyword analysis of 1,000 MMAU paths finds AudioRSI leads on all five measured behaviors, though counts reflect wording and grow with trace length.Source: §4.3 · pp. 7–8

4.4 Can the Model Judge Its Own Reasoning?

  • Three annotators agreed with the critic’s audio-grounded judgment in the audited cases, while an external judge preferred AudioRSI reasoning in over four of five comparisons against Qwen2.5-Omni-7B and about three of four against Ke-Omni-R.The audit is small and illustrative; the external judge may favor explicit wording, so the authors treat it as corroboration, not proof.Source: §4.4 · pp. 8–9, Table 6 · p. 8

4.5 Ablation Study

  • Removing self-critique costs more than either critic-strategy variant, while refreshed critics stay ahead of a frozen judge in the second half of training and the frozen run later stays flat.The period sweep is an inverted U: T=200 is best overall, T=100 peaks on Hard, and the longest and shortest periods give up most or all gain over outcome-only training; the sweep uses MMAU test-mini.Source: §4.5 · pp. 9–10, Figure 3 · p. 9

5 Conclusion

The conclusion argues that an Audio LLM can provide its own process-level reward and that handing improved policies the critic role offers initial evidence for recursive improvement. The reported results also motivate testing the approach beyond one backbone and training corpus.

5 Conclusion

  • AudioRSI answers the question of whether an Audio LLM can supply its own reward, and gives initial evidence that this reward can improve as the model does.The model judges reasoning against the audio it has heard, without hand-crafted rules or an external judge.Source: §5 · p. 10
  • Among methods without external reward models, AudioRSI leads the 7B open-source tier on three of four OOD benchmarks and every AVQA-trained system on the fourth, while developing structured reasoning without a hand-written reward for it.The conclusion reports that several benchmark margins are small.Source: §5 · p. 10
  • The evidence uses one backbone and one training corpus, making tests across backbones and data the natural next step.Source: §5 · p. 10

A Discussion

AudioRSI addresses the question of whether an audio reasoning model can supply its own process-level reward by using one model as both actor and critic, then periodically making the improved policy the next critic. The discussion reports evidence for useful self-critique and recursive gains, describes emergent reasoning behaviors, and notes training cost and future directions.

A Discussion

  • AudioRSI uses one model as actor and critic, with each improved policy becoming the critic that supervises its successor.The paper frames recursive self-critique as a reward that improves with the policy, rather than relying on a separately trained reward or critic model.Source: A Discussion · p. 16
  • In a small double-blind audit, the model’s judgments of reasoning quality match human judgment.The critic rewards reasoning aligned with audio events and penalizes hallucinated guesses; GPT-4o Audio also prefers AudioRSI reasoning in head-to-head comparisons, with caveats noted in App. D.7.Source: A Discussion · p. 16
  • A recursively refreshed critic stays ahead of a static external judge through the second half of training, while the static-judge baseline narrows the gap late in training.The synchronization period balances stability and adaptivity: T=200 is best overall, while the Hard split peaks at T=100; online self-comparison outperforms offline references overall, with Hard and Music exceptions.Source: A Discussion · p. 17
  • Self-critique training yields sequential temporal analysis, perception-grounded inference, and explicit negative verification without hand-crafted rewards specifying these behaviors.The paper attributes these patterns to the critic treating them as hallmarks of sound reasoning; its evaluation criteria align with the observed behaviors.Source: A Discussion · p. 17
  • The findings suggest recursive self-critique plus outcome checks is more effective than outcome checks alone or with a frozen judge in this setting.Generalization to code generation and mathematical reasoning is proposed as future work, not established by these results.Source: A Discussion · pp. 16–17

A.1 Limitations

  • At K=8, self-critique uses 56 critic passes per question versus 8 actor rollouts, and the practical overhead ratio is unquantified without a controlled wall-clock comparison.The paper identifies added training-time inference cost as a limitation.Source: §A.1 · p. 18

A.2 Future Work

  • Future work proposes testing whether self-critique generalizes to visual reasoning, mathematics, and code generation, where perceptual grounding or verifiable outcomes may complement critique.The paper also proposes studying combinations with human feedback, hand-crafted or verifiable rewards, and specialized reward models.Source: §A.2 · p. 19

B.1 Baselines and Compared Methods

The evaluation compares AudioRSI with base models, RL baselines, and leading proprietary and open-source Audio LLMs. It uses four benchmarks disjoint from AVQA training data and specifies the training setup, computational cost, and recursive GRPO procedure.

B.1 Baselines and Compared Methods

  • AudioRSI is built on Qwen2.5-Omni-7B, and RL methods share that initialization; the main RL baseline, Ke-Omni-R, uses GRPO with outcome-only rewards.A frozen Ke-Omni-R critic variant tests periodic synchronization, while broader comparisons include Omni-R1 and leading proprietary and open-source audio models.Source: §B.1 · p. 20

B.2 Details of Evaluation Benchmarks

  • AudioRSI improves on Ke-Omni-R at every MMAU difficulty level, with the largest gain on Hard samples and the largest domain margins on Sound and Speech.The test-mini split contains 1,000 questions; evaluation also includes MMSU, MMAU-Pro, and MMAR, all disjoint from AVQA training data.Source: §B.2 · p. 21, §B.3 · p. 22

B.3 Training Setups and Hyperparameters

  • Training uses AVQA’s audio track only, with actor outputs formatted as reasoning and answer segments and a critic prompted to compare two reasoning paths.Critic outcomes map to rewards of 1.0 for a candidate win, 0.5 for a tie, and 0.0 for a reference win.Source: §B.3 · p. 22
  • The synchronization period is the main new hyperparameter: AudioRSI sweeps T∈{50, 100, 200, 500, 1000} and defaults to T=200.Most GRPO settings match Ke-Omni-R, including group size K=8, learning rate η=1e-5, and KL coefficient β=0.01; training runs for at most 6,400 steps.Source: §B.3 · p. 22, Table 9 · p. 22

B.4 Computational Resources

  • At K=8, each question requires 56 critic passes alongside 8 actor rollouts; training 6,400 steps takes 3–4 days on eight NVIDIA H200 GPUs.MMAU test-mini evaluation takes 1–2 hours per run on one H200 GPU.Source: §B.4 · p. 23

C Algorithm Pseudocode

  • The algorithm generates K candidate paths, scores them with outcome rewards and pairwise self-critique, combines rewards for GRPO updates, and periodically synchronizes the critic.At synchronization, the actor replaces the incumbent critic only if its outcome accuracy on the current training minibatch is at least as high.Source: C Algorithm Pseudocode · pp. 24–25

D.1 Results on MMSU

AudioRSI ranks first overall on MMSU, with its clearest gain over outcome-only RL on Reasoning rather than Perception. On MMAU-Pro, it leads the challenging Multi-Audio split but ranks second on Avg.

D.1 Results on MMSU

  • MMSU, MMAU-Pro, and MMAR cover broader reasoning skills, input modalities, and domain combinations than MMAU test-mini alone.The section presents these benchmarks as broader evidence of generalization beyond the training distribution.Source: §D.1 · p. 26
  • 62.52% overall accuracy ranks AudioRSI first on MMSU, ahead of Audio Flamingo 3 (62.30%) and Ke-Omni-R (62.08%).On Reasoning it scores 80.54%, above Qwen2.5-Omni-7B (79.83%) and Ke-Omni-R (78.06%); on Perception it scores 45.62%, below Ke-Omni-R (47.09%) and Gemini 1.5 Pro (46.10%). The overall margins are narrow.Source: §D.1 · p. 26, Table 10 · p. 26

D.2 Results on MMAU-Pro

  • 37.9% on MMAU-Pro Multi-Audio puts AudioRSI ahead of Ke-Omni-R (35.6%) and Gemini 2.5 Flash (21.2%) on the benchmark’s hardest split.AudioRSI’s Avg is 55.9%, second to Gemini 2.5 Flash (59.2%); comparisons with base-model and proprietary rows cross protocols.Source: §D.2 · p. 27

D.3 Results on MMAR

On MMAR, AudioRSI leads the 7B open-source tier among methods without external reward models, while comparisons across domains are uneven. On MMAU test-mini, it exceeds same-protocol outcome-only RL baselines under matched backbone and training data.

D.3 Results on MMAR

  • 64.0% overall on MMAR is the best 7B open-source result among methods without external reward models, and is competitive with Gemini 2.0 Flash (65.6%) and GPT-4o Audio (63.5%).AudioRSI scores 56.8% on Music, ahead of the nearest 7B baselines at 51.5% and Gemini 2.0 Flash at 51.0%; on M-Sp it scores 68.3%, but trails Gemini 2.0 Flash and Omni-R1 (AVQA) on S-M. The paper draws no conclusion on two splits with very few questions.Source: §D.3 · p. 28, Table 12 · p. 28

D.4 Results on MMAU

  • 76.1% on MMAU test-mini exceeds same-protocol Ke-Omni-R (74.6%) and reported Omni-R1 (AVQA) (75.8%) under the same Qwen2.5-Omni-7B backbone and AVQA data.Omni-R1 (VGGS-GPT) reports 77.0% but uses a different training distribution, while Audio-Thinker adds an external Qwen3-8B reward model, confounding reward-signal comparisons.Source: §D.4 · p. 29, Table 13 · p. 29

D.5 Training Dynamics and Out-of-Distribution Generalization

AudioRSI’s out-of-distribution accuracy rises during training and stays above a static external-judge baseline from step 2k onward. Reasoning length rises and later recedes from its peak while accuracy continues to improve, with no sustained accuracy drop accompanying length growth.

D.5 Training Dynamics and Out-of-Distribution Generalization

  • From step 2k onward, AudioRSI exceeds the static external-judge baseline at every checkpoint, and its MMAU test-mini accuracy rises again as the baseline flattens.MMAU test-mini is out of distribution relative to the training questions, so the curves track generalization rather than fit to training data.Source: §D.5 · p. 30, Figure 4 · p. 30
  • The authors propose that a frozen critic may become miscalibrated as the actor improves, but they do not test this explanation directly.The static external-judge baseline ends lower.Source: §D.5 · p. 30
  • Reasoning length peaks in the second half of training and stays below that peak in the final stretch, while out-of-distribution accuracy keeps improving.The authors report no length explosion or sustained accuracy drop accompanying length growth.Source: §D.5 · p. 30, Figure 5 · p. 31

D.6 Quantitative Analysis of Emergent Reasoning Behaviors

Across quantitative analyses and case studies, AudioRSI’s reasoning is more structured and is preferred by an external judge, while its critic identifies audio-grounding and reasoning flaws. The case studies span multiple audio domains, but their human-agreement check covers only four cases.

D.6 Quantitative Analysis of Emergent Reasoning Behaviors

  • Premise & Deduction appears in 94.4% of AudioRSI paths, versus 62.7% for Ke-Omni-R and 53.3% for Qwen2.5-Omni-7B.The authors describe this keyword measure as premise-to-conclusion wording, not a direct measure of reasoning correctness.Source: §D.6 · p. 32, Table 14 · p. 32
  • Explicit Justification appears in 22.2% of AudioRSI paths, versus 3.5% for Ke-Omni-R and 3.2% for Qwen2.5-Omni-7B; Evidential Support and Sequential Organization also show gains.The keyword analysis samples 1,000 paths per model; the authors caution that incidence measures wording, not reasoning correctness.Source: §D.6 · p. 32, Table 14 · p. 32

D.7 LLM-as-Judge Evaluation of Reasoning Quality

  • GPT-4o Audio preferred AudioRSI’s reasoning in 81.2% of comparisons against Qwen2.5-Omni-7B and 74.7% against Ke-Omni-R.The evaluation used 1,000 MMAU test-mini samples with randomized presentation order; the authors treat it as corroboration, not proof, because the judge scores below the compared models and may favor structured wording.Source: §D.7 · p. 33, Table 15 · p. 33

D.8 Case Studies: Critic Quality and Human Agreement

  • In four double-blind cases across Sound, Speech, and Music, the critic matched the unanimous labels of three annotators and identified audio-grounding, reasoning, and terminology flaws.The authors frame this as a qualitative validity check, not a powered agreement study or an estimate of agreement at scale.Source: §D.8 · p. 34, §D.8.1 · p. 34, §D.8.2 · p. 34

D.8.5 The Foundation of Self-Critique Training

  • The authors connect the critic’s audio-grounded, process-level criteria to structured reasoning emerging in training, while noting that four cases cannot establish this connection on their own.The critic’s examples include sequence checking, terminological accuracy, reasoning flaws despite correct answers, and rejection of fabricated audio content.Source: §D.8.5 · p. 36

D.9 Qualitative Analysis: AudioRSI vs. Ke-Omni-R

  • Qualitative comparisons show Ke-Omni-R selecting answers that contradict its own reasoning, relying on contextual assumptions, and asserting nonexistent audio content.AudioRSI’s examples instead use sequential markers, acoustic observations, and explicit checks for absent content.Source: §D.9.1 · p. 37, §D.9.2 · p. 38, §D.9.3 · p. 38

D.10 Qualitative Analysis: AudioRSI vs. Qwen2.5-Omni-7B

  • Against Qwen2.5-Omni-7B, AudioRSI tracks multiple emotions per speaker and distinguishes paper-like acoustic features from a metal attribution.The examples contrast AudioRSI’s sadness-and-joy attribution and rustling/tearing observations with Qwen’s simplified emotions and soda-can guess.Source: §D.10.1 · p. 40, §D.10.2 · p. 41
  • AudioRSI integrates clanking dishes with chatter to identify a diner, while Qwen maps generic impact sounds and crowd noise to a supermarket.In another example, AudioRSI verifies that a dog is absent where Qwen hallucinates one.Source: §D.10.3 · p. 41, §D.10.4 · pp. 41–42

D.11 Additional Ablation Studies

The ablations indicate that self-critique, critic refreshing, online references, and a moderate synchronization interval each contribute to AudioRSI’s performance. The default setting leads overall in the tested sweep, although fixed references score better on the Hard split.

D.11 Additional Ablation Studies

  • Both very frequent updates (T=50) and infrequent updates (T=1000) lose ground; the authors attribute this to noisy evaluation at one extreme and growing actor–critic divergence at the other.The default is best among the tested settings, with performance rising through T=200, plateauing at T=500, and declining at T=1000.Source: §D.11 · p. 43
  • The recursive self-critic wins overall against the frozen Ke-Omni-R judge, with the largest domain gains on Sound and Music.Even the frozen-judge run ends slightly above Ke-Omni-R overall, though below it on Music and marginally on Medium; the recursive refresh adds further gain.Source: §D.11 · p. 43
  • Online self-comparison outperforms fixed offline Ke-Omni-R references overall; the authors suggest it keeps comparisons capability-matched and exposes the critic to current-policy diversity.Fixed references score slightly better on Hard, where the authors suggest they may provide a stable high-quality anchor.Source: §D.11 · pp. 43–44
  • T=200 gives the best overall performance in the tested synchronization-period sweep, while T=100 gives the highest Hard-split score at 75.85%.The Hard score at T=200 is 74.15%; overall accuracy favors T=200, not T=100.Source: §D.11 · pp. 43–44
  • The frozen-critic comparison uses an audio-capable Ke-Omni-R, keeping audio access fixed, but changes both critic refreshing and model identity.The authors describe this as a conservative baseline and identify the period sweep as the within-model test of refreshing.Source: §D.11 · p. 44
  • 76.10% overall accuracy for AudioRSI exceeds 74.60% for outcome-only rewards, with larger margins on Sound and Speech.Removing self-critique causes the largest drop among the three design ablations; the Music result does not drop.Source: §D.11 · pp. 44–45
Loading 2610.36578v1…