PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-R1: Audio Reasoning through Reinforcement Learning on Self-Supervised Tasks

Jihoon Jeong, Mirco Ravanelli, Zhepei Wang, Cem Subakan

arXiv:2610.34609v1cs.SDcs.AIeess.AS

TL;DR

Audio reasoning in large audio-language models remains underexplored and underdeveloped, with scalable post-training challenged by costly supervised data and rewards that may not ensure reliance on audio. Audio-R1 uses automatically answered self-supervised audio tasks to provide verifiable rewards for RL post-training. The paper reports consistent gains across models and benchmarks, including downstream gains beyond training-task accuracy and improved reasoning-trace quality.

  • Problem

    LALM audio reasoning remains underdeveloped, while supervised reasoning data is costly to scale and conventional answer rewards may not ensure reliance on acoustic evidence.

    Source: §1 · p. 1

  • Method

    Audio-R1 uses automatically determined answers from three self-supervised audio tasks to verify responses and provide RL rewards.

    Source: §1 · p. 2, §3.2 · p. 2

  • Results

    Audio-R1 consistently improves performance across four base models and three benchmarks, with further gains from joint training and larger downstream improvements than SFT.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support self-supervised RL post-training as a promising direction for developing more general-purpose audio capabilities.

    Source: §6 · p. 4

    The tasks cover selected forms of audio correspondence, contextual completion, and temporal ordering, while final-answer rewards do not verify intermediate reasoning and the LLM judge cannot establish trace faithfulness.

    Source: G. LIMITATIONS · p. 12

Abstract

from arXiv · show

Reasoning is increasingly important for large audio-language models (LALMs), yet remains relatively underexplored compared with text-only and vision-language models. We introduce Audio-R1, a reinforcement learning (RL) post-training framework that revisits audio self-supervised learning as a source of verifiable rewards for improving audio understanding and reasoning task performance. Audio-R1 uses answers automatically determined during the construction of self-supervised tasks to verify model responses and compute rewards. Experiments across multiple LALMs and diverse audio benchmarks show consistent performance gains, while ablations demonstrate the benefits of each self-supervised task and additional gains from joint training. While both supervised fine-tuning and Audio-R1 improve self-supervised task accuracy, Audio-R1 further improves performance on downstream tasks distinct from the training tasks. These results highlight the potential of RL post-training with self-supervised audio tasks to improve downstream performance and reasoning-trace quality.

1. INTRODUCTION

Audio reasoning in LALMs remains underdeveloped, while scalable post-training is constrained by costly supervised data and rewards that may not ensure reliance on audio. Audio-R1 uses self-supervised tasks with automatically determined answers as verifiable rewards; reported results include gains across models and benchmarks, joint-training benefits, and improved reasoning-trace quality.

1. INTRODUCTION

  • Audio reasoning in LALMs remains underdeveloped relative to text-only and vision-language models, and explicit reasoning does not consistently outperform direct answering.Recent studies also report that explicit reasoning can underperform direct answering.Source: §1 · p. 1
  • High-quality supervised reasoning data is costly to collect at scale, and answer-correctness rewards may not ensure models rely on acoustic evidence.These challenges motivate deriving supervision from audio content and structure.Source: §1 · p. 1
  • Audio-R1 turns automatically determined answers from three self-supervised audio tasks into rewards for verifying LALM responses.Identity Detection, Region Inpainting, and Temporal Ordering target audio information at different granularities.Source: §1 · p. 2
  • SFT and Audio-R1 both improve task accuracy, but Audio-R1 yields larger downstream improvements and higher reasoning-trace quality scores than the base model and SFT.The reasoning-trace comparison uses an audio-conditioned judge.Source: §1 · p. 2
  • Audio-R1 consistently improves performance across four base models and three benchmarks, with gains from individual self-supervised tasks and further improvements from joint training.The reported benchmarks are MMAR, MMAU-mini, and MMAU-Pro.Source: §1 · p. 2

2. RELATED WORK

Prior audio SSL primarily pre-trains general representations, while RL approaches for LALMs may depend on annotations, generated reasoning traces, or task-specific evaluators. Audio-R1 instead uses answers derived from self-supervised task construction to compute post-training rewards.

Self-Supervised Learning

  • Audio-R1 extends SSL from representation pre-training to LALM post-training by using task answers derived from audio to compute rewards.The stated focus shifts toward improving audio reasoning beyond the training tasks.Source: §2 · p. 2

Reinforcement Learning-Based Reasoning for Large Audio-Language Models

  • Compared with prior RL approaches, Audio-R1 derives rewards from self-supervised task construction rather than externally supplied answers or judgments.Prior approaches may rely on human-annotated question-answer pairs, generated reasoning traces, or task-specific evaluators.Source: §2 · p. 2

3. METHODOLOGY

Audio-R1 applies rule-based, outcome-based RL to responses on self-supervised audio tasks whose answers are known by construction. Its three tasks probe complementary audio information, and its reward combines answer correctness with format compliance.

3.1. Preliminary

  • Audio-R1 uses GRPO within RL with verifiable rewards, sampling multiple responses and updating the policy using group-relative rewards.The rule-based verifier compares responses with a ground-truth answer, and normalized group advantages guide policy updates without reasoning-process supervision.Source: §3.1 · p. 2

3.2. Task Design

  • Three automatically constructed tasks provide known answers for scalable verification and probe cross-clip correspondence, local contextual compatibility, and temporal structure.Identity Detection identifies same-recording candidates; Region Inpainting selects the missing segment; Temporal Ordering recovers chronological segment order.Source: §3.2 · pp. 2–3

3.3. Reward Design

  • The reward combines a binary final-answer correctness signal with a binary format-compliance signal.The required response places reasoning inside <think> tags and the final answer inside <answer> tags.Source: §3.3 · p. 3

4. EXPERIMENTAL SETUP

Experiments use a diverse audio pool to train Audio-R1 and evaluate transfer on three audio benchmarks alongside held-out self-supervised tasks.

4. EXPERIMENTAL SETUP

  • The training pool contains approximately 24K recordings from six named datasets, converted to 16 kHz mono.The sources are MELD, LibriSpeech, VoxCeleb2, Multispeaker, YouTube-8M, and CoVoST 2.Source: §4 · p. 3
  • Training applies LoRA to the LLM backbone and audio aligner, freezes the encoder, and optimizes with GRPO plus KL regularization.The setup uses β = 0.04, a format-reward weight of λ = 0.2, eight sampled responses per prompt, and one epoch on 8×H200 GPUs.Source: §4 · p. 3
  • Transfer is evaluated on MMAR, MMAU-mini, and MMAU-Pro, with held-out tests assessing the three self-supervised capabilities.The held-out tests use recordings from MMAU-mini and MMAR.Source: §4 · p. 3
  • Table 1 compares accuracy across closed-source, RL-finetuned, and open-weight models, while Fig. 2 plots before-and-after accuracy across three self-supervised tasks and three downstream benchmarks.For open-weight models, Table 1 places each base model next to its “+ Audio-R1” row; Fig. 2 uses radar plots.Source: Table 1 · p. 3, Figure 2 · p. 3

5. RESULTS

Audio-R1 improves downstream benchmark performance across evaluated backbones, and joint training provides further gains over single-task training. Compared with SFT, it also transfers better downstream and receives higher reasoning-quality scores.

5. RESULTS

  • The results section compares backbones, task combinations, and learning objectives to assess performance, transfer, and reasoning-trace quality.The three research questions cover cross-backbone effectiveness, complementary task benefits, and differences between Audio-R1 and SFT.Source: §5 · p. 3

5.1. Q1: Is Audio-R1 effective and generalizable across different audio-language models?

  • Audio-R1 improves all three downstream benchmarks across four backbones; Audio-R1-Qwen3-Omni scores 70.3% on MMAR, 77.5% on MMAU-mini, and 63.6% on MMAU-Pro.The reported sub-category gains include temporal reasoning, acoustic analysis, music, emotion, and knowledge-based questions.Source: §5.1 · p. 4

5.2. Q2: Do the self-supervised tasks provide complementary benefits for capability learning and downstream transfer?

  • Joint training exceeds single-task training by +16.9, +15.4, and +32.4 pp on the three self-supervised capabilities and gains +4.0 pp on the downstream-benchmark average.Each single-task run performs best on its own task and also improves the other two self-supervised tasks relative to the base model; post-training results are means ± standard deviations over three runs.Source: §5.2 · p. 4, Table 2 · p. 4

5.3. Q3: How do learning approaches differ in generalization and reasoning quality?

  • Audio-R1 transfers better downstream than SFT and scores higher than both SFT and the base model on all three reasoning-quality dimensions in both evaluations.SFT generally scores below the base model on reasoning quality, although both approaches produce comparable gains on self-supervised task accuracy.Source: §5.3 · p. 4, Figure 3 · p. 4

6. CONCLUSION

Audio-R1 derives verifiable rewards from three self-supervised audio tasks without human annotations or external supervision. Experiments report consistent gains across four LALM backbones, and the authors identify further task design and broader real-world generalization as directions for future work.

6. CONCLUSION

  • Audio-R1 uses three self-supervised audio tasks to derive verifiable rewards without human annotations or external supervision.The conclusion presents this as an RL post-training framework.Source: §6 · p. 4
  • Experiments report consistent audio-understanding gains across four LALM backbones, while task ablations highlight benefits from multi-task training with complementary signals.Source: §6 · p. 4
  • Future work can explore richer self-supervised tasks, adaptive task selection and mixing, and generalization to more diverse real-world audio settings.Source: §6 · p. 4

Self-Supervised Tasks

The appendix prompts operationalize three self-supervised audio tasks and describe how a judge evaluates reasoning chains. The tasks use exact-copy detection, contextual segment selection, and chronological ordering.

A.1. Task Prompts

  • Identity Detection asks whether each candidate is an exact copy of the anchor recording or comes from a different source.The prompt requires choosing among options describing which candidates match the anchor.Source: §A.1 · p. 7
  • Region Inpainting selects the candidate segment that best fits a missing part, using surrounding audio context such as speech, speaker identity, acoustic continuity, and temporal flow.Source: §A.1 · p. 7
  • Temporal Ordering reconstructs the original chronology of three shuffled conversation segments using discourse, reference, turn-taking, and prosodic cues.The prompt asks the model to transcribe segments and identify links before selecting an order.Source: §A.1 · p. 7

A.2. Judge Prompt

  • The judge evaluates reasoning-chain completeness, logical coherence, and audio groundedness on a 1–5 scale, separately from final-answer correctness.The prompt defines groundedness as whether audio-related claims are supported by the recording rather than generic priors or guesses.Source: §A.2 · p. 8

B. BENCHMARK GAINS BY SUBTASK

Audio-R1 improves accuracy broadly across Qwen2.5-Omni benchmark sub-categories, with a similar upward pattern across the other three backbones.

B. BENCHMARK GAINS BY SUBTASK

  • Across MMAU-mini, MMAR, and MMAU-Pro, Qwen2.5-Omni’s improvements span many categories rather than concentrating in a few.Table 3 reports percentage-point changes for categories meeting its minimum item-count thresholds, sorted by gain.Source: Table 3 · p. 8, B. BENCHMARK GAINS BY SUBTASK · p. 9
  • Temporal reasoning and acoustic analysis gains align with abilities exercised by Temporal Ordering and Region Inpainting, respectively.Improvements also extend to music, emotion, and knowledge-based questions not directly targeted by these tasks.Source: B. BENCHMARK GAINS BY SUBTASK · p. 9
  • Across all nine backbone–benchmark combinations, more sub-categories improve than decline, indicating a modest but consistent upward tendency.Figure 4 summarizes sub-category outcomes for the other three backbones.Source: Figure 4 · p. 9, B. BENCHMARK GAINS BY SUBTASK · p. 9

C. TASK DIFFICULTY AND REWARD-SIGNAL DENSITY

During multi-task GRPO training, rollout accuracy rises across all three self-supervised tasks, but their learning trajectories and persistence of mixed-reward groups differ.

C. TASK DIFFICULTY AND REWARD-SIGNAL DENSITY

  • Across all three tasks, rollout accuracy trends upward, while the share of prompt groups with mixed rewards changes over training.Mixed-reward groups receive nonzero advantages; groups whose rollouts are all correct or all wrong receive none.Source: Figure 5 · p. 9, C. TASK DIFFICULTY AND REWARD-SIGNAL… · p. 9
  • Identity Detection approaches ceiling quickly, and its mixed-reward groups nearly disappear.The base model starts at 80.7% on Identity Detection, compared with 25% accuracy under uniform guessing.Source: C. TASK DIFFICULTY AND REWARD-SIGNAL… · p. 9
  • Region Inpainting improves more gradually, with mixed-reward groups declining but persisting during training.Source: C. TASK DIFFICULTY AND REWARD-SIGNAL… · p. 9
  • Temporal Ordering improves from near-chance performance but fluctuates considerably, with substantial mixed-reward signal remaining late in training.The base model scores 27.3% on Temporal Ordering, close to 25% uniform-guess accuracy.Source: C. TASK DIFFICULTY AND REWARD-SIGNAL… · p. 9
  • The trajectories expose a fixed-mixture limitation: nearly solved tasks may continue using rollout budget despite offering little accuracy-based learning signal.The authors suggest an adaptive curriculum could shift later updates toward tasks with persistent mixed outcomes.Source: C. TASK DIFFICULTY AND REWARD-SIGNAL… · p. 9

D. SINGLE-TASK AND JOINT-TRAINING UPDATE RELATIONSHIPS

Comparing parameter updates shows that joint training relates to each single-task update differently than the single-task updates relate to one another.

D. SINGLE-TASK AND JOINT-TRAINING UPDATE RELATIONSHIPS

  • All updates are measured relative to the same shared base-model initialization.The paper defines each single-task and joint update as the trained parameters minus that initialization.Source: D. SINGLE-TASK AND JOINT-TRAINING… · p. 10
  • Single-task updates are weakly aligned with one another, while the joint update is more closely aligned with each single-task update.Figure 6 shows off-diagonal cosine similarities and diagonal update norms.Source: Figure 6 · p. 10, D. SINGLE-TASK AND JOINT-TRAINING… · p. 10
  • The joint update norm exceeds every single-task update norm but remains below the norm of their summed updates.This indicates that joint training combines task directions with a changed balance rather than simply adding independently learned updates.Source: D. SINGLE-TASK AND JOINT-TRAINING… · p. 10

E. SFT AND RL: ACCURACY AND REASONING QUALITY

With the same self-supervised instances and answers, Audio-R1 combines stronger downstream transfer with better reasoning-quality scores than SFT. The examples show both cases where Audio-R1 rejects a misleading audio match and a failure where its reasoning and final answer disagree; the authors also delimit what their experiments establish.

E. SFT AND RL: ACCURACY AND REASONING QUALITY

  • Audio-R1 gains +4.0 vs. SFT’s +1.5 benchmark-transfer points, with higher logical-coherence scores on all three tasks and stronger scores on other judge dimensions.SFT produces shorter traces and lower completeness scores on all three tasks, as well as lower audio-groundedness on Inpainting and Ordering; both methods use the same data, differing only in learning objective.Source: E. SFT AND RL: ACCURACY AND REASONING… · p. 10

F. REASONING EXAMPLES

  • In Identity Detection, Audio-R1 rejects both candidates, unlike the base model, which treats a related gunshot-like sound as evidence of the same recording.In Region Inpainting, the base model and SFT conclude that no candidate fits, while Audio-R1 selects the candidate containing “Versailles” from the spoken context.Source: F. REASONING EXAMPLES · p. 11
  • In Temporal Ordering, Audio-R1’s trace states the correct order (2, 1, 3) but outputs (D), corresponding to 2, 3, 1—a mismatch outcome-only rewards do not directly penalize.The passage identifies this as a failure case.Source: F. REASONING EXAMPLES · p. 11

G. LIMITATIONS

  • The experiments use candidate-based tasks covering selected forms of audio correspondence, contextual completion, and temporal ordering, and do not establish robustness to broader real-world distributions.The authors also note that their selected tasks rely on base models having usable capabilities, so the experiments do not establish that RL develops reasoning skills absent from pretraining.Source: G. LIMITATIONS · p. 12

Related papers

Loading 2610.34609v1…