PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

Fangxu Yu, Tao Feng, Dehai Min, Zinan Lin, Weijia Xu, Michael Xu, Philip S. Yu, Ge Liu, Tianyi Zhou

arXiv:2608.02831v1cs.SDcs.CL

TL;DR

Existing audio-reasoning rewards are either outcome-focused or coarse, fixed, and insufficiently grounded in acoustic evidence. AudioRubrics evolves audio-grounded rubrics from waveforms and rollouts, substantially outperforming baselines across three benchmarks while improving perception.

  • Problem

    Existing process-based rewards use coarse, fixed criteria that assess reasoning surface quality without verifying grounding in acoustic evidence.

  • Method

    AudioRubrics synthesizes per-sample audio-grounded rubrics and evolves them from model rollouts to maintain a learning signal targeting current weaknesses.

  • Results

    AudioRubrics achieves the best overall accuracy on all three evaluated audio reasoning benchmarks and consistently outperforms comparable baselines.

  • Takeaways & Limitations

    On MMSU perception, AudioRubrics surpasses the best comparable baseline by 4.3 points, an 8.9% relative improvement, across all three perception dimensions.

  • Takeaways & Limitations

    AudioRubics hinges on a sufficiently capable rubric generator and judge, as weaker models produce noisier rewards and lower performance.

Abstract

from arXiv · show

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with self-evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence. Our project page is available at https://audiorubrics.github.io.

1 INTRODUCTION

AudioRubrics addresses the limitations of outcome-only and coarse process-based rewards by grounding evolving rubric supervision in raw audio and model rollouts. Evaluations across three audio reasoning benchmarks show consistent improvements over open-source models and reinforcement-learning baselines.

  • Motivation: Outcome-based RLVR rewards only final answers, while process-based methods use coarse, fixed criteria that may reward fluent reasoning without verifying audio grounding.Together, these designs leave either the reasoning trace or its acoustic grounding insufficiently supervised.
  • Method: AudioRubrics constructs initial per-question rubrics directly from raw waveforms, grounding each criterion in acoustic evidence.The framework combines rubric-based rewards with verifiable RLVR rewards.
  • Evaluation: AudioRubrics evaluates audio reasoning on MMAU, MMAR, and MMSU, consistently outperforming a wide range of open-source models and RL baselines.The three benchmarks are MMAU, MMAR, and MMSU.
  • Analysis: The rubrics become progressively more advanced as the policy improves, indicating that the evaluation standard co-evolves with the model during training.Ablation studies verify the contribution of the framework’s components.

2 PRELIMINARY

GRPO optimizes responses using weighted rewards over sampled candidates while regularizing policy updates with KL divergence. Sample-wise rubrics provide explicit, weighted criteria judged by a language model and optimized as reinforcement-learning rewards.

  • GRPO: GRPO samples G candidate responses and optimizes their weighted reward scores.The reward function assigns each candidate a score, which contributes to the objective used to encourage higher-reward responses.
  • GRPO: KL-divergence regularization penalizes deviations between the updated policy πθ and reference policy πref.The coefficient βKL scales this stability term, while the clipping operation is omitted for simplicity.
  • Rubrics: Sample-wise rubrics specify per-example natural-language criteria, each paired with a weight whose sum equals 1.For a query x, rubric items rx,k and weights wx,k define the evaluation structure.
  • Rubrics: A judge language model outputs binary satisfaction decisions for each rubric item, whose weighted results form the rubric-based score.The judge conditions on the query and response, and reinforcement learning optimizes expected rubric scores over training questions.
  • Rubrics: Concrete rubric items reduce susceptibility to judge bias and support consistent, comparable evaluation across language-model judge runs.The criteria are intended to make evaluation more objective than relying on less-defined judgments.

3 AUDIORUBRICS: RL WITH EVOLVING RUBRICS AS REWARDS

AudioRubrics augments outcome-based RLVR with audio-grounded, evolving rubric rewards that supervise reasoning quality while adapting to the current policy. Its combined objective preserves answer correctness, filters and reweights informative criteria, and controls reasoning length.

  • 3.1 TASK FORMULATION: The task gives a large audio language model an audio input, textual query, and ground-truth answer, requiring reasoning over audio before predicting the answer.
  • 3.2 REWARD DESIGN: Accuracy and format rewards enforce correct answers and the required <think> . . . </think><answer> . . . </answer> reasoning-then-answer structure.Together, these deterministic terms define accuracy-only RLVR.
  • 3.2 REWARD DESIGN: AudioRubrics initializes weighted, per-question rubrics from the raw waveform using an audio-capable model across predefined evaluation dimensions, including auditory-evidence grounding.One positively phrased criterion is generated per dimension, with normalized importance weights.
  • 3.2 REWARD DESIGN: At each iteration, the rubric generator proposes non-redundant criteria from current rollouts, judges every rollout, and carries surviving criteria forward so standards become progressively harder.Rollout-induced rubrics can include negative criteria that capture recurring failure modes.
  • 3.2 REWARD DESIGN: Zero-variance rubrics are removed because they distinguish neither mastered nor unreachable criteria, while survivors are ranked by variance and truncated to the most discriminative set.This filtering preserves criteria that differentiate rollouts within each group.
  • 3.2 REWARD DESIGN: Rubric weights are reassigned per question and updated independently for each prompt group at every training step, keeping the reward informative for the current policy.
  • 3.2 REWARD DESIGN: The overthinking penalty counterbalances rubric-driven verbosity, while the combined objective anchors answer correctness, densely supervises reasoning, and bounds trace length.Accuracy-only RLVR is recovered when both rubric and penalty coefficients are zero, and training maximizes the GRPO objective.

4 EXPERIMENTS

AudioRubrics is evaluated across three audio reasoning benchmarks and outperforms similarly sized models and methods, including on audio perception. Ablations show that evolving rubrics, reward weighting, overthinking control, and rubric-model capability materially affect performance.

  • Benchmarks: Evaluation spans MMAU Test-mini, MMAR, and MMSU, covering speech, sound, music, real-world video reasoning, and fine-grained spoken-language cues.The benchmarks contain 1,000, 1,000, and 5,000 examples, respectively.
  • Main results: AUDIORUBRICS achieves the best overall accuracy on all three benchmarks and leads similarly sized models on key Sound and Speech categories.It ranks first on Sound for both MMAU and MMAR and leads all similar-size models on Speech for both benchmarks.
  • Main results: 4.3 points, or an 8.9% relative improvement, separate AUDIORUBRICS from the best comparable-size baseline on MMSU perception.Gains are consistent across all three perception dimensions.
  • Reward ablations: Rubric rewards improve over GRPO across the tested range, with performance peaking at γ = 0.5 before excessive rubric weighting weakens accuracy optimization.Beyond the peak, the accuracy reward is increasingly downweighted.
  • Reward ablations: Performance peaks at δ = 0.15: smaller penalties permit overlong reasoning traces, whereas larger penalties excessively compress them and harm reasoning.The penalty controls overthinking while preserving the accuracy signal.
  • Ablations and adaptation: Evolving rubrics add improvement beyond RL training and initialized rubric rewards, while newly adopted rubrics rise from roughly 35% early to about 60% by training’s end.The evolving criteria increasingly target weaknesses that emerge as the policy improves.

5 RELATED WORK

Prior work has advanced audio understanding from adapted audio representations to large audio-language models, while rubric-based rewards extend verifiable reinforcement learning toward open-ended, multi-criteria evaluation.

  • Audio Understanding and Reasoning: Audio understanding progressed from adapting CLIP to audio toward large audio-language models that integrate audio into language-model backbones and use large-scale pretraining.The passage also identifies supervised reasoning traces and tool use as later ways to sharpen reasoning, but the excerpt truncates before completing that discussion.
  • Rubric as Rewards: Rubric-based rewards address the limits of scalar verifiable rewards by scoring open-ended responses against fine-grained, instance-specific criteria.Verifiable rewards work well when correctness is programmatically checkable, whereas open-ended tasks require multi-criteria judgment.

6 CONCLUSION

AUDIORUBRICS introduces rubric-based rewards for reinforcement learning in audio reasoning and evolves those rubrics from model rollouts to sustain learning. Experiments show substantial improvements over diverse baselines on audio understanding and reasoning benchmarks.

  • AUDIORUBRICS is a reinforcement learning framework that supervises audio reasoning with rubric-based rewards.
  • The framework evolves rubrics from model rollouts and elicits harder criteria as model capability improves, sustaining the learning signal.
  • Extensive experiments show substantial improvements over a wide range of baselines on audio understanding and reasoning benchmarks.

A IMPLEMENTATION DETAILS

AudioRubrics uses Qwen2.5-Omni-7B and trains it with GRPO under specified rollout, optimization, sequence-length, precision, and reward-weight settings.

  • Model and training: The base model is Qwen2.5-Omni-7B.The paper identifies Qwen2.5-Omni-7B as the model used for training.
  • Model and training: GRPO training runs for 400 steps with batch size 8, learning rate 1e-6, constant scheduling, and 8 rollouts per prompt.Sampling uses temperature 1.0, and the KL coefficient is βKL = 0.001.
  • Model and training: Training uses maximum prompt and response lengths of 4096 and 1024 tokens, respectively, under bf16 precision.These sequence-length limits and precision are specified for RLVR training.

B QUALITATIVE ANALYSIS · B.1 CASE STUDY · B.2 COMPARISON BETWEEN INITIAL AND EVOLVED RUBRICS

The qualitative analysis shows that AUDIORUBRICS improves audio reasoning by grounding interpretations in dialogue state and speaker intent rather than surface lexical cues. Comparing rubric sources, evolved rubrics target rollout-specific, question-dependent distinctions that fixed initial rubrics overlook.

  • B QUALITATIVE ANALYSIS: AUDIORUBRICS reconstructs the interaction step by step and tracks the evolving dialogue state to infer communicative intent.The GRPO baseline instead anchors on the most lexically salient audio token and gives a one-line surface justification.
  • B.2 COMPARISON BETWEEN INITIAL AND EVOLVED RUBRICS: Initial rubrics cover easy location subtasks—grounding the sound and ruling out water options—that both right and wrong rollouts already pass.Because these criteria are satisfied by both rollout types, they provide limited discrimination.
  • B.2 COMPARISON BETWEEN INITIAL AND EVOLVED RUBRICS: Evolved rubrics derive from rollouts and target the hard location split between field and woods.The generator weights this distinction as highly as the grounding rubric.
  • B.1 CASE STUDY: The case study asks why the uncle says the audio’s last sentence, with answer choices including Superman imitation, praise, misunderstanding, and admiration.These alternatives frame the distinction between surface topic recognition and underlying communicative intent.
  • B.1 CASE STUDY: The GRPO baseline latches onto the surface topic Superman and selects A, while AUDIORUBRICS correctly infers that the uncle misreads a correction as praise.This contrast demonstrates the model’s ability to follow speaker intent across the dialogue.
  • B.2 COMPARISON BETWEEN INITIAL AND EVOLVED RUBRICS: Evolved rubrics add question-specific discrimination that the fixed rubric set lacks.Their rollout-conditioned focus targets distinctions that initial criteria do not separate.

C ADDITIONAL EXPERIMENTAL RESULTS · C.1 ANALYSIS ON RESPONSE LENGTH

The response-length analysis shows that vanilla GRPO collapses toward near-zero responses, removing the length penalty causes unbounded growth, and the full method maintains stable response length throughout training.

  • C.1 ANALYSIS ON RESPONSE LENGTH: Vanilla GRPO steadily collapses response length toward near-zero by the end of training.The policy learns terse, under-reasoned answers that maximize outcome reward without producing an intermediate reasoning trace.
  • C ADDITIONAL EXPERIMENTAL RESULTS: Figure 7 compares average response length across three reward configurations over the course of training.
  • C ADDITIONAL EXPERIMENTAL RESULTS: The observed pathologies are opposite: vanilla GRPO favors response collapse, whereas removing the length penalty favors unbounded growth.
  • C ADDITIONAL EXPERIMENTAL RESULTS: The full method avoids both response-length pathologies while preserving a stable training trajectory.
  • C.1 ANALYSIS ON RESPONSE LENGTH: Removing the length penalty produces unbounded response-length growth during training.
  • C.1 ANALYSIS ON RESPONSE LENGTH: The full reward design maintains a stable response length throughout training.

C.2 ROBUSTNESS TO DIFFERENT MODEL SCALE · D PROMPTS

AudioRubrics remains effective at the 3B scale, outperforming GRPO across all three evaluated benchmarks. The prompts and algorithm specify audio-grounded rubric generation, adaptive selection, variance filtering, weighting, and response judging.

  • C.2 ROBUSTNESS TO DIFFERENT MODEL SCALE: 1.23%, 1.72%, and 3.65% relative improvements over GRPO were achieved on MMAU, MMAR, and MMSU, respectively, at the 3B scale.The comparison used GRPO trained under an identical setting, and gains persisted despite reduced model capacity.
  • D PROMPTS: The pipeline includes benchmark inference, static rubric-generation, evolving-rubric, weight-assignment, and rubric-judging prompts.The prompts correspond to inference and the two calls used in the evolving rubric process.
  • D PROMPTS: Algorithm 1 combines previous and newly generated rubrics, judges each rubric across rollouts, and polarity-normalizes verdicts into rubric scores.Positive rubrics use the verdict directly, while negative rubrics use 1 minus the verdict.
  • D PROMPTS: Variance filtering drops criteria passed or failed by every rollout, uses a 0.5 neutral fallback when none survive, and retains the top-M discriminative rubrics.The retained set becomes the previous rubric set for the next iteration.
  • D PROMPTS: The weight-assignment prompt outputs K positive numbers summing to 1.0, prioritizing rubrics that directly test grounding in the specific audio.The resulting weights replace the initial weights and are used to compute rubric rewards.
  • D PROMPTS: The static prompt asks an audio-understanding evaluator to listen to the clip and generate exactly 5 rubrics for each question-answer pair.The reference answer is treated as one possible student response rather than necessarily a good one.
  • D PROMPTS: The evolving prompt seeks discriminative criteria that distinguish response quality and capture grounding in acoustic evidence rather than text-only priors.It targets subtle quality differences that existing rubrics miss in audio reasoning responses.
  • D PROMPTS: The judging prompt asks whether each candidate response satisfies each of 5 binary rubrics using Yes or No decisions.The evaluator listens to the attached audio while reading the question, response, and rubrics.
Loading 2608.02831v1…