Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

1 INTRODUCTION

• Outcome-based methods reward only final answers, leaving reasoning unsupervised, while process-based methods score reasoning but use coarse, fixed criteria (Li et al., 2025; Rouditchenko et al., 2025; Fan et al., 2025).The former can allow correct answers without sound reasoning; the latter can reward fluent reasoning unsupported by the audio.Source: §1 · pp. 1–2
• Uniform fixed rubrics mismatch questions requiring perception alone versus multi-step inference, and become less informative as the policy improves and satisfies them.Perception-only questions can incur unnecessary reasoning overhead and hallucination risk, while multi-step questions need greater emphasis on logical soundness.Source: §1 · p. 2
• AUDIORUBRICS generates per-question rubrics from raw waveforms and combines them with verifiable RLVR rewards to assess reasoning logic and audio grounding.During training, it removes criteria with identical verdicts across a rollout group and adds harder criteria based on where stronger and weaker rollouts diverge; a linear length penalty discourages redundant, circular, or hallucinated reasoning.Source: §1 · p. 2
• Across MMAU, MMAR, and MMSU, AUDIORUBRICS consistently outperforms a wide range of open-source models and RL baselines.Rubrics become progressively more advanced as the policy improves, and ablations verify the contribution of each component.Source: §1 · p. 2

2.1 GROUP RELATIVE POLICY OPTIMIZATION (GRPO)

• GRPO samples candidate responses, scores them, and optimizes a weighted reward while using KL regularization to limit policy divergence.The KL term penalizes divergence between the current policy and a reference policy; the clipping operation is omitted for simplicity.Source: §2.1 · p. 3

2.2 RUBRIC AS REWARDS

• Sample-wise rubrics specify weighted natural-language criteria per query, and a judge LM scores a response by checking whether it satisfies each criterion.Rubric weights sum to one, and training optimizes the expected rubric score over training questions.Source: §2.2 · p. 3
• Rubric-based rewards use concrete criteria to reduce susceptibility to judge bias and support consistent, comparable scores across judge-model runs.The cited prior work on rubrics is Viswanathan et al. (2026) and Gunjal et al. (2025).Source: §2.2 · p. 3
• AudioRubrics initializes audio-grounded rubrics per question, refines them from policy rollouts, discards non-discriminative criteria, and combines rubric and outcome rewards for GRPO.The resulting rubric scores are used in the GRPO optimization loop.Source: Figure 2 · p. 4

3 AUDIORUBRICS: RL WITH EVOLVING RUBRICS AS REWARDS

• AudioRubrics addresses unchecked, potentially audio-ungrounded reasoning by adding an evolving rubric reward that directly scores reasoning alongside the outcome reward.As the policy improves, the rubrics evolve to keep the reward signal informative throughout training.Source: §3 · p. 4

3.1 TASK FORMULATION

• For each audio sample, the LALM reasons over audio input and a textual query to predict an answer matching the ground truth.The dataset samples comprise an audio input, query, and corresponding ground-truth answer.Source: §3.1 · p. 4

3.2 REWARD DESIGN

• The framework initializes one positively phrased, weighted rubric per evaluation dimension using the raw waveform, anchoring criteria in acoustic evidence.A single audio-capable model serves as both rubric generator and judge; initial rubrics are generated once per question before training.Source: §3.2 · pp. 4–5
• At each iteration, the model proposes nonredundant rubrics from current rollouts, including negative criteria that capture recurring failure modes absent from the initial positive rubrics.Surviving rubrics from the previous iteration are carried forward, allowing harder criteria to accumulate as the policy improves.Source: §3.2 · p. 5
• Rubrics with identical judgments across a rollout group are pruned, and the most discriminative survivors are retained to provide group-relative learning signal.A rubric is discarded if all rollouts satisfy it or none do, since either case gives every rollout the same verdict.Source: §3.2 · p. 5
• The judge reweights retained rubrics and computes rubric rewards separately for each prompt group from its actual rollouts, updating the signal at every training step.These per-group updates keep the reward targeted to the current policy.Source: §3.2 · p. 5
• A reasoning-length penalty counterbalances the rubric reward’s preference for longer traces, aiming to keep reasoning informative but concise.The penalty scales with reasoning-trace token length relative to a reference length.Source: §3.2 · p. 5
• The final reward combines outcome correctness, dense rubric supervision, and the length penalty, and the LALM is optimized with GRPO.Setting both rubric and length weights to zero recovers accuracy-only RLVR.Source: §3.2 · pp. 5–6

4.1 EXPERIMENTAL SETUPS

• Across the three benchmarks, the experiments compare AUDIORUBRICS with proprietary, open-source, and training-based models using greedy decoding and accuracy.The evaluation covers MMAU Test-mini, MMAR, and MMSU, spanning audio questions across sound, music, speech, and mixed modalities, from perception to multi-step reasoning.Source: §4.1 · p. 6

4.2 MAIN RESULTS

• AUDIORUBRICS attains the best overall accuracy among similar-size models on all three benchmarks, with strong results across sound, speech, and mixed-modality splits.It ranks first on the Sound category of MMAU and MMAR, leads similar-size models on Speech in both, and leads or matches the best similar-size baseline on three of MMAR’s four mixed-modality splits.Source: §4.2 · p. 6, Table 2 · p. 7, Table 3 · p. 8
• By 4.3 points (8.9% relative improvement), AUDIORUBRICS surpasses the best comparable-size baseline on MMSU perception, with gains across all three perception dimensions.The authors interpret this as evidence that process-level supervision helps anchor reasoning in heard acoustic evidence.Source: §4.2 · pp. 6–7, Table 2 · p. 7

4.3 FURTHER ANALYSIS ON REWARD DESIGN AND JUDGEMENT

• Accuracy peaks at rubric weight γ = 0.5; higher weights reduce performance by downweighting accuracy reward and prioritizing rubric satisfaction over correct answers.Adding rubric reward consistently improves over GRPO across the tested weight range.Source: §4.3 · pp. 7–8, Table 4 · p. 7
• At overthinking-penalty weight δ = 0.15, performance peaks; smaller penalties allow overlong traces, while larger penalties compress reasoning excessively.Overlong traces can weaken the accuracy reward, whereas excessive compression weakens reasoning ability.Source: §4.3 · p. 8, Table 5 · p. 8
• Replacing Gemini-3.1-Pro with GPT-audio-1.5 as rubric generator and judge lowers performance below vanilla GRPO, indicating that rubric quality depends on capable, well-grounded generation and judging.The Gemini-3.1-Pro run pulls ahead after warmup, while the GPT-audio-1.5 variant learns more slowly and plateaus lower.Source: Figure 3 · p. 7, §4.3 · p. 8

4.4 ABLATION STUDY

• Ablations show that RL, rubric rewards, evolving rubrics, and the overthinking penalty each contribute to performance when combined.Initial rubrics assess reasoning quality and audio perception; evolving rubrics target current weaknesses with more advanced criteria, while the penalty limits excessively long traces that can hurt answer accuracy.Source: §4.4 · pp. 8–9, Table 6 · p. 9

4.5 RUBRICS EVOLVING ANALYSIS

• Newly evolved rubrics rise from roughly 35% of adopted reward criteria early in training to about 60% by the end, as easier initial criteria are increasingly pruned.This shift lets supervision track the model’s current weaknesses with more advanced, fine-grained criteria rather than relying on a frozen rubric list.Source: §4.5 · p. 9

Audio Understanding and Reasoning

• Audio reasoning advances from CLIP-adapted representations to pretrained LALMs, whose reasoning is sharpened by trace fine-tuning, tool use, or reinforcement learning (Radford et al., 2021; Wu et al., 2022; Ghosh et al., 2025; Chu et al., 2023; Xu et al., 2025; Chu et al., 2024; Li et al., 2026; Chen et al., 2026a; Tong et al., 2026; Lee et al., 2025; Fan et al., 2025; Tian et al., 2025; Yu et al., 2026a).Unlike methods rewarding only the final answer or using fixed, hand-crafted criteria, AUDIORUBRICS supervises reasoning with evolving, audio-grounded rubrics tailored to each question.Source: §5 · p. 9

Rubric as Rewards

• Rubric rewards address open-ended tasks with fine-grained, instance-specific criteria elicited offline or online, and have extended from text to vision-language and omni-modal reasoning (Bi et al., 2025; Raghavendra et al., 2026; Mu et al., 2024; Zhou et al., 2025; Tyagi et al., 2026; Kim et al., 2024; Gupta et al., 2025; Liu et al., 2025; Shen et al., 2026; Shao et al., 2025; Sheng et al., 2026; Rezaei et al., 2025; Xu et al., 2026; Jia et al., 2025; Yu et al., 2026d; Chen et al., 2026b; Qiu et al., 2026; Kong et al., 2026).The passage contrasts this multi-criteria judgment with scalar verifiable rewards, which excel when correctness is programmatically checkable.Source: §5 · p. 9
• AUDIORUBRICS extends rubric-based training to audio reasoning with evolving rubrics, supplying LALMs a finer-grained, adaptive learning signal.Its approach differs from prior text- and vision-centric rubric efforts by grounding supervision in audio reasoning.Source: §5 · p. 9

• AUDIORUBRICS supervises audio reasoning with rubric-based rewards, evolving rubrics from the model’s own rollouts to elicit harder criteria as the model improves.Source: §6 · pp. 9–10
• AUDIORUBRICS substantially outperforms a wide range of baselines on audio understanding and reasoning benchmarks.Source: §6 · p. 10

A IMPLEMENTATION DETAILS

• Qwen2.5-Omni-7B (Xu et al., 2025) is trained with GRPO for 400 steps using batch size 8, learning rate 1e-6, and G=8 rollouts at temperature 1.0.Training also uses a constant learning-rate schedule and KL coefficient βKL=0.001.Source: A IMPLEMENTATION DETAILS · p. 14
• Reward weights are α=0.9 for accuracy, β=0.1 for format, γ=0.5 for rubric reward, and δ=0.15 for overthinking penalty, with target reasoning length L=256 tokens.The weights follow the sensitivity analyses in Section 4.3.Source: A IMPLEMENTATION DETAILS · p. 14
• Gemini-3.1-Pro serves as rubric generator and judge Φ, keeping the top M=5 discriminative rubrics per group and eliciting up to Nnew=3 rubrics at each step.The checkpoint from the final training step is used for all evaluations.Source: A IMPLEMENTATION DETAILS · p. 14
• Training uses maximum prompt and response lengths of 4096 and 1024 tokens, respectively, bf16 precision, and four H100 GPUs.The batch size is 8.Source: A IMPLEMENTATION DETAILS · p. 14

B.1 CASE STUDY

• AudioRubrics correctly infers the uncle misreads a correction as praise, while GRPO answers A based on the surface topic, Superman.The case study contrasts tracking speaker intent across the dialogue with latching onto a lexically salient token.Source: §B.1 · p. 14, Figure 5 · p. 15
• GRPO gives a one-line justification that restates the surface topic without reasoning about the speaker’s underlying intent.It anchors on the most lexically salient audio token.Source: §B.1 · p. 14
• AudioRubrics reconstructs the interaction step by step, tracks dialogue state, and rules out distractors against conversational flow rather than isolated keywords.The case-study figure describes this as tracking the speaker’s intent across the dialogue.Source: §B.1 · p. 14, Figure 5 · p. 15
• The evolving rubric-based process reward credits intermediate reasoning steps, incentivizing grounding in the full context rather than a single salient cue.Source: §B.1 · p. 14

B.2 COMPARISON BETWEEN INITIAL AND EVOLVED RUBRICS

• Kept initial rubrics cover sound grounding and ruling out water options, subtasks that both correct and incorrect rollouts already pass.Source: §B.2 · p. 14
• Evolved rubrics derive from rollouts and target the location question’s field-versus-woods distinction, weighted as highly as sound grounding.The contrast is between the easy subtasks covered by kept initial rubrics and the harder answer split targeted by evolved rubrics.Source: §B.2 · pp. 14–15
• Evolved rubrics add question-specific discrimination that the fixed set lacks.Source: §B.2 · p. 15

C.1 ANALYSIS ON RESPONSE LENGTH

• The full method maintains stable response length, while vanilla GRPO collapses toward near-zero length and removing the length penalty causes unbounded growth.The response-length paths indicate that rubric-based process rewards prevent degenerate collapse, while length regularization discourages uninformative padding.Source: §C.1 · p. 15, Figure 7 · p. 15
• Rubric-based process rewards and length regularization complement each other, supporting substantive reasoning while keeping rationales concise yet sufficiently detailed.The paper reports that this stable, well-calibrated reasoning length correlates with accuracy improvements in Table 2.Source: §C.1 · p. 15
• In a qualitative case, the rubric generator adds a highly weighted “field vs. woods” disambiguation criterion while retaining static criteria for easier subtasks.The new criterion targets the crux of the question and is weighted as highly as the retained grounding criterion.Source: Figure 6 · p. 16

C.2 ROBUSTNESS TO DIFFERENT MODEL SCALE

• At 3B scale, AudioRubrics outperforms identically trained GRPO on all three benchmarks, with relative gains of 1.23% on MMAU, 1.72% on MMAR, and 3.65% on MMSU.The gains persist despite the smaller model’s reduced capacity for reasoning-oriented rewards to take effect.Source: §C.2 · p. 16

D PROMPTS

• The prompt appendix includes an inference prompt for benchmark evaluation and a static prompt for generating initial per-question rubrics.The inference prompt specifies greedy decoding at temperature 0.Source: D PROMPTS · p. 16, Figure 8 · p. 17, Figure 9 · p. 18
• For each prompt group, the algorithm combines retained rubrics with newly generated, polarity-tagged candidates and judges each rubric against the rollouts.The retained set is initialized from weighted rubrics and carried forward between iterations.Source: D PROMPTS · p. 17
• The algorithm filters out rubrics passed or failed by every rollout, keeps the top-M criteria by discrimination, and uses a neutral fallback if none survive.The fallback assigns a constant reward, which contributes no advantage.Source: D PROMPTS · p. 17
• The retained rubrics receive normalized importance weights, which are used with polarity-normalized verdicts to score each rollout.The updated weights supersede the initial weights.Source: D PROMPTS · p. 17
• The appendix also supplies prompts for evolving rubrics, jointly generating candidates and judging rollouts, assigning weights, and checking response compliance.These correspond to the two generator–judge calls in the pipeline.Source: D PROMPTS · p. 17, Figure 10 · p. 19, Figure 11 · p. 20