Incentivizing Consistent, Effective and Scalable Reasoning Capability in Audio LLMs via Reasoning Process Rewards

1 INTRODUCTION

• Audio LLM reasoning can degrade inference performance, with longer chains worsening results and sometimes underperforming direct answers.The authors identify hallucination, inconsistency, and logically unsound reasoning as problems associated with inadequate training.Source: §1 · pp. 1–2
• The authors argue that outcome-only rewards and template imitation leave reasoning-process failures unaddressed, allowing errors to accumulate over longer chains.They contrast process-oriented training with supervised fine-tuning and verifiable-reward methods focused on final-answer correctness and format.Source: §1 · p. 2
• CESAR uses GRPO and multi-faceted process rewards for correctness, format, consistency, structure, causal reasoning, domain knowledge, and calibrated reasoning depth.The approach shifts from outcome verification to cultivating reasoning as a controllable, trainable skill.Source: §1 · p. 2
• CESAR models gain from test-time scaling at model-specific reasoning sweet spots, while poorly trained models can degrade with longer chains.The paper describes these gains as training-free performance improvements at inference.Source: §1 · pp. 2–3
• CESAR achieves state-of-the-art results on MMAU Test-mini, surpasses GPT-4o Audio and Gemini 2.5 Pro, and reaches near-human-level performance on MMSU reasoning tasks.The paper also reports that enhanced reasoning improves multimodal reasoning and perception capabilities.Source: §1 · p. 3
• AI-as-judge evaluations and qualitative comparisons report reduced hallucination, stronger logical coherence, and improved reasoning-answer consistency.The evaluation framework also reports commanding win rates against strong baselines.Source: §1 · p. 3

Audio Large Language Models

• Audio LLMs progressed from contrastive cross-modal understanding to decoder-based generation and increasingly comprehensive multimodal systems.The related-work section describes the research frontier as enabling models to reason about the acoustic world.Source: §2 · p. 3

The Limits of Supervised Reasoning

• CoT supervised fine-tuning can teach reasoning-template imitation rather than analytical skill, leaving traces brittle and prone to failure on complex unseen problems.The section links these limitations to hallucination and inconsistency.Source: §2 · p. 3

The Untapped Potential of Reinforcement Learning

• Audio RL methods using outcome-only rewards optimize final-answer correctness but do not penalize logical fallacies or reward coherent analytical processes.The paper positions granular, process-oriented rewards as its response to this gap.Source: §2 · p. 3

3.1 PROBLEM FORMULATION

• Outcome-only rewards can leave reasoning uncontrolled, inconsistent with answers, and lacking structured strategies, so CESAR treats reasoning as a trainable capability.CESAR’s process-oriented rewards are intended to make reasoning more controllable rather than relying on spontaneous emergence.Source: §3.1 · p. 4, §3.2 · p. 4

3.2 FROM OUTCOME-BASED TO PROCESS-ORIENTED REASONING CONTROL

• CESAR combines verifiable correctness and format rewards with process rewards for reasoning quality, consistency, and conciseness.Accuracy receives weight 5.0, while the other reward components receive weight 1.0.Source: §3.2 · p. 4

3.2.1 FOUNDATION: VERIFIABLE CORRECTNESS AND STRUCTURAL INTEGRITY

• Binary accuracy and format rewards preserve answer correctness and require structured outputs with both reasoning and answer components.The accuracy signal anchors process improvements in correct answers, while format compliance requires XML-tagged reasoning and answer sections.Source: §3.2.1 · pp. 4–5

3.2.2 SEMANTIC COHERENCE AND REASONING-ANSWER ALIGNMENT

• The consistency reward aligns each reasoning trace with both its answer and the question context, targeting disconnected conclusions and off-topic hallucination.The question context includes the question text and available choices.Source: §3.2.2 · p. 5

3.2.3 INCENTIVIZING STRUCTURED REASONING AND PENALIZING OVERTHINKING

• Keyword rewards encourage structured patterns, logical reasoning, and audio-domain knowledge, while a length penalty discourages inefficient overthinking.The penalty targets redundant, verbose traces that can accumulate errors and is normalized by the maximum output length, set to 256 in practice.Source: §3.2.3 · pp. 5–6

3.3 CULTIVATING REASONING CAPABILITY VIA ONLINE RL

• CESAR uses online GRPO to cultivate reasoning from sampled responses and applies answer-preserving question paraphrases to improve robustness to linguistic variation.Augmentation changes question phrasing while keeping the audio and ground-truth answer unchanged.Source: §3.3 · p. 6

3.4 UNLOCKING REASONING CAPABILITY VIA TEST-TIME SCALING

• CESAR’s test-time scaling analysis finds performance gains at its reasoning sweet spot, while baseline models show limited improvement or continued degradation.The method evaluates performance across maximum thinking lengths and defines the sweet spot as the length at which performance peaks.Source: §3.4 · p. 6
• The MMAU Test-mini and MMSU tables report accuracy results, while the task-wise MMAU figure normalizes scores by CESAR.The supplied table captions identify benchmark scope but do not provide cell values for comparisons.Source: Table 1 · p. 7, Figure 2 · p. 7, Table 2 · p. 7

4.1 EXPERIMENTAL SETUP

• The evaluation uses MMAU Test-mini and MMSU as out-of-distribution benchmarks, with Qwen2.5-Omni-7B trained using GRPO on augmented AVQA data.Training samples K = 8 responses per example; correctness is weighted α1 = 5.0 and other rewards α2−5 = 1.0.Source: §4.1 · p. 7

4.2 MAIN RESULTS: STATE-OF-THE-ART PERFORMANCE ACROSS BENCHMARKS

• CESAR establishes state-of-the-art performance on MMAU Test-mini, with gains over the base model in both reasoning and non-reasoning modes and improvements over outcome-only RL methods.The reasoning mode also delivers larger benefits than the Ke-Omni-R baseline.Source: §4.2 · p. 7

4.3 CURING TEST-TIME INVERSE SCALING AND UNLOCKING SCALABLE REASONING

• 77.1% peak accuracy is reached by full CESAR with an approximately 35-40-token reasoning chain, while CESAR without the overthinking penalty peaks at 76.50%.Baselines instead show catastrophic collapse or volatile performance without clear benefit from longer reasoning.Source: §4.3 · p. 8

4.4 AI-AS-JUDGE EVALUATION: QUANTIFYING REASONING QUALITY BEYOND ACCURACY

• CESAR's reasoning process achieves commanding AI-judge win rates against both baselines, and including the Overthinking Penalty further elevates the win rate.Figure 3's right panel reports the GPT-4o Audio head-to-head evaluation.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 QUALITATIVE ANALYSIS: CONCRETE EVIDENCE OF REASONING IMPROVEMENTS

• Qualitative examples contrast CESAR's structured acoustic analysis with baseline errors, including Ke-Omni-R reasoning that identifies three telephone rings but answers “2.”Table 3 marks structured reasoning in green and reasoning errors in red italics.Source: §4.5 · p. 9, Table 3 · p. 9

4.6 ABLATION STUDY: QUANTIFYING COMPONENT CONTRIBUTIONS

• Ablations report that removing RL causes a catastrophic performance collapse, while Keywords yields the largest single gain over outcome-only RL and the remaining components add gains toward peak performance.Table 4 progressively removes components; its scores use reasoning at each variant's reasoning sweet spot.Source: §4.6 · p. 9, Table 4 · p. 9

• CESAR shifts from outcome verification to process rewards with GRPO, addressing test-time inverse scaling attributed to inadequate optimization of reasoning processes.The conclusion reports state-of-the-art results surpassing GPT-4o Audio and Gemini 2.5 Pro.Source: §5 · p. 10
• Test-time scaling is described as harmful for poorly trained models but beneficial for CESAR at discovered reasoning sweet spots.Source: §5 · p. 10
• Across out-of-distribution benchmarks, the conclusion reports synergistic improvement in multimodal reasoning and perception, supported by AI-as-judge and qualitative evaluations of reasoning quality.Source: §5 · p. 10

• CESAR shifts from outcome verification to process rewards, framing reasoning as a controllable skill measurable beyond task accuracy.The discussion contrasts this approach with supervised fine-tuning that mimics templates and outcome-only reinforcement learning.Source: A DISCUSSION · pp. 15–16
• The base Qwen2.5-Omni model fell from 68.60% to 65.20% with unguided reasoning, which CESAR reverses by rewarding internal consistency.The discussion attributes longer-chain degradation to opportunities for logical errors and hallucinations to compound.Source: A DISCUSSION · p. 15
• AI-as-judge evaluations report commanding CESAR win rates and reductions in hallucination, alongside improved logical coherence and reasoning-answer consistency.The authors present these evaluations as quantitative evidence of improved reasoning quality.Source: A DISCUSSION · p. 16
• A model-specific reasoning sweet spot becomes accessible after robust reasoning is cultivated; test-time scaling then identifies an optimal inference budget.The discussion says the value of increased computation depends on the learned policy’s quality.Source: A DISCUSSION · p. 16
• 5.1% direct-answering improvement over the base model (73.70% vs. 68.60%) accompanied improved MMSU perception scores.The authors describe this as a synergistic effect of deliberate reasoning on intuitive answering and perception.Source: A DISCUSSION · p. 16
• 88.72% semantic reasoning exceeded the human score of 82.16%, while CESAR’s 48.45% foundational perception remained below human acuity at 91.24%.The authors identify foundational perception as the remaining performance bottleneck.Source: A DISCUSSION · p. 16

A.1 LIMITATIONS

• CESAR’s reasoning approaches human levels, but foundational perception remains a bottleneck: 48.45% versus human acuity at 91.24%.The authors identify this as a limitation affecting current models and a priority for the field.Source: §A.1 · p. 17
• GRPO training samples multiple responses per input, making the regimen computationally intensive and potentially difficult for groups with limited hardware.A standard training run requires significant GPU resources.Source: §A.1 · p. 17
• The multi-faceted reward suite requires balancing component weights; normalization and a higher accuracy-reward weight yielded the best reported results.The authors suggest prioritizing accuracy may guide early optimization before process-oriented rewards refine the policy.Source: §A.1 · p. 17

A.2 FUTURE WORKS

• Current audio encoders appear to be the main perceptual bottleneck; the authors propose richer audio representations and architectural innovations as research directions.They also propose using improved reasoning capabilities to assess whether perceptual advances improve end-to-end performance.Source: §A.2 · p. 18
• Testing process-oriented training in vision, robotics, or other modalities could help determine whether its principles extend beyond audio.The authors suggest early experiments in visual question answering or robotic planning.Source: §A.2 · p. 18

A.3 THE USE OF LARGE LANGUAGE MODELS (LLMS)

• The authors used LLMs for language polishing, grammar, readability, and flow, while stating that core research contributions were developed independently.They say LLMs did not contribute to conceptual development, experimental validation, or result interpretation.Source: §A.3 · p. 19

A.4 ETHICS

• The work aims to enhance multimodal reasoning without introducing additional ethical concerns or resolving existing ones.Source: §A.4 · p. 20

B.1 BASELINE METHODS

• The baseline comparison includes zero-shot and CoT-prompted Qwen2.5-Omni-7B, outcome-only RLVR Ke-Omni-R, proprietary systems, and SFT-based audio models.Ke-Omni-R shares the Qwen2.5-Omni-7B base architecture and GRPO training but rewards final-answer correctness; other comparisons include Gemini and GPT-4o Audio.Source: §B.1 · p. 21

B.2 EVALUATION BENCHMARKS

• MMAU, MMSU, and MMAR test CESAR on out-of-distribution audio tasks spanning expert-level cognition, separate perception and reasoning skills, and demanding multi-step reasoning.The benchmarks contain approximately 1,000 MMAU questions across 27 skills, 5,000 MMSU pairs across 47 tasks, and 1,000 MMAR tasks with longer, composite audio clips.Source: §B.2 · p. 22

B.3 TRAINING DATA AND AUGMENTATION STRATEGY

• Answer-invariant question rephrasing expands AVQA with linguistic variations intended to discourage superficial text correlations and target specific reasoning capabilities.Examples include temporal reasoning, counting, and comparative analysis question templates.Source: §B.3 · p. 23

B.4 TRAINING HYPERPARAMETERS AND PROMPTING CONFIGURATION

• CESAR fine-tunes Qwen2.5-Omni-7B with GRPO, shares core hyperparameters and inference prompts with Ke-Omni-R, and varies the maximum reasoning length to study test-time scaling.Training uses AdamW with a 1e-5 learning rate, global batch size 32, and K = 8 responses per input; the accuracy reward weight is 5.0 and other process-reward weights are 1.0.Source: §B.4 · p. 24

B.5 COMPUTATIONAL RESOURCES

• A final-model training run on augmented AVQA takes approximately 2–3 days on eight NVIDIA H200 GPUs.Each GPU provides 141GB of HBM3e memory.Source: §B.5 · p. 25

B.6 DETAILED KEYWORDS

• The Keywords Reward scores structured analytical patterns, logical rigor, and domain-specific terminology to guide reasoning traces toward structured, logical, domain-aware analysis.It uses exact phrase matching and regular expressions to detect terms and patterns.Source: §B.6 · p. 26

B.7 REPRODUCIBILITY

• The paper documents its methodology, configurations, augmentation, benchmarks, and reward specifications, and states that code and trained models will be made publicly available upon publication.The training pipeline builds on Ke-Omni-R code, Qwen2.5-Omni-7B, and publicly available training datasets.Source: §B.7 · p. 28

C ALGORITHM PSEUDOCODE

• The pseudocode describes an online GRPO loop that samples responses, calculates multi-faceted rewards, computes advantages, and updates policy parameters.The reward combines accuracy, format compliance, semantic consistency, keyword-based reasoning scores, and an overthinking penalty.Source: C ALGORITHM PSEUDOCODE · p. 29

D.1 BENCHMARK RESULTS ON MMAU

• 77.10% accuracy makes CESAR state of the art on MMAU Test-mini and surpasses the Gemini 2.5 series and GPT-4o Audio.The model has 7B parameters; the table reports accuracy against proprietary and open-source audio models.Source: §D.1 · p. 31, Table 8 · p. 31
• 73.70% accuracy without reasoning exceeds the base model’s 68.60% in the same setting.The authors interpret this as process-oriented RL improving direct answering as well as explicit reasoning.Source: §D.1 · p. 32
• Reasoning shifts base Qwen2.5-Omni-7B from 68.60% to 65.20% but shifts CESAR from 73.70% to 77.10% on MMAU Test-mini.These changes are a 3.4-point decline for the base model and a 3.4-point gain for CESAR.Source: §D.1 · p. 32

D.2 BEYOND AGGREGATE SCORES: A TASK-LEVEL ANALYSIS OF CONTROLLABLE REASONING CAPABILITY

• The two CESAR variants cover larger radar-chart areas than Ke-Omni-R and Qwen2.5-Omni-7B, while displaying distinct task-difficulty profiles.Figure 4 normalizes performance to CESAR (Ours) as the 100% baseline on each axis.Source: Figure 4 · p. 33, §D.2 · p. 34
• The depth specialist uses substantially more reasoning tokens for challenging tasks, whereas the calibrated generalist achieves comparable overall performance under the overthinking penalty.Figure 5 includes reasoning-token efficiency and highlights trade-offs between reasoning depth and computational cost.Source: Figure 5 · p. 33, §D.2 · p. 34
• CESAR without the overthinking penalty is a depth specialist, excelling on Hard tasks at the cost of efficiency on simpler problems.The variant consistently exceeds the 100% baseline on Hard tasks.Source: §D.2 · p. 34
• Full CESAR is a calibrated generalist, stable on Easy and Medium tasks while remaining competitive on Hard tasks.The authors characterize it as efficient and consistent across problem complexities.Source: §D.2 · p. 34
• The authors attribute the variants’ different reasoning profiles to systematic modulation of the overthinking penalty, enabling profiles suited to research analysis or efficient production use.The depth specialist is described as suited to complex tasks where thorough analysis justifies its computational cost; the generalist is described as suited to production systems prioritizing efficiency and consistency.Source: §D.2 · p. 34
• The authors present the two radar visualizations as evidence that CESAR can explore and engineer distinct reasoning profiles rather than relying on reasoning patterns to emerge spontaneously.They frame reasoning behavior as a design parameter that can be specified and validated.Source: §D.2 · pp. 34–35

D.3 BENCHMARK RESULTS ON MMSU

• 81.07% average reasoning accuracy approaches the human benchmark of 86.77% on MMSU; CESAR scores 88.72% on Semantic Reasoning.The authors report that the reasoning average is state of the art and the Semantic Reasoning score is super-human.Source: Table 9 · p. 36, §D.3 · p. 36
• 48.45% vs. 46.10% average perception accuracy puts 7B CESAR ahead of Gemini 1.5 Pro on MMSU.The table reports accuracy across perception and reasoning tasks and includes human performance as an upper-bound reference.Source: §D.3 · p. 36, Table 9 · p. 36
• The authors report that reasoning-oriented training also benefits perception, associating consistent, logical connections with improved processing of acoustic representations.They describe this as a synergy between improved reasoning and foundational perception.Source: §D.3 · pp. 36–37
• Perception remains a gap relative to human performance: CESAR’s perception score still trails the human score of 91.24%.The authors call this asymmetry a perceptual bottleneck and identify perception as a direction for future research.Source: §D.3 · p. 37

D.4 TEST-TIME SCALING ANALYSIS: FROM INVERSE SCALING TO CONTROLLABLE AND SCALABLE REASONING

• Base Qwen2.5-Omni-7B collapses as chains lengthen, whereas RL models stay robust; full CESAR peaks around 30–40 tokens, then stabilizes.The full method outperforms both CESAR without the Overthinking Penalty and Ke-Omni-R in the zoomed-in comparison.Source: Figure 6 · p. 38, §D.4 · p. 39

D.5 QUANTIFYING REASONING QUALITY BEYOND ACCURACY: AN AI-AS-JUDGE FRAMEWORK

• CESAR’s reasoning wins 76.4% of comparisons against Qwen2.5-Omni-7B and 63.85% against Ke-Omni-R in AI-as-judge evaluations.The framework evaluates reasoning quality separately from final-answer correctness.Source: Figure 7 · p. 40, §D.5 · p. 40

D.6 QUALITATIVE ANALYSIS: THE ANATOMY OF CULTIVATED REASONING

• Qualitative trace comparisons are presented as evidence that process rewards address hallucination, logical fallacies, and inconsistency.The examples contrast brittle, uncontrolled base-model output with reasoning shaped by CESAR.Source: §D.6 · p. 41, §D.6.1 · p. 41

D.6.1 COMPARISON WITH BASE MODEL (QWEN2.5-OMNI-7B): UNTRAINED VS. CULTIVATED REASONING

• Against the base model, CESAR grounds choices, decomposes acoustic scenes, and avoids a final-answer typo that nullifies otherwise sound reasoning.The examples highlight grounding in explicit answer choices, analysis of multiple acoustic components, and faithful answer execution.Source: §D.6.1 · p. 41

D.6.2 COMPARISON WITH RL BASELINE (KE-OMNI-R): THE NECESSITY OF PROCESS REWARDS

• Against outcome-only Ke-Omni-R, examples expose factual hallucination and reasoning-answer contradictions, while CESAR’s consistency reward links the final answer to its trace.The examples also contrast CESAR’s concise, domain-informed reasoning with Ke-Omni-R’s convoluted, self-contradictory monologue.Source: §D.6.2 · pp. 42–44

D.6.3 EFFICIENT REASONING VS. PATHOLOGICAL OVERTHINKING: THE CRITICAL ROLE OF THE OVERTHINKING PENALTY

• Without the Overthinking Penalty, a CESAR variant can abandon a correct laughter answer after speculative contextual reasoning, while full CESAR stays grounded in acoustic evidence.The comparison describes the penalty as critical for keeping reasoning tethered to evidence and preventing contextual hallucinations.Source: §D.6.3 · pp. 44–45

Ablation results

• 9.4 points separate Ke-Omni-R’s 74.60% reasoning accuracy from the base model’s 65.20%; the base model alone shows test-time inverse scaling.Every RL-trained variant gains performance from reasoning, unlike the base model.Source: Table 10 · p. 46, §D.7 · p. 46

Process-specific rewards

• Removing the Consistency reward causes a significant performance drop, while removing Key Words lowers accuracy 1.0%, from 76.20% to 75.20%.The consistency reward targets traces disconnected from final answers; Key Words reward logical structure and domain-specific terminology.Source: §D.7 · p. 46

Calibrated training

• Removing the Overthinking Penalty reduces the performance gap between reasoning and non-reasoning modes from 3.4 points (77.10% vs 73.70%) to 1.5 points (76.50% vs 75.00%).The authors associate the penalty with learning when deliberate reasoning is needed.Source: §D.7 · pp. 46–47
• Removing data augmentation causes a minor accuracy drop, while its role in exposing linguistic diversity is described as important for generalization and avoiding superficial correlations.The ablation characterizes augmentation as a smaller numerical contributor than the core process rewards.Source: §D.7 · p. 46

Overall ablation conclusion

• The authors conclude that the full framework improves both reasoning and non-reasoning settings, with components supporting more accurate, effective, consistent, and logical reasoning.The ablation presents the components as complementary contributions to the overall approach.Source: §D.7 · p. 47

Superiority of Process-Oriented Reinforcement Learning

• 62.70% vs. 36.80%: CESAR establishes a nearly 26-point lead over Audio-Reasoner on MMAR, while SFT models are described as brittle on complex multi-hop reasoning.The authors attribute the comparison to process-oriented RL versus imitation from static CoT datasets.Source: Table 11 · p. 48, §D.8 · p. 49

The Acoustic-Linguistic Divide

• CESAR achieves the highest MMAR scores in Sound (66.06%) and Music (55.83%), while top proprietary systems perform strongly on Speech-dominated tasks.The passage distinguishes acoustic reasoning from transcript-oriented methods’ strength on speech-centric tasks.Source: §D.8 · p. 48

The Frontier of Mixed-Modality Reasoning

• Mixed-modality categories remain difficult: even top-performing models show high variance and lack consistent dominance across MMAR’s Mix categories.The authors identify compositional understanding and temporal grounding across overlapping sources as an ongoing challenge.Source: Table 11 · p. 48, §D.8 · p. 49