Source-linked AI summary
Step-Audio-R1 Technical Report
Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, Daijiao Liu, Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Gang Yu
TL;DR
Audio language models have generally failed to benefit from extended reasoning, motivating the question of whether audio intelligence can deliberate effectively. Step-Audio-R1 introduces MGRD to ground reasoning in acoustic features, and it surpasses Gemini 2.5 Pro while performing comparably to Gemini 3 Pro across audio benchmarks.
Problem
Audio language models often perform worse with longer reasoning because they rely on textual surrogate reasoning rather than acoustic properties.
Method
MGRD iteratively distills reasoning from textual abstractions into acoustic grounding while using standardized reasoning formats and multimodal training.
Results
Step-Audio-R1 outperforms Gemini 2.5 Pro and achieves performance comparable to Gemini 3 Pro across comprehensive audio understanding and reasoning benchmarks.
Takeaways & Limitations
The results show that reasoning can benefit audio intelligence when extended deliberation is appropriately anchored in acoustic features.
Abstract
from arXiv · showhide
Recent advances in reasoning models have demonstrated remarkable success in text and vision domains through extended chain-of-thought deliberation. However, a perplexing phenomenon persists in audio language models: they consistently perform better with minimal or no reasoning, raising a fundamental question - can audio intelligence truly benefit from deliberate thinking? We introduce Step-Audio-R1, the first audio reasoning model that successfully unlocks reasoning capabilities in the audio domain. Through our proposed Modality-Grounded Reasoning Distillation (MGRD) framework, Step-Audio-R1 learns to generate audio-relevant reasoning chains that genuinely ground themselves in acoustic features rather than hallucinating disconnected deliberations. Our model exhibits strong audio reasoning capabilities, surpassing Gemini 2.5 Pro and achieving performance comparable to the state-of-the-art Gemini 3 Pro across comprehensive audio understanding and reasoning benchmarks spanning speech, environmental sounds, and music. These results demonstrate that reasoning is a transferable capability across modalities when appropriately anchored, transforming extended deliberation from a liability into a powerful asset for audio intelligence. By establishing the first successful audio reasoning model, Step-Audio-R1 opens new pathways toward building truly multimodal reasoning systems that think deeply across all sensory modalities.
1 Introduction
Audio language models often degrade with longer reasoning, unlike text and vision systems, because their deliberations rely on textual surrogates rather than acoustic properties. Step-Audio-R1 addresses this gap with MGRD and reports strong audio reasoning performance.
- Audio models often perform worse as reasoning length increases, across architectures, training methods, and model scales.
- Existing models frequently reason from transcripts or captions instead of directly analyzing acoustic properties.
- MGRD iteratively shifts reasoning from textual abstractions toward acoustic properties through self-distillation and refinement.
- Step-Audio-R1 outperforms Gemini 2.5 Pro and reaches performance comparable to Gemini 3 Pro across comprehensive audio benchmarks.
- The results indicate that reasoning can transfer to audio when deliberation is appropriately grounded in modality-specific evidence.
2 Model Overview
Step-Audio-R1 combines an audio encoder, adaptor, and language-model decoder to produce textual answers from audio. Its central innovation, MGRD, progressively grounds reasoning in acoustic features rather than transcribed semantics.
- The model consists of an audio encoder, an audio adaptor, and an LLM decoder for audio-based reasoning tasks.
- The frozen Qwen2 audio encoder outputs 25-Hz features, while the adaptor downsamples them to 12.5 Hz before connecting to the LLM.
- The Qwen2.5 32B decoder generates reasoning content followed by a final textual reply from latent audio features.
- MGRD iteratively transforms semantic thoughts into native audio reasoning grounded in acoustic nuances.
3 Data Preparation
The data preparation combines text and audio training sources to elicit reasoning while preserving audio understanding. A standardized think-tag format aligns samples with and without native chains of thought.
- Cold-start training uses 5 million samples, including 1B text tokens and 4B audio-side tokens.
- Audio data covers speech recognition, paralinguistic understanding, and standard audio question-answer dialogues.
- Audio chain-of-thought data generated by self-distillation constitutes 10% of the total audio dataset.
- Text data spans knowledge QA, continuation, role-playing, chat, emotional dialogue, mathematics, and code.
- Samples without native reasoning are prepended with an empty <think> tag followed by the response.
- The later RL dataset contains 5,000 samples: 2,000 text-only examples and 3,000 augmented speech-based QA examples.
4 Post-Training Recipes
Post-training first establishes reasoning and format alignment, then iteratively distills acoustically grounded chains through supervised refinement and multimodal reinforcement learning. The final model generates extended reasoning that attends to audio properties.
- 4.1 Foundation Training: Reasoning Initialization and Format Alignment: Supervised initialization combines task reasoning, conversational deliberation, and audio data to preserve multimodal capabilities.
- 4.1 Foundation Training: Reasoning Initialization and Format Alignment: RLVR uses verified outcomes to explore reasoning strategies while maintaining answer accuracy.
- 4.2 Modality-Grounded Reasoning Distillation: Each refinement iteration selects perceptual questions, samples candidate chains, and filters them for acoustic grounding, coherence, and correctness.
- 4.2 Modality-Grounded Reasoning Distillation: MGRD jointly trains on distilled acoustic reasoning and textual reasoning to anchor thoughts in acoustic properties.
- 4.2 Modality-Grounded Reasoning Distillation: Multimodal RL combines format and accuracy rewards, with correctness weighted at 0.8 and reasoning generation at 0.2.
- 4.2 Modality-Grounded Reasoning Distillation: Reasoning chains shift from textual surrogates to acoustic analyses such as minor-key progressions and descending melodic contours.
- 4.2 Modality-Grounded Reasoning Distillation: The final model generates extended reasoning chains that genuinely attend to audio properties, unlocking test-time compute scaling benefits.
5 Evaluation
Step-Audio-R1 is evaluated across speech-to-text and speech-to-speech benchmarks, covering audio understanding, reasoning, conversational speech, and real-time dialogue. It achieves strong benchmark performance while combining reasoning quality with rapid responsiveness.
- Evaluation Scope: The evaluation spans expert audio understanding, multi-step logical reasoning, spoken mathematical problems, conversational speech, and interactive spoken dialogue.These benchmarks assess both comprehension and generative reasoning across speech-based settings.
- Evaluation on Speech-to-Text Benchmarks: 83.6% average score places Step-Audio-R1 above Gemini 2.5 Pro and slightly below Gemini 3 Pro across speech-to-text benchmarks.The evaluation includes Big Bench Audio, Spoken MQA, MMSU, MMAU, and Wild Speech.
- Evaluation on Speech-to-Speech Benchmarks: 96.1% speech reasoning performance and 0.92 s first-packet latency show that Step-Audio-R1 Realtime combines strong reasoning with sub-second responsiveness.The speech-to-speech evaluation measures both reasoning over spoken content and response speed.
6 Empirical Study and Analysis
The empirical studies show that format rewards, acoustically informative data selection, and targeted correction support stable and effective audio reasoning. Extended, grounded reasoning improves performance, while uninformative data and text-derived self-cognition biases undermine training.
- 6.1 Extended Reasoning Benefits Audio: Format rewards improve MMAU accuracy from 76.5 to 77.7 and produce earlier, more stable convergence than training without them.Both configurations eventually reach approximately 0.75-0.80 mean reward, but the format-rewarded model reaches the 0.70 threshold earlier and shows lower later variance.
- 6.1 Extended Reasoning Benefits Audio: Without format rewards, reasoning length collapses from about 3000 tokens to below 1500, whereas format rewards sustain approximately 2300-2800-token chains.The format reward acts as a regularizer that preserves extended thought chains despite accuracy rewards favoring brevity.
- 6.1 Extended Reasoning Benefits Audio: 77.7 MMAU accuracy with full reasoning exceeds abbreviated or absent reasoning, showing that grounded extended deliberation benefits audio understanding.The study attributes prior reasoning degradation to inadequate grounding mechanisms rather than fundamental incompatibility with audio.
- 6.2 Strategic Data Selection: Moderately difficult problems yield higher, more stable rewards and sustained reasoning than consistently failed problems, whose training eventually collapses.Moderately difficult examples contain both successful and failed trajectories, providing more informative learning signals than persistently unsolved cases.
- 6.2 Strategic Data Selection: Scaling the audio reinforcement-learning dataset to 200K examples produces no performance improvement, indicating that curated data quality outweighs indiscriminate quantity.The larger dataset introduces noise from acoustically ambiguous or inherently unsolvable problems.
- 6.3 Self-Cognition Correction Through Iterative Refinement: Targeted iterative correction reduces self-cognition errors from 6.76% to 2.63%, then to 0.02% after DPO alignment with 8,000 preference pairs.The final preference-optimization stage provides the largest reduction, effectively eliminating incorrect claims of inability to process audio.
7 Conclusion
The paper identifies textual surrogate reasoning as the main obstacle to audio reasoning and introduces Step-Audio-R1 with MGRD to ground deliberation in acoustic analysis. Evaluations report stronger performance than Gemini 2.5 Pro and results comparable to Gemini 3 Pro across diverse audio benchmarks.
- Textual surrogate reasoning makes models rely on transcripts or captions instead of acoustic properties.
- MGRD iteratively shifts reasoning from text-based patterns to genuine acoustic analysis.
- Step-Audio-R1 is reported to outperform Gemini 2.5 Pro and perform comparably to Gemini 3 Pro across complex audio understanding and reasoning benchmarks.
8 Contributors
The listed contributors include core contributors Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, and Daijiao Liu, alongside additional contributors.
- Fei Tian, Xiangyu Tony Zhang, Yuxin Zhang, Haoyang Zhang, Yuxin Li, and Daijiao Liu are listed as core contributors.
- The contributor list includes affiliations or contribution markers represented by superscript numerals.
- Yayue Deng, Donghang Wu, Jun Chen, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Daxin Jiang, and Gang Yu are listed as contributors.
A.1 Audio Reasoning Examples
The examples present audio reasoning across music and speech-related analysis, including requests for expert musical appreciation and analysis of acoustic details. The displayed reasoning discusses structure, instrumentation, harmony, rhythm, and production choices.
- The examples cover paralinguistic and musical analysis, including vocal characteristics, structure, instrumentation, and production.
- The analysis adapts its explanation to the user’s request for a professional, detailed musical perspective.
- The model’s reasoning considers song structure, instrumental layers, vocal harmonies, rhythm, key, and tempo.
- The reasoning also plans to organize the response from broad musical character toward detailed layers and explanatory interpretation.
I. Structural & Harmonic Genius (The “Maestro’s Blueprint”)
The supplied examples describe musical structure, harmony, production, and lyrical alignment, alongside a separate background-scene classification example. They emphasize how specific musical elements are interpreted as contributing to tension, texture, and setting.
- Structural & Harmonic Genius: The clip is described as having a C# minor key, an AABA structure, and harmonic tension centered on a B7#9 chord.
- Production Mastery: Layered harmonies, sustained vocals, arpeggiated piano inversions, half-time rhythm, and sub-bass are presented as production features.
- Lyrical-Musical Synergy: Several examples connect lyrics with musical execution, including accelerating piano arpeggios, intensified vibrato, and a brief drum drop.
- Structural & Harmonic Genius: The analysis characterizes #9 chords as uncommon in mainstream pop and associates harmonic dissonance with the song’s danger motif.
- The background-scene reasoning acknowledges that the transcript does not provide direct access to the actual audio background.
- A separate classification example selects traffic street as the most plausible background scene for a clear formal statement with subtle urban ambience.
A.2 Case Studies in Self-Recognition
The case study shows the model inferring occupational information from technical terminology and explanatory speech patterns, while acknowledging that it cannot directly hear the sound. These clues point broadly toward statistics, data science, or machine learning roles.
- The model explicitly states that it cannot directly hear the sound and instead infers professional information from specialized terminology and speech content.
- Terms such as “prior,” “noise parameter,” and “scores” are interpreted as clues associated with Bayesian statistics, signal processing, machine learning, and model evaluation.
- The combined terminology is used to suggest data scientist, machine learning engineer, statistician, researcher, or university lecturer as possible occupational profiles.
- The analysis connects the explanatory flow and spontaneous “Um...” to technical communication, teaching, presentations, or academic discussions.
After Modality-Grounded Reasoning Distillation
After distillation, the analysis combines technical vocabulary with vocal delivery to infer an academic or technical occupation. The strongest suggested profiles are university lecturer, data scientist, or researcher, although the inference remains probabilistic.
- The terminology “prior” and “noise parameter” is interpreted as evidence of discussion involving Bayesian models or statistical inference.
- The steady, explanatory delivery and natural pause are linked to teaching, technical sharing, or team discussions rather than scripted reading.
- The overall conclusion places the speaker in academic or technical work involving statistics, machine learning, teaching, research, or technical communication.
- The most likely occupational profile is a university professor or lecturer in statistics, computer science, or data science.
- Data scientist, machine learning engineer, researcher, and quantitative analyst remain additional possible roles supported by the technical subject matter.