Step-Audio-R1 Technical Report

• Audio models often perform better with minimal or no reasoning [11] [12], while performance declines as reasoning chains lengthen across architectures, training methods, and model scales [11] [13] [14] [12] [13].Source: §1 · p. 2
• Case studies identify textual-surrogate reasoning as a cause of degradation: models reason from transcripts or captions instead of acoustic properties, reflecting text-derived COT initialization [17] [12] [13].For example, a model may attribute musical melancholy to lyrics rather than minor-key progressions and descending melodic contours.Source: §1 · p. 2
• Step-Audio-R1 benefits from test-time compute scaling, outperforming Gemini 2.5 Pro [14] and demonstrating capabilities competitive with Gemini 3 Pro across comprehensive audio benchmarks.Source: §1 · p. 2
• MGRD iteratively self-distills and refines audio-task reasoning, shifting from textual abstractions toward chains grounded in acoustic properties.The framework starts from text-based reasoning initialization and curates chains through iterative cycles.Source: §1 · p. 2

2 Model Overview

• The architecture, drawn from Step-Audio 2 [13], comprises an audio encoder, an audio adaptor, and an LLM decoder.Source: §2 · p. 3
• The frozen Qwen2 audio encoder [19] outputs at 25 Hz, while a 2× downsampling adaptor reduces its features to 12.5 Hz for the LLM.Source: §2 · p. 3
• The Qwen2.5 32B [20] decoder consumes adaptor audio features and produces text, generating reasoning content before its final reply.Source: §2 · p. 3
• MGRD iteratively refines initially semantic reasoning into “native audio think,” grounding it in acoustic nuances rather than only transcribed text.The report says this grounding leads to a more holistic and accurate final response.Source: §2 · p. 3

3.1 Data for Cold-Start

• Cold-start training combines SFT and RLVR over 5 million samples, with 1B tokens from text-only data and 4B tokens from audio-side data.Source: §3.1 · p. 4
• Audio training includes ASR, paralinguistic understanding, and AQTA dialogues; self-distilled AQTA CoT makes up 10% of the audio dataset.Source: §3.1 · p. 4
• Text data includes single- and multi-turn dialogues on knowledge QA, continuation, role-play, chat, and emotional conversation, plus math and code CoT.Source: §3.1 · p. 4
• Samples without native CoT receive an empty <think> tag to standardize the reasoning format.The format is <think>\n\n</think>\n{response}.Source: §3.1 · p. 4

3.2 Data for RL

• The RL phase uses 5,000 samples: 2,000 high-quality text-only math and code samples and 3,000 augmented speech-based QA samples.The speech-data augmentation methods are described in Section 4.2.Source: §3.2 · p. 4

4.1 Foundation Training: Reasoning Initialization and Format Alignment

• Foundation training combines task and conversational chain-of-thought supervision with direct-answer audio data, then applies RLVR [3] [21] and PPO [22] using binary verification without a KL penalty.Audio examples use empty reasoning markers to preserve the structural format without audio deliberation; outcome-based verification supports exploration while maintaining answer accuracy.Source: §4.1 · pp. 4–5

4.2 Modality-Grounded Reasoning Distillation

• The distillation selects questions requiring low-level acoustic analysis, filters sampled chains for acoustic grounding, coherence, and correctness, then jointly fine-tunes on audio and text reasoning.This selection targets features such as timbre, temporal patterns, pitch, and rhythm, while joint training preserves textual reasoning proficiency.Source: §4.2 · pp. 5–6
• Multimodal RL uses binary verification for text and format-plus-accuracy rewards for audio, weighting audio correctness 0.8 and reasoning generation 0.2.The reward design encourages reasoning generation while prioritizing correct audio answers.Source: §4.2 · p. 6
• Across iterations, reasoning shifts from textual surrogates to acoustic analysis, yielding a final model that attends to audio properties and enables test-time compute scaling.The framework aims to make reasoning attend to audio properties rather than rely on disconnected deliberations.Source: §4.2 · pp. 5–7

4.3 Implement Details

• RL questions were retained when 3–6 of 8 sampled responses were correct; PPO used 16 candidates, zero KL penalty, 0.2 clipping, and sequences up to 10,240 tokens.The pass@8 filter excludes overly simple and potentially harmful or nonsensical questions; PPO assigns binary rewards at the final token and uses discount factor and GAE lambda values of 1.0.Source: §4.3 · p. 7

5.1 Evaluation on Speech-to-Text Benchmarks

• 83.6% average score on speech-to-text benchmarks significantly exceeds Gemini 2.5 Pro and is slightly below Gemini 3 Pro.The comparison spans Big Bench Audio, Spoken MQA, MMSU, MMAU, Wild Speech, and Average Score.Source: Table 1 · p. 7, §5.1 · p. 8
• The speech-to-text evaluation compares Step-Audio-R1 with Gemini 2.5 Pro, Gemini 3 Pro, Step-Audio 2, and the base model across five audio benchmarks.The suite probes expert-level audio understanding and reasoning with MMSU [23] and MMAU [24], multi-step audio reasoning with Big Bench Audio1, verbal math reasoning with Spoken MQA [25], and conversational speech with Wild Speech [26].Source: §5.1 · pp. 7–8

5.2 Evaluation on Speech-to-Speech Benchmarks

• 96.1% speech reasoning performance and 0.92 s first-packet latency give Step-Audio-R1 Realtime strong reasoning while maintaining sub-second responsiveness.It outperforms GPT Realtime 0825 and Gemini 2.5 Flash Native Audio Dialog on speech reasoning performance; the model adapts the listen-while-thinking and think-while-speaking architecture [27] [28].Source: Table 2 · p. 8, §5.2 · p. 8

6.1 Extended Reasoning Benefits Audio: Evidence from Format Reward Ablation

• Without format rewards, reasoning shrank from approximately 3,000 to below 1,500 tokens by iteration 60, while rewarded models sustained 2,300–2,800-token chains.The report attributes this collapse to optimization favoring brief answers without explicit incentives for reasoning; it argues MGRD grounds extended deliberation in acoustic properties.Source: Figure 4 · p. 9, §6.1 · pp. 9–10
• 77.7 vs 76.5 MMAU accuracy was achieved with the think format reward, and full reasoning outperformed abbreviated or absent deliberation.The study also reports that format-rewarded training converged more stably than training without the reward.Source: §6.1 · p. 9

6.2 Strategic Data Selection: Quality Over Quantity in Audio RL

• 0.75–0.80 reward by iteration 20 and 2,300–2,800 tokens persisted with moderately difficult data, while failed-only training reached 0.45–0.70 and 1,800–2,000 tokens.Failed-only training also showed higher reward variance and collapsed after iteration 50.Source: Figure 5 · p. 10, §6.2 · pp. 10–11
• Moderately difficult examples provide both successful reasoning paths and failure cases for policy updates, whereas scaling unfiltered audio RL data to 200K examples brought no performance improvement.The report links consistently failed problems to ambiguity or insufficient audio information, which can leave the model without successful reasoning exemplars.Source: §6.2 · pp. 10–11

6.3 Self-Cognition Correction Through Iterative Refinement

• The correction pipeline filters self-distillation responses with an LLM judge, then applies DPO to preference pairs contrasting accurate audio-capability responses with text-only claims.The targeted query set covers sound identification, audio quality assessment, and acoustic property analysis.Source: §6.3 · pp. 11–12
• 0.02% self-cognition error after DPO with 8,000 preference pairs followed 6.76% for the base model and 2.63% after iterative self-distillation.The report interprets these errors as learned biases from training data rather than fundamental model limitations.Source: §6.3 · p. 12, Table 3 · p. 12

7 Conclusion

• Audio language models have historically failed to benefit from long reasoning, often performing worse as reasoning length increases.The conclusion identifies this as the challenge addressed by the work.Source: §7 · p. 12
• The report attributes this failure to “textual surrogate reasoning,” in which models rely on transcripts or captions instead of acoustic properties.This tendency shifts reasoning away from the audio itself.Source: §7 · p. 12
• Step-Audio-R1 outperforms Gemini 2.5 Pro and achieves performance comparable to Gemini 3 Pro across complex audio understanding and reasoning benchmarks.The report presents these evaluations as evidence that extended reasoning can become an asset for audio intelligence when connected to the correct input.Source: §7 · p. 12
• Step-Audio-R1 is presented as the first model to successfully unlock and benefit from deliberate thinking in the audio domain.The report frames this as a contribution to audio reasoning.Source: §7 · p. 12
• Modality-Grounded Reasoning Distillation (MGRD) iteratively shifts the model’s reasoning basis from text-based patterns to genuine acoustic analysis.MGRD is identified as the work’s core contribution.Source: §7 · p. 12
• The results support reasoning as a capability that works across modalities when connected to the correct input.The report suggests this could open paths toward truly multimodal systems.Source: §7 · p. 12

A.1 Audio Reasoning Examples

• The examples highlight reasoning about emotion, vocal characteristics, prosody, and complex environmental sound scenes, extending beyond simple speech recognition.The section frames these capabilities as processing rich acoustic information across diverse scenarios.Source: §A.1 · p. 16
• In response to a request for a professional musical appraisal, the model analyzes a song through its genre, instrumentation, vocal delivery, and production.Its reasoning characterizes the clip as pop/R&B, noting piano, synth pads, and a front-and-center vocal.Source: §A.1 · p. 16
• The illustrated analysis links vocal harmonies, vocal fry, off-beat drums, C# minor, and an approximately 72 BPM tempo to the song’s groove and mood.It also connects sparse instrumentation and breathy vocals with tension and vulnerability around the theme of dangerous attraction.Source: §A.1 · p. 16

I. Structural & Harmonic Genius (The “Maestro’s Blueprint”)

• The analysis identifies C# minor, an Emaj7 → C#m9 → F#m7b5 → B7#9 progression, and an AABA form that builds tension toward the chorus.It describes B7#9 as dissonant and unresolved, and says the clip reaches the bridge, where tension peaks before the chorus.Source: §I · p. 17

II. Production Mastery (The “Invisible Conductor”)

• The production analysis describes a doubled, subtly detuned lower-octave vocal, a 3.2-second sustained “touch,” and arpeggiated piano inversions.It says the vocal doubling adds thickness without muddiness and the piano texture mirrors “I just can’t get enough.”Source: §I · p. 17
• The arrangement pairs a half-time feel (72 BPM perceived as 36 BPM) and swung hi-hats with a 33Hz sub-bass drone.The analysis says the bass adds physical weight and the groove contributes to a “dangerously sexy” feel.Source: §I · p. 17

IV. Why This Isn’t “Just Pop Music” (The Maestro’s Verdict)

• The verdict calls #9 chords in mainstream pop rare and cites a 33-dB verse-to-chorus dynamic range as evidence of the track’s harmonic and dynamic ambition.It attributes the sense of danger to harmonic tension, including a B7#9 interval described as a minor 9th (C#-D#).Source: §I · p. 17

Background sound scene

• The separate speaker-scene answer selects D. Traffic street as the most plausible setting for clear speech over subtle, consistent urban ambience.The supplied reasoning favors it over train or subway, church, and concert; the passages acknowledge that the audio itself was not available to the reasoner.Source: §I · p. 19

A.2 Case Studies in Self-Recognition

• After MGRD training, the model acknowledges and uses its audio modality for in-depth reasoning, correcting its pre-training claim that it cannot process audio.The “Before” response reflects influence from the text-only backbone initialization.Source: §A.2 · p. 19

• The response suggests roles including data scientist, machine-learning engineer, statistician, researcher, or university lecturer, with scenarios such as model review or teaching.It notes that the audio cannot be heard directly and bases its inference on the snippet’s terminology and context.Source: Before Modality-Grounded Reasoning… · pp. 20–21
• “Prior,” “noise parameter,” and “scores” are linked respectively to Bayesian statistics, statistical or signal-processing models, and model evaluation.Source: Before Modality-Grounded Reasoning… · p. 20
• The terminology points toward data science, machine learning, statistics, or related quantitative research fields.Source: Before Modality-Grounded Reasoning… · p. 20

Professional Terminology

• References to a “wide prior” and “noise parameter” are treated as clues to Bayesian modeling or statistical inference.The response associates these terms with statistics, data science, machine learning, and signal processing.Source: After Modality-Grounded Reasoning… · pp. 21–22

Voice and Tone

• A steady, clear, explanatory tone and an opening “Um” are taken to suggest technical teaching or discussion rather than scripted speech.The response describes the natural pauses as consistent with an impromptu or semi-impromptu explanation.Source: After Modality-Grounded Reasoning… · p. 22

Possible Occupational Inferences

• A quantitative analyst is also proposed as a possibility because the response links Bayesian methods and noise parameters to financial modeling.It frames this as an additional occupational possibility, alongside academic and technical roles.Source: After Modality-Grounded Reasoning… · p. 22
• The speaker is inferred to work in statistics or machine learning, most plausibly as a university instructor, data scientist, or researcher.The response also characterizes the clip as resembling an academic lecture, technical training, or an internal team discussion.Source: After Modality-Grounded Reasoning… · p. 22