Source-linked AI summary
Step-Audio-R1.5 Technical Report
Yuxin Zhang, Xiangyu Tony Zhang, Daijiao Liu, Fei Tian, Yayue Deng, Jun Chen, Qingjian Lin, Haoyang Zhang, Yuxin Li, Jinglan Gong, Yechang Huang, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Gang Yu, Xiangyu Zhang, Daxin Jiang
TL;DR
Audio reasoning models can score well on discrete benchmarks while missing the prosodic, emotional, and conversational qualities that shape real-world interaction. Step-Audio-R1.5 integrates RLHF with audio reasoning, improving multi-turn conversational quality while preserving analytical rigor.
Problem
RLVR-based audio reasoning reduces temporally extended audio to discrete labels, leaving prosodic naturalness, emotional continuity, and conversational coherence insufficiently evaluated.
Method
Step-Audio-R1.5 integrates RLHF using holistic human preference judgments and a rubric-guided reward model alongside structured Chain-of-Thought generation.
Results
5.47 points improve the average score over Step-Audio-R1 (72.50), with substantial gains on complex multi-turn and perceptual benchmarks.
Takeaways & Limitations
Step-Audio-R1.5 improves multi-turn conversational quality while preserving analytical rigor, indicating that human feedback can address the verifiable reward trap.
Abstract
from arXiv · showhide
Recent advancements in large audio language models have extended Chain-of-Thought (CoT) reasoning into the auditory domain, enabling models to tackle increasingly complex acoustic and spoken tasks. To elicit and sustain these extended reasoning chains, the prevailing paradigm -- driven by the success of text-based reasoning models -- overwhelmingly relies on Reinforcement Learning with Verified Rewards (RLVR). However, as models are strictly optimized to distill rich, continuous auditory contexts into isolated, verifiable text labels, a fundamental question arises: are we fostering true audio intelligence, or merely reducing a continuous sensory medium into a discrete puzzle? We identify this as the "verifiable reward trap." While RLVR yields remarkable scores on standardized objective benchmarks, it systematically degrades the real-world conversational feel of audio models. By prioritizing isolated correctness over acoustic nuance, RLVR reduces dynamic interactions to mechanical "answering machines," severely compromising prosodic naturalness, emotional continuity, and user immersion, particularly in long-turn dialogues. To bridge the gap between mechanical objective verification and genuine sensory empathy, we introduce Step-Audio-R1.5, marking a paradigm shift toward Reinforcement Learning from Human Feedback (RLHF) in audio reasoning. Comprehensive evaluations demonstrate that Step-Audio-R1.5 not only maintains robust analytical reasoning but profoundly transforms the interactive experience, redefining the boundaries of deeply immersive long-turn spoken dialogue.
1 Introduction
The section identifies a verifiable reward trap in audio reasoning: RLVR optimizes isolated textual correctness while neglecting prosodic, emotional, and conversational qualities. Step-Audio-R1.5 addresses this limitation by adding human-preference supervision, preserving analytical reasoning while improving multi-turn spoken interaction.
- Problem: Audio RLVR benchmarks reduce temporally extended inputs to discrete labels, limiting optimization to isolated answer correctness.This structural limitation leaves prosodic naturalness, emotional continuity, and conversational coherence outside the reward signal.
- Problem: The verifiable reward trap makes prolonged RLVR-trained models more accurate yet increasingly terse, mechanical, emotionally flat, and experientially hollow in multi-turn dialogue.The model can degenerate into a technically accurate but literal “answering machine.”
- Approach: Step-Audio-R1.5 complements RLVR with RLHF trained on holistic human preferences over end-to-end interactions.Its unified supervisory signal combines correctness, fluency, and emotional resonance rather than relying solely on binary correctness checks.
- Evaluation: Step-Audio-R1.5 preserves RLVR-developed analytical reasoning while substantially improving multi-turn interaction quality.On AudioMultiChallenge, it evaluates Inference Memory, Instruction Retention, Self Coherence, and Voice Editing under naturalistic multi-turn conditions, rivaling or exceeding Gemini-2.5-Flash in key dimensions.
2 Architecture
Step-Audio-R1.5 uses a streamlined three-component architecture for extended audio reasoning: an audio encoder, an audio adaptor, and an LLM decoder. Its frozen acoustic front end connects continuous audio to textual Chain-of-Thought generation through temporal downsampling and explicitly separated reasoning traces.
- Architecture: The architecture comprises an audio encoder, an audio adaptor, and an LLM decoder tailored for extended audio-based reasoning.This design builds on the structural foundation of Step-Audio-R1.
- Audio Front End: The Qwen2 audio encoder operates at 25 Hz and remains strictly frozen throughout training to preserve auditory perception.It was pretrained on diverse speech and audio understanding tasks.
- Audio Front End: The audio adaptor applies temporal downsampling at a rate of 2, compressing latent representations from 25 Hz to 12.5 Hz for connection with textual space.This adaptor bridges the continuous acoustic modality and discrete textual space.
- Reasoning Decoder: The Qwen2.5 32B LLM decoder ingests downsampled audio features and generates purely textual outputs.The decoder is initialized from Qwen2.5 32B.
- Reasoning Decoder: For Chain-of-Thought reasoning, the decoder first synthesizes explicit intermediate reasoning traces before autoregressively generating the final reply.This structurally decouples internal analysis from the external response.
3 Training Method
Step-Audio-R1.5 uses audio-centric mid-training with auxiliary text reasoning, followed by cold-start SFT and unified RLHF. The alignment framework combines rubric-guided evaluation with pairwise preferences and jointly optimizes heterogeneous interaction objectives to reduce forgetting.
- Mid-training: Audio-centric mid-training strengthens audio understanding, audio-grounded reasoning, and deliberative capability before post-training alignment.Its supervised objective combines audio-grounded reasoning data with auxiliary text-only reasoning data.
- Mid-training: The unified supervised objective trains on audio-grounded samples containing audio, textual context, reasoning traces, and responses, alongside text-only samples.Diverse audio-centric supervision provides broad perceptual coverage and robust reasoning over acoustically grounded contexts.
- Cold-start SFT: Cold-start SFT initializes interaction-oriented behavior because audio understanding alone does not ensure natural, coherent, instruction-sensitive multi-turn dialogue.The stage emphasizes dialogue continuity, instruction following, and related interaction behaviors using instruction-rich conversational data.
- Unified RLHF: RLHF addresses heterogeneous spoken-interaction targets by combining rubric-conditioned evaluation for explicit constraints with pairwise preference judgments for weakly specifiable qualities.The generated reward model supports both supervision modes within one framework.
- Unified RLHF: Jointly optimizing the two supervision forms provides a more stable alignment route because decoupled training can cause later optimization in one regime to degrade behaviors learned in the other.The policy is optimized with a PPO-style objective using generated-reward advantages and reference-policy regularization.
- Unified RLHF: Relative reward compares each policy response with a reference response in the same multi-turn context, rather than assigning an absolute quality score.This formulation suits spoken dialogue because important interaction qualities are difficult to calibrate with a single absolute score.
4 Evaluation
The evaluation combines speech-to-text benchmarks for acoustic reasoning with specialized tests of multi-turn dialogue, fine-grained perception, paralinguistic understanding, and public audio reasoning. Step-Audio-R1.5 achieves competitive results, including a 77.97 average and a 5.47-point improvement over Step-Audio-R1.
- Evaluation Setup: The evaluation uses speech-to-text benchmarks to isolate acoustic understanding and reasoning through text-based responses.This format enables direct comparison with state-of-the-art large language models.
- Evaluation Benchmarks: AudioMultiChallenge evaluates interruptions, hesitations, repairs, long-context dialogue, instruction retention, self-coherence, and voice editing across four dimensions.It measures Inference Memory, Instruction Retention, Self Coherence, and Voice Editing.
- Evaluation Benchmarks: Step-Caption evaluates fine-grained description of vocal characteristics and emotional state across 905 curated single- and multi-speaker audio samples annotated on 16 dimensions.The samples come from YouTube and Bilibili and primarily cover Chinese and English.
- Evaluation Benchmarks: The suite also includes Step-DU, converted AQTA Step-SPQA tasks, and public benchmarks for expert audio reasoning, multi-step logic, and spoken mathematical reasoning.Step-DU uses 87 samples and asks about vocal characteristics such as age, gender, speaking rate, or rhythm.
- Comparison Protocol: All baselines are evaluated through official APIs in a unified framework, covering Gemini 3 Flash, Gemini 3 Pro, qwen3.5-omni-flash, and qwen3.5-omni-plus.The framework is intended to ensure direct comparability under identical conditions.
- Results: 77.97 average score places Step-Audio-R1.5 second overall, while its 41.15 Audio MC score trails only Gemini-family models.The model has 32B parameters and improves 5.47 points over Step-Audio-R1’s 72.50 average, driven primarily by complex multi-turn and long-context tasks.
5 Conclusion
The conclusion attributes early audio reasoning models’ mechanical, emotionally flat responses to the verifiable reward trap rather than to Chain-of-Thought itself. It argues that RLVR’s focus on isolated semantic correctness blinds models to the multidimensional nuances of genuine human interaction.
- Early audio reasoning models’ mechanical, emotionally flat responses are attributed to the verifiable reward trap, not an inherent Chain-of-Thought limitation.
- RLVR optimization for isolated semantic correctness structurally blinds models to the multidimensional nuances of genuine human interaction.
6 Contributors
The section lists the paper’s core contributors and additional contributors. The core contributor list includes eight named individuals, while the additional list includes thirteen.
- Core Contributors: Core Contributors: Yuxin Zhang, Xiangyu Tony Zhang, Daijiao Liu, Fei Tian, Yayue Deng, Jun Chen, and Qingjian Lin are listed.Fei Tian is marked with ∗,† in the source passage.
- Contributors: Contributors: Haoyang Zhang, Yuxin Li, Jinglan Gong, Yechang Huang, Liang Zhao, Chengyuan Yao, Hexin Liu, Eng Siong Chng, Xuerui Yang, Gang Yu, Xiangyu Zhang, and Daxin Jiang are listed.The source passage contains thirteen names, including affiliations indicated by superscript numerals.