Source-linked AI summary
StepAudio 2.5 Technical Report
Bin Lin, Bo Zhao, Boyong Wu, Chao Yan, Chen Wu, Cheng Yi, Chengyuan Yao, Daijiao Liu, Fei Tian, Feng Tian, Haiyang Sun, Haoyang Zhang, Jiangjie Zhen, Jinglan Gong, Jun Chen, Li Xie, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Runze Li, Shenghua Hu, Siyi Zhou, Wenwen Qu, Xiangyu Li, Xiangyu Tony Zhang, Xuerui Yang, Yang Yang, Yechang Huang, Yu Fu, Yuchu Luo, Yuxin Li, Yuxin Zhang, Zhengyan Sheng, Brian Li, Chang Zeng, Changlin Zhang, Chen Geng, Chenghao Dong, Chengli Feng, Dan Zhou, Danni Wan, Di Chen, Die Zhang, Dongqing Pang, Guanglong Yang, Guoqiang Hu, Huangxi Zhu, Jianzheng Gao, Jinghua Liang, Jinmei Wan, Junjie Yuan, Kang An, Lei Lei, Limin Zhong, Lun Cai, Mengqiang Ren, Min Xu, Mingliang Li, Mingxiao Li, Na Wang, Qiang Tong, Qiaoling Huang, Qingfu Du, Rui Wang, Shengchen Zhou, Shi Qiu, Shihao Peng, Shiliang Yang, Siqi Tu, Tianjiao Deng, Ting Xu, Tong Wang, WeiMing Niu, Wuxun Xie, Xianwei Zhang, Xianyu Feng, Xiaojia Liu, Xing Chen, Xiongbin Wu, Yan Wu, Yang Li, Yi Liu, Yifan Zhang, Yile Liu, Yongshen Long, Yu Luo, Yuanhao Ding, Yuhao Wang, Yuhe Yin, Yunfang Xu, Yuxiang Yang, Zhiguo Huang, Zhiyue Wu, Zichao Li, Zichao Zhou, Daxin Jiang, Future Li, Gang Yu, Xiangyu Zhang, Yibo Zhu
TL;DR
Existing unified audio-language models struggle to match specialized systems across ASR, TTS, and realtime interaction. StepAudio 2.5 uses task-tailored RLHF and specialized decoding to shape one shared backbone into three operational modes, achieving state-of-the-art results across all three capabilities.
Problem
Unified audio-language models have struggled to match specialized systems across ASR, TTS, and realtime spoken interaction, leaving this unification challenge open.
Method
StepAudio 2.5 uses a shared multimodal backbone with task-specific operational regimes, applying tailored RLHF and specialized decoding across ASR, TTS, and Realtime.
Results
StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and Realtime, outperforming leading unified models and specialized systems.
Takeaways & Limitations
A singular audio-language foundation can internalize distinct deployment objectives for speech understanding, generation, and live interaction.
Takeaways & Limitations
Prosodic and emotional TTS quality remain difficult to assess reliably because automated judges struggle with them and subjective MOS scores can be inconsistent.
Abstract
from arXiv · showhide
Unified audio-language modeling has emerged as a prominent trend in modern speech systems, promising to bring the reasoning capabilities of large language models to auditory tasks. However, existing unified foundations often struggle to match the depth of specialized systems across automatic speech recognition (ASR), text-to-speech synthesis (TTS), and realtime spoken interaction. Bridging this gap remains an open challenge. This report presents StepAudio 2.5, a unified audio-language foundation model that matches or exceeds specialized systems across all three capabilities. Rather than treating these tasks as architecturally distinct, we operate on the premise that once text and audio share a multimodal representational space, task specialization becomes a matter of operational regimes: data construction, optimization targets, and decoding constraints. Guided by this insight, we advance the post-training paradigm from standard supervised learning to task-tailored Reinforcement Learning from Human Feedback (RLHF), using it as the primary mechanism to define complex optimization targets. We leverage this RLHF-centric alignment, alongside specialized decoding, to shape a shared backbone into three distinct operational modes. Concretely, the ASR branch advances transcription efficiency via verifiable multi-token decoding; the TTS branch achieves controllable, expressive synthesis through preference-based RLHF and context-rich supervision; and the Realtime branch realizes low-latency, persona-consistent dialogue via generative reward modeling within an RLHF framework. On standard benchmarks, StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and Realtime, demonstrating that a singular audio-language foundation can successfully internalize the distinct deployment objectives of speech understanding, generation, and live interaction.
1 Introduction
StepAudio 2.5 presents a unified audio-language foundation model for ASR, TTS, and realtime spoken interaction. Its central thesis is that shared text-audio representations allow task differences to be handled through task-specific data, objectives, and decoding constraints rather than separate architectures.
- Motivation: Unified audio-language modeling preserves paralinguistic cues, emotional state, and conversational context that cascaded ASR–language-model–TTS pipelines can lose through textual intermediate representations.The motivation is end-to-end preservation of speech information across the processing pipeline.
- Challenge: A single model must simultaneously satisfy accurate, efficient long-form transcription, controllable and expressive synthesis, and low-latency, persona-consistent interaction.These deployment requirements make unified support for all three capabilities challenging.
- Central thesis: StepAudio 2.5 treats ASR, TTS, and realtime interaction as operational regimes shaped by task-specific data, objectives, and decoding constraints within a shared multimodal representation.The report frames the system as a singular audio-language foundation rather than loosely assembled parallel endpoints.
- Post-training strategy: Post-training is the primary alignment lever, moving beyond supervised fine-tuning to tailor the shared multimodal prior to each capability’s deployment objective.The introduction presents a unified alignment paradigm rather than separate engineering tracks.
- Capability-specific regimes: The ASR branch uses verifiable multi-token decoding, TTS combines semantic-to-audio alignment with context-rich preference-based RLHF, and Realtime uses progressive SFT followed by generative-reward RLHF.These mechanisms target transcription efficiency, controllable generation, persona and paralinguistic sensitivity, and low-latency dialogue.
2 Unified Foundation Architecture
StepAudio 2.5 uses a shared audio-language foundation built from a frozen audio encoder, adaptor, and LLM decoder operating in a unified text-and-audio sequence space. The shared stack supports ASR, TTS, and Realtime as distinct deployment regimes with different output spaces and objectives.
- Architecture: A frozen audio encoder feeds compact acoustic embeddings through a lightweight adaptor into a large text-LLM decoder that handles unified text and audio tokens.The encoder provides stable acoustic abstraction, while the decoder manages semantics, context, instruction following, and generation.
- Model specializations: The shared StepAudio 2.5 foundation organizes three specializations—ASR, TTS, and Realtime—within the same audio-language stack.These systems serve recognition, synthesis, and live spoken interaction under different deployment regimes.
- ASR: In ASR, audio embeddings condition the decoder to generate narrow, discrete transcript tokens anchored by the speech signal.Recognition uses the shared decoder with an output space constrained by the acoustic input.
- TTS: In TTS, text and control instructions condition the decoder to generate audio tokens or intermediate representations for faithful, natural, and expressive realization.Compared with ASR, TTS operates over a much richer output space where expressive fidelity is central.
- Realtime: Realtime couples audio understanding and response generation under strict turn-level latency constraints while maintaining conversational state, persona consistency, and contextual appropriateness.The architecture treats recognition, synthesis, and realtime dialogue as different ways to query the same multimodal memory through deployment-specific regimes.
3 Shared Data Engine and Foundation Pretraining
StepAudio 2.5 builds a shared audio-language foundation through automated multimodal data production and staged continual pretraining on text and audio. The resulting operational interface supports ASR, TTS, and Realtime as distinct uses of a common backbone.
- Shared data engine: The automated data pipeline jointly supports speech understanding, TTS, and speech dialogue by filtering low-quality segments and restructuring audio into semantically complete base samples.Raw audio is processed with sound event detection and voice activity detection; adjacent VAD segments are merged and re-segmented.
- Foundation pretraining: 2.2T tokens of text and audio data are used for continual pretraining after initialization from a textual MoE language model.The training curriculum follows a staged recipe rather than an unspecified scaling process.
- Foundation pretraining: 3B ASR tokens first align speech and text feature spaces through adaptor training while the audio encoder and language model remain frozen.This stage establishes the interface through which acoustic features are consumed by the text-native decoder.
- Foundation pretraining: 800B text tokens and 800B speech tokens form the main 16K-sequence multimodal mixture, spanning ASR, TTS, translation, interleaved continuation, and speech-to-speech conversation.The speech data treats audio as a general sequence modality rather than only as transcription input.
- Shared operational interface: The shared pretraining interface is reused across three operational directions: ASR maps audio evidence to text tokens, TTS maps text semantics to audio tokens, and Realtime couples listening, reasoning, and response generation under turn-level latency constraints.This interface is presented as the technical consequence of the pretraining recipe, rather than merely background context.
4 ASR Specialization
StepAudio 2.5 specializes ASR with an MTP-5 lookahead decoder whose future-token proposals are accepted only after autoregressive verification. This preserves recognition accuracy while achieving 0.0053 real-time factor and substantially improving decoding efficiency.
- Architecture: MTP-5 adds five future-token branches, producing a six-token proposal per forward step while retaining autoregressive verification of the accepted prefix.The main branch predicts x_t+1, while branch h predicts x_t+1+h for h ∈ {1, . . . , 5}.
- Training: Staged MTP training uses frozen-branch alignment followed by joint calibration, first optimizing only MTP blocks and then unfreezing the adapter and decoder.The peak learning rates are 2 × 10^-4 for branch alignment and 2 × 10^-5 for joint calibration.
- Recognition performance: 2.97% average CER on Chinese benchmarks and 3.68% average WER on English benchmarks establish strong recognition performance, including 0.71% on AISHELL-1 and 1.38% on LibriSpeech clean.The model also achieves 2.63% on FLEURS zh and 2.76% on VoxPopuli cleaned AA.
- Recognition stability: MTP-5 leaves recognition accuracy essentially unchanged, with average fluctuations within 0.06 absolute points relative to the SFT-only ASR model.Autoregressive verification ensures the final transcript is determined by the verified path.
- Decoding efficiency: 0.0053 RTF makes StepAudio 2.5 ASR faster than Qwen3-ASR-1.7B and substantially faster than VibeVoice-ASR, FunASR-Nano, and Doubao-ASR-2603 under the same serving setup.RTF is measured on 100 clips of 30 seconds each using a single NVIDIA H800 GPU with single concurrency.
- MTP acceptance behavior: Increasing MTP branches from three to five yields a 39% gain in average accepted length, whereas increasing from five to seven yields about 22% additional improvement.Acceptance rates decay by approximately 0.9 per branch from the second position, with high failure rates at the sixth and seventh positions driving diminishing returns.
5 TTS
StepAudio 2.5 TTS treats audio tokens as a language for pure next-token prediction, using the LLM backbone without an encoder-adapter module. Supervised fine-tuning and RLHF provide global, inline, and context-dependent expressive control, yielding a 67.6% overall win rate against three strong TTS baselines.
- Architecture: StepAudio 2.5 TTS eliminates the encoder-adapter module and models audio tokens as a new language through pure next-token prediction.This design relies solely on the LLM backbone and makes text–audio representation alignment the central challenge.
- Supervised fine-tuning: SFT uses zero-shot voice cloning to learn global speaker, style, and prosodic control, then supports fine-grained inline expression control.The training framework aligns natural-language instructions with corresponding audio token sequences.
- Reinforcement learning: RLHF aligns generated audio tokens with human preferences, improving interpretation of complex instructions and the naturalness, expressiveness, and perceptual quality of speech.A generative reward model compares policy responses with high-quality reference responses under the same prompt.
- Evaluation: Arena-style pairwise preference evaluation is adopted because CER, speaker similarity, LLM judges, and MOS have limitations for paralinguistic, prosodic, and emotional quality.The protocol standardizes evaluation and improves inter-rater consistency among human evaluators.
- Evaluation: 67.6% overall win rate was achieved in pairwise arena evaluations against MiniMax-2.8-HD, Elevenlabs-v3, and Gemini-3.1-Flash-TTS.The evaluation used 774 prompts and reported consistent gains across all comparisons.
6 Realtime Specialization
StepAudio 2.5 Realtime specializes the unchanged foundation architecture for low-latency spoken interaction through staged data and optimization, addressing coherence, persona, paralinguistic, and reward-sparsity challenges. It combines progressive SFT with RLHF and achieves leading results across five interactive evaluation suites.
- 6 Realtime Specialization: Realtime interaction requires conversational coherence, persona consistency, paralinguistic sensitivity, and optimization strategies for sparse conversational rewards.These challenges arise under stringent latency constraints and include maintaining dialogue state, handling adversarial inputs, interpreting non-verbal cues, and optimizing attributes without a single ground-truth target.
- 6 Realtime Specialization: The specialized training pipeline preserves the unchanged architecture while combining audio-centric mid-training, multi-stage SFT, and RLHF.The foundation provides audio-grounded perception and long-form reasoning, while staged optimization introduces dialogue-specific behaviors.
- 6.1.2 Supervised Fine-Tuning: Progressive SFT injects conversational alignment, persona and stylistic control, and paralinguistic sensitivity while dynamic rehearsal limits catastrophic forgetting and stylistic drift.Training uses instruction-rich spoken dialogue, persona-conditioned data, real spoken interactions, and interleaved general-purpose instruction and reasoning data.
- 6.1.3 Reinforcement Learning from Human Feedback: RLHF uses a PPO-style objective with KL regularization, generative rubric rewards, and preference comparisons over multi-turn dialogues and single-turn prompts.Multi-turn data supports consistency across exchanges, while single-turn prompts support longer-form reasoning and richer preference articulation.
- 6.2 Evaluation: +10.0 margin on the subjective human evaluation compared to the next-best system, while +16.6 margin on Step-SPQA accompanied strong Step-Dialogue-Understanding performance.StepAudio 2.5 Realtime consistently outperforms competitive baselines across all five suites, with evaluation combining subjective mobile-app sessions and objective API-based tests.
7 Conclusion
StepAudio 2.5 is presented as a unified audio-language foundation with three downstream specializations, built on a staged multimodal curriculum that progressively aligns speech and text, extends the token interface to audio, scales unified training, and refines long-context data. The report frames the ASR branch as a favorable application of verifiable multi-token decoding.
- Unified foundation: StepAudio 2.5 provides a unified audio-language foundation with three downstream specializations.The passage identifies the model as a shared foundation supporting specialized downstream branches.
- Backbone training: The shared backbone is learned through a staged multimodal curriculum aligning speech and text, extending the token interface to audio, scaling unified multimodal training, and refining high-quality long-context data.These are the four curriculum stages listed in the conclusion passage.
- ASR specialization: The ASR branch treats speech recognition as a particularly favorable application of verifiable multi-token decoding.This conclusion passage specifically characterizes the ASR branch through its decoding approach.
Authors
The paper lists its contributors in alphabetical order, distinguishing core contributors, contributors, and sponsors.
- Contributors are listed in alphabetical order.
- The core-contributor roster includes 34 named individuals.
- The sponsor roster names five individuals.