PaperlayerView exact revision on arXiv

Source-linked AI summary

StepAudio 2.5 Technical Report

Bin Lin, Bo Zhao, Boyong Wu, Chao Yan, Chen Wu, Cheng Yi, Chengyuan Yao, Daijiao Liu, Fei Tian, Feng Tian, Haiyang Sun, Haoyang Zhang, Jiangjie Zhen, Jinglan Gong, Jun Chen, Li Xie, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Runze Li, Shenghua Hu, Siyi Zhou, Wenwen Qu, Xiangyu Li, Xiangyu Tony Zhang, Xuerui Yang, Yang Yang, Yechang Huang, Yu Fu, Yuchu Luo, Yuxin Li, Yuxin Zhang, Zhengyan Sheng, Brian Li, Chang Zeng, Changlin Zhang, Chen Geng, Chenghao Dong, Chengli Feng, Dan Zhou, Danni Wan, Di Chen, Die Zhang, Dongqing Pang, Guanglong Yang, Guoqiang Hu, Huangxi Zhu, Jianzheng Gao, Jinghua Liang, Jinmei Wan, Junjie Yuan, Kang An, Lei Lei, Limin Zhong, Lun Cai, Mengqiang Ren, Min Xu, Mingliang Li, Mingxiao Li, Na Wang, Qiang Tong, Qiaoling Huang, Qingfu Du, Rui Wang, Shengchen Zhou, Shi Qiu, Shihao Peng, Shiliang Yang, Siqi Tu, Tianjiao Deng, Ting Xu, Tong Wang, WeiMing Niu, Wuxun Xie, Xianwei Zhang, Xianyu Feng, Xiaojia Liu, Xing Chen, Xiongbin Wu, Yan Wu, Yang Li, Yi Liu, Yifan Zhang, Yile Liu, Yongshen Long, Yu Luo, Yuanhao Ding, Yuhao Wang, Yuhe Yin, Yunfang Xu, Yuxiang Yang, Zhiguo Huang, Zhiyue Wu, Zichao Li, Zichao Zhou, Daxin Jiang, Future Li, Gang Yu, Xiangyu Zhang, Yibo Zhu

arXiv:2605.23463v1eess.AS

TL;DR

Unified audio-language models still face a gap with specialized systems across ASR, TTS, and realtime interaction. StepAudio 2.5 uses task-specific operational regimes on a shared foundation and achieves state-of-the-art results across all three capabilities on standard benchmarks.

  • Problem

    Unified audio-language models still need to narrow the gap with specialized speech systems across ASR, TTS, and realtime interaction.

    Source: §1 · pp. 2–3

  • Method

    StepAudio 2.5 uses a shared audio-language foundation with task-specific operational regimes, including specialized decoding and RLHF-based training.

    Source: §1 · pp. 2–3, §7 · p. 16

  • Results

    Across standard benchmarks, StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and Realtime, outperforming leading unified models and specialized task systems.

    Source: §1 · p. 3

  • Takeaways & Limitations

    A single audio-language foundation can support downstream specializations for speech recognition, synthesis, and low-latency spoken dialogue.

    Source: §7 · p. 16

    Evaluation of prosodic quality and emotional expression remains challenging: LLM judges are unreliable, while subjective MOS requires trained annotators and can yield inconsistent scores.

    Source: §5.3 · p. 11

Abstract

from arXiv · show

Unified audio-language modeling has emerged as a prominent trend in modern speech systems, promising to bring the reasoning capabilities of large language models to auditory tasks. However, existing unified foundations often struggle to match the depth of specialized systems across automatic speech recognition (ASR), text-to-speech synthesis (TTS), and realtime spoken interaction. Bridging this gap remains an open challenge. This report presents StepAudio 2.5, a unified audio-language foundation model that matches or exceeds specialized systems across all three capabilities. Rather than treating these tasks as architecturally distinct, we operate on the premise that once text and audio share a multimodal representational space, task specialization becomes a matter of operational regimes: data construction, optimization targets, and decoding constraints. Guided by this insight, we advance the post-training paradigm from standard supervised learning to task-tailored Reinforcement Learning from Human Feedback (RLHF), using it as the primary mechanism to define complex optimization targets. We leverage this RLHF-centric alignment, alongside specialized decoding, to shape a shared backbone into three distinct operational modes. Concretely, the ASR branch advances transcription efficiency via verifiable multi-token decoding; the TTS branch achieves controllable, expressive synthesis through preference-based RLHF and context-rich supervision; and the Realtime branch realizes low-latency, persona-consistent dialogue via generative reward modeling within an RLHF framework. On standard benchmarks, StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and Realtime, demonstrating that a singular audio-language foundation can successfully internalize the distinct deployment objectives of speech understanding, generation, and live interaction.

1 Introduction

StepAudio 2.5 treats ASR, TTS, and realtime dialogue as operational regimes of one shared audio-language foundation, shaped by task-specific post-training and decoding. The report states that it achieves state-of-the-art results across the three capabilities on standard benchmarks.

Motivation

  • Unified audio-language foundations can preserve speech information lost in cascaded ASR-language-model-TTS pipelines, but meeting the distinct accuracy, expressivity, latency, and persona demands together remains challenging.The report identifies these deployment objectives as poorly aligned and notes that existing unified systems can lag specialized systems on some capabilities.Source: §1 · p. 2

Approach

  • StepAudio 2.5 treats task differences as operational regimes of a shared text-audio representational space, shaped by data, objectives, decoding, and RLHF-centered post-training.Capability-specific SFT and specialized decoding complement RLHF, which the report positions as the primary means of capturing human preferences and paralinguistic behavior.Source: §1 · p. 2

ASR

  • For ASR, a verifiable multi-token decoding head exploits acoustic determinism to emit multiple tokens per step and improve the quality-efficiency frontier.The shared decoder is coupled to this head.Source: §1 · pp. 2–3

Results

  • StepAudio 2.5 achieves state-of-the-art results across ASR, TTS, and realtime benchmarks, outperforming leading unified models and specialized systems.The report presents this as the result of its unified foundation and capability-specific alignment and decoding.Source: §1 · p. 3

TTS

  • For TTS, semantic-to-audio alignment supports controllable generation, combining context-rich supervision with human-preference-driven RLHF.The report describes preference-based RLHF as part of adapting the shared backbone for synthesis.Source: §1 · p. 3

Realtime

  • For realtime dialogue, progressive SFT develops persona and paralinguistic sensitivity, followed by RLHF using a generative reward model and explicit interaction rubrics.This branch targets low-latency spoken dialogue.Source: §1 · p. 3

2 Unified Foundation Architecture

StepAudio 2.5 uses a shared audio-language backbone in which an audio encoder and adaptor feed a text-LLM decoder that handles semantics, context, instructions, and generation. ASR, TTS, and Realtime specialize this shared stack through different output spaces and deployment regimes.

2.1 Shared Backbone

  • A frozen audio encoder and lightweight adaptor feed acoustic embeddings to a text-LLM decoder, which handles semantics, context, instruction following, and generation.The decoder operates over a unified sequence space containing conventional text tokens and audio tokens; the shared stack supports ASR, TTS, and Realtime specializations.Source: §2.1 · p. 3

2.2 Task Specialization as Directional Inference

  • ASR, TTS, and Realtime use the shared multimodal foundation in distinct directional inference regimes.ASR maps audio embeddings to transcript tokens; TTS uses text and control instructions to generate audio representations; Realtime combines audio understanding and response generation under turn-level latency constraints while maintaining conversational state, persona consistency, and contextual appropriateness. Recognition, synthesis, and dialogue route supervision through different output spaces and deployment regimes.Source: §2.2 · pp. 3–4, Figure 1 · p. 4

3 Shared Data Engine and Foundation Pretraining

StepAudio 2.5 combines an automated, quality-aware data pipeline with staged multimodal pretraining to create a shared operational interface between audio and text. That interface supports ASR, TTS, and realtime dialogue on one backbone.

3.1 A Common Data Production Pipeline

  • An automated pipeline filters, segments, annotates, validates, and grades audio data to support speech understanding, TTS, and dialogue training.It uses SED and VAD, audio- and text-level annotations, transcript cross-validation, semantic assessment, and metadata-based quality categories for stage-specific sampling.Source: §3.1 · p. 4

3.2 Progressive Foundation Training

  • Initialized from a textual MoE LLM, StepAudio 2.5 undergoes continual pretraining on 2.2T tokens of text and audio data.The first stage uses 3B ASR tokens to align speech and text feature spaces, training only the adaptor while freezing the audio encoder and LLM.Source: §3.2 · pp. 4–5
  • Unified multimodal pretraining uses 800B text tokens and 800B speech tokens, with speech spanning transcription, synthesis, translation, interleaved continuation, and speech-to-speech conversation.After alignment, speech tokens are added to the vocabulary and training begins at a 16K sequence length.Source: §3.2 · p. 5
  • A 128B-token warmup stabilizes the speech vocabulary and adapts MoE experts before the main multimodal stage anneals router settings and auxiliary loss.During warmup, adaptor, embedding, and output layers use larger learning rates, while the MoE router uses a smaller rate; the main stage returns layer-specific rates to the base rate.Source: §3.2 · p. 5
  • The 600B-token cooldown increases sequence length to 32K and emphasizes higher-quality multimodal supervision and longer-context refinement.It adds Audio Caption and Instruct TTS data to the data types used in the main training stage.Source: §3.2 · p. 5
  • Pretraining establishes an operational audio-text interface reused for ASR, TTS, and realtime dialogue on a shared backbone.The model learns to map audio evidence into text tokens, text-side semantics into audio tokens, and listening, reasoning, and response generation under turn-level latency constraints.Source: §3.2 · p. 5

4 ASR Specialization

StepAudio 2.5 ASR adds verified MTP-5 lookahead proposals to an encoder-adapter-decoder recognizer and trains the branches in stages. Evaluation reports recognition results across Chinese, English, and long-form benchmarks, faster decoding, and little accuracy change after MTP is added.

4 ASR Specialization

  • MTP-5 adds five future-token branches to the ASR encoder-adapter-decoder, proposing six tokens per forward step and accepting only the verified prefix.When a proposed token disagrees with normal decoding, later proposals are rejected and autoregressive decoding resumes; thus MTP serves strictly as an acceleration primitive.Source: §4 · p. 5

4.1 Training Pipeline

  • MTP training proceeds through frozen-branch alignment followed by joint calibration of the adapter and LLM decoder.The preceding SFT stage trains a reliable autoregressive recognizer on short- and long-form data while freezing the audio encoder; the two MTP stages first optimize only the branches, then unfreeze the adapter and decoder to reduce mismatch.Source: §4.1 · p. 6

4.2 Data

  • ASR data combines approximately 100K hours of short-form audio with a 50K-hour long-form pseudo-labeled dataset.Short-form data covers Mandarin, English, code-switching, domain terminology, and challenging acoustic conditions. Long-form recordings are transcribed by three systems, fused with token-level ROVER voting, filtered by disagreement rate, concatenated, and refined for punctuation, normalization, and session-level consistency.Source: §4.2 · pp. 7–8

4.3 Evaluation

  • Adding MTP-5 leaves recognition accuracy essentially unchanged, with average fluctuations within 0.06 absolute points across Chinese, English, and long-form benchmarks.The report attributes this stability to staged training and autoregressive verification, which keeps the final transcript determined by the verified path.Source: §4.3 · p. 8
  • 2.97% Chinese average CER, 3.68% English average WER, and 3.70% average long-form error rate are reported for StepAudio 2.5 ASR.The report also highlights 0.71% CER on AISHELL-1, 1.38% WER on LibriSpeech clean, and 2.76% WER on VoxPopuli cleaned AA.Source: §4.3 · p. 8, Table 1 · p. 9
  • 0.0053 RTF is reported for StepAudio 2.5 ASR, which is faster than Qwen3-ASR-1.7B despite its larger decoder and substantially faster than the other compared systems.RTF was measured on 100 clips of 30 seconds each; the report attributes the speedup to emitting several verified transcript tokens in many decoding steps.Source: §4.3 · p. 8, Table 2 · p. 9
  • MTP-5 is selected as the efficiency-complexity trade-off: MTP-3 to MTP-5 raises average accepted length by 39%, while MTP-5 to MTP-7 adds about 22%.Acceptance declines by approximately 0.9 per branch from the second position; failures at positions six and seven can trigger KV-cache rollbacks.Source: §4.3 · pp. 8–9, Table 3 · p. 9

5 TTS

StepAudio 2.5 TTS treats audio tokens as a language for next-token prediction, then uses supervised fine-tuning and preference-based reinforcement learning to align speech with instructions. Its arena evaluation reports a 67.6% overall win rate against three strong TTS baselines.

5 TTS

  • Audio tokens are modeled as a new language for pure next-token prediction, eliminating the encoder-adapter and making text-audio alignment the central challenge.The TTS branch relies solely on the LLM backbone.Source: §5 · pp. 9–10

5.1 Training Pipeline

  • After pretraining establishes shared text-audio representations, SFT aligns instructions with audio, and human-feedback RL further improves instruction following, naturalness, expressiveness, and perceptual quality.The RL stage uses a generative reward model to score policy outputs against high-quality reference responses.Source: §5.1 · p. 10

5.2 SFT Data

  • Synthesized speech supports global control, while recorded speech supports joint global and inline control, with annotations derived from transcripts, context, and quantized acoustic features.The two annotation outputs describe utterance-level style and local expressive behavior.Source: §5.2 · pp. 10–11

5.3 Evaluation

  • Because common objective, judge-based, and MOS evaluations have limitations, the authors use arena-style pairwise human preferences and aggregated win rates.The protocol screens evaluators, randomizes model-pair selection and ordering, collects reasons for judgments, and checks consistency.Source: §5.3 · pp. 11–12
  • 67.6% overall win rate was achieved in pairwise evaluations against three strong TTS baselines using 774 prompts.Figure 4 presents the arena win rates.Source: Figure 4 · p. 12, §5.3 · p. 12

6 Realtime Specialization

StepAudio 2.5 Realtime keeps the foundation architecture unchanged and uses specialized data and staged optimization to address the coherence, persona, paralinguistic, and reward-sparsity challenges of low-latency multi-turn interaction.

6.1 Training Pipeline

  • The conversational training pipeline has three stages: audio-centric mid-training, progressive supervised fine-tuning, and RLHF.The pipeline is designed to instill conversational capabilities while preserving perceptual and reasoning abilities.Source: §6.1 · p. 13

6.1.1 Audio-Centric Mid-Training

  • Audio-centric mid-training provides the inherited baseline of audio-grounded perception and long-form reasoning before dialogue-specific behaviors are introduced.Source: §6.1.1 · p. 13

6.1.2 Progressive Supervised Fine-Tuning (SFT)

  • Progressive SFT trains conversational alignment, persona and stylistic control, and paralinguistic sensitivity, while rehearsal interleaves dialogue data with general instruction and reasoning tasks.The curriculum handles multi-turn continuity and spoken-language artifacts, supports compositional generalization across persona traits, and adjusts tone and pacing to vocal cues; rehearsal guards against forgetting and stylistic drift.Source: §6.1.2 · pp. 13–14

6.1.3 RLHF with Generated Rewards

  • RLHF uses a PPO-style objective with KL regularization and a generative reward model combining preference comparisons with rubric scores for instruction-sensitive qualities.Multi-turn dialogues train consistency across exchanges, while single-turn prompts support longer-form reasoning and richer preference articulation.Source: §6.1.3 · p. 14

6.2 Data

  • Realtime SFT data combines spoken multi-turn dialogue, persona-conditioned scenarios, and paralinguistic-cue dialogues, alongside a general-capability mixture and consistency and deduplication checks.The persona stream expands more than 10,000 human-reviewed native personas into a million-scale matrix; the paralinguistic stream labels cues such as hesitation, laughter, sighs, and pacing changes.Source: §6.2 · p. 14

6.3 Evaluation

  • +10.0 on subjective human evaluation versus the next-best system and +16.6 on Step-SPQA accompany outperformance across all five suites.The authors attribute the results to persona and naturalness conditioning, and report strong Step-Dialogue-Understanding performance alongside the Step-SPQA margin.Source: Figure 5 · p. 15, §6.3 · p. 15

7 Conclusion

StepAudio 2.5 is a unified audio-language foundation whose shared backbone supports ASR, TTS, and realtime dialogue through distinct optimization and deployment regimes. Its capabilities are built on a staged multimodal curriculum and task-specific approaches to recognition, synthesis, and interaction.

7 Conclusion

  • StepAudio 2.5 uses a shared backbone, learned through a staged multimodal curriculum, to support three downstream speech specializations.The curriculum aligns speech and text, extends the token interface to audio, scales unified multimodal training, and refines the model with high-quality long-context data.Source: §7 · p. 16
  • ASR uses verifiable multi-token decoding, while TTS combines semantic-to-audio alignment, context-rich supervision, and reinforcement learning.The report characterizes these as task-specific regimes built on the same foundation.Source: §7 · p. 16
  • Realtime dialogue extends the shared foundation to low-latency interaction with persona stability and paralinguistic sensitivity.Together, the three capabilities are presented as distinct regimes of a shared foundation rather than isolated speech endpoints.Source: §7 · p. 16

Related papers

Loading 2605.23463v1…