PaperlayerView exact revision on arXiv

Source-linked AI summary

Step-Audio 2 Technical Report

Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang, Jin Yang, Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu, Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu, Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, Yuxiang Zhang, Zidong Yang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang, Yibo Zhu

arXiv:2507.16632v3cs.CLcs.SDeess.AS

TL;DR

Step-Audio 2 targets limitations in expressive, reliable end-to-end speech interaction, including weak use of paralinguistic information, hallucination, and restricted voice control. It combines latent audio encoding, discrete audio-token generation, reasoning-centric reinforcement learning, retrieval, and external tools. The model achieves state-of-the-art results across ASR, audio understanding, speech translation, and speech conversation, outperforming open-source and commercial solutions.

  • Problem

    Existing LALMs often neglect paralinguistic information, produce only text, and suffer hallucination and limited timbre and speaking-style choices.

  • Method

    Step-Audio 2 combines latent audio encoding, interleaved text-and-audio-token generation, reasoning-centric reinforcement learning, retrieval-augmented generation, and external web and audio search tools.

  • Results

    Step-Audio 2 achieves state-of-the-art performance across ASR, audio understanding, speech translation, and general speech conversation, outperforming open-source and commercial solutions.

  • Takeaways & Limitations

    The system provides end-to-end speech interaction with responsiveness to speaking styles and emotions and can use retrieved knowledge and speech to produce more expressive responses.

Abstract

from arXiv · show

This paper presents Step-Audio 2, an end-to-end multi-modal large language model designed for industry-strength audio understanding and speech conversation. By integrating a latent audio encoder and reasoning-centric reinforcement learning (RL), Step-Audio 2 achieves promising performance in automatic speech recognition (ASR) and audio understanding. To facilitate genuine end-to-end speech conversation, Step-Audio 2 incorporates the generation of discrete audio tokens into language modeling, significantly enhancing its responsiveness to paralinguistic information such as speaking styles and emotions. To effectively leverage the rich textual and acoustic knowledge in real-world data, Step-Audio 2 integrates retrieval-augmented generation (RAG) and is able to call external tools such as web search to mitigate hallucination and audio search to switch timbres. Trained on millions of hours of speech and audio data, Step-Audio 2 delivers intelligence and expressiveness across diverse conversational scenarios. Evaluation results demonstrate that Step-Audio 2 achieves state-of-the-art performance on various audio understanding and conversational benchmarks compared to other open-source and commercial solutions. Please visit https://github.com/stepfun-ai/Step-Audio2 for more information.

1 Introduction

Existing large audio language models often miss paralinguistic information in expressive speech interaction and face hallucination and limited voice choices. Step-Audio 2 addresses these gaps with end-to-end audio processing, reasoning-oriented training, retrieval, and external tools, achieving state-of-the-art performance across audio tasks.

  • Existing LALMs may align speech semantics to text while neglecting paralinguistic information important for intentional understanding.
  • Text-only outputs prevent models from turning understood paralinguistic information into coherent and expressive speech responses.
  • Hallucination and limited timbres and speaking styles remain challenges because existing LALMs lack real-world textual and acoustic knowledge.
  • Step-Audio 2 directly processes raw audio and outputs discrete text and audio tokens while modeling semantic, paralinguistic, and non-vocal information.
  • Retrieval-augmented generation plus web and audio search supports more reliable responses and voice-style or timbre switching.
  • Step-Audio 2 achieves state-of-the-art performance across ASR, audio understanding, speech-to-speech translation, and speech-to-speech conversation.The model is trained on 680 billion text tokens and 8 million hours of real and synthesized audio.

2 Related Work

Related work spans unified audio-language models, codec-based speech generation, direct speech-to-speech translation, and cascaded speech pipelines. These approaches motivate end-to-end modeling while highlighting latency and modular-mismatch concerns.

  • Recent audio-language models pair speech encoders with trainable adapters that project audio features into an LLM-compatible textual embedding space.
  • Codec language models generate discrete speech representations autoregressively or with masked language modeling before waveform synthesis.
  • Direct speech-to-speech translation bypasses intermediate text by generating target spectrograms or discrete acoustic units from source speech.
  • Cascaded systems separate ASR, LLM, and TTS components, but their modular pipeline incurs high latency and mismatches between modules.

3 Methodology

Step-Audio 2 combines an audio encoder, adaptor, LLM decoder, and detokenizer with retrieval tools and staged training. Its training uses multimodal pretraining, supervised fine-tuning, reasoning-centric datasets, and reinforcement learning for audio interaction.

  • Architecture: Step-Audio 2 consists of an audio encoder, audio adaptor, LLM decoder, and audio detokenizer that together support end-to-end audio perception and generation.
  • Architecture: The frozen audio encoder feeds latent features through a downsampling adaptor, while the decoder emits interleaved discrete text and audio tokens.The adaptor reduces the encoder output frame rate from 25 Hz to 12.5 Hz.
  • Tools: Audio search retrieves speech from a large voice library, allowing the model to mimic speaking styles or switch timbres from voice instructions.
  • Architecture: The audio detokenizer converts output audio tokens into waveforms through Flow Matching and a HiFi-GAN vocoder.
  • Training: The model is initialized from a textual LLM and continually pretrained on textual and audio data before multimodal task expansion and cooldown.The initial continual pretraining uses 1.356T tokens over 21 days.
  • Training: Supervised fine-tuning covers complex acoustic understanding, paralinguistic speech conversation, multilingual ASR, TTS, translation, and tool invocation.
  • Training: Reasoning-centric datasets and two-stage PPO reinforcement learning are used to improve audio understanding and speech interaction.

4 Evaluation

Step-Audio 2 is evaluated across ASR, paralinguistic understanding, general audio comprehension, translation, tool calling, and speech-to-speech conversation. It achieves strong results across these benchmarks, including leading or competitive performance against open-source and commercial baselines.

  • Automatic speech recognition: Step-Audio 2 achieves 3.14% average WER on English and 3.08% average CER on Chinese test sets, outperforming existing open-source and commercial ASR models.It also performs comparably to leading systems on Arabic, Japanese, and Cantonese recognition and achieves the lowest average CER on selected accented-Mandarin and dialect tests.
  • Paralinguistic information understanding: StepEval-Audio-Paralinguistic evaluates understanding across 11 dimensions, and Step-Audio 2 achieves an average accuracy of 83.09.The benchmark covers attributes including gender, age, timbre, emotion, pitch, rhythm, speaking speed, speaking style, and vocal activity, plus sound-related tasks.
  • Audio understanding: Step-Audio 2 achieves the highest MMAU average score of 78.0, with the best results on sound and speech tracks and results on par with the best on music.The comparison includes Audio Flamingo 3, Gemini 2.5 Pro, GPT-4o Audio, Kimi-Audio, Omni-R1, Qwen2.5-Omni, and Step-Audio-AQAA.
  • Speech translation: Using BLEU, Step-Audio 2 achieves the highest average score on both CoVoST 2 speech-to-text translation and CVSS speech-to-speech translation.The evaluation measures bidirectional Chinese-English translation performance.
  • Tool calling: Step-Audio 2 matches textual LLMs in tool-calling accuracy with speech input and significantly outperforms Qwen3-32B on the audio search tool.StepEval-Audio-Toolcall evaluates tool invocation, selection, and parameter extraction in Chinese speech conversations.
  • Speech-to-speech conversation: In Chinese speech-to-speech conversation, Step-Audio 2 achieves the highest average scores of 83.32 on the basic track and 68.25 on the pro track.In English speech-to-speech conversation, it is slightly outperformed by GPT-4o Audio but exceeds the other approaches.

5 Conclusion

Step-Audio 2 is presented as an end-to-end audio language model for speech and audio understanding and intelligent speech interaction. It combines latent audio encoding, reinforcement learning, discrete audio-token generation, and external tools, with state-of-the-art results across several tasks.

  • Step-Audio 2 is designed for enterprise speech and audio understanding and intelligent speech interaction.
  • The model uses a latent audio encoder and reinforcement learning to enhance speech and audio comprehension.
  • Generating discrete audio tokens within language modeling enables end-to-end speech interaction and improves responsiveness to paralinguistic information.
  • Step-Audio 2 supports multi-modal retrieval-augmented generation through external web and audio search tools.
  • Trained on 8 million hours of speech and audio, Step-Audio 2 achieves state-of-the-art performance across ASR, audio understanding, speech translation, and general speech conversation.

B Introduction and evaluation results of Step-Audio 2 mini

Step-Audio 2 mini is an open-source, developer-friendly variant designed for fair comparison with similarly sized open-source models. It uses a different audio encoder initialization, supports only web search, and achieves results on par with Step-Audio 2.

  • Step-Audio 2 mini is released as a special open-source version of Step-Audio 2.
  • Step-Audio 2 mini uses the Qwen2-Audio encoder, is initialized with Qwen2.5-7B, and is trained on the same dataset as Step-Audio 2.
  • Step-Audio 2 mini is limited to using the web search tool.
  • Step-Audio 2 mini has a parameter count intended for fair comparisons with Qwen-Omni and Kimi-Audio.
  • Evaluation results show Step-Audio 2 mini performs on par with Step-Audio 2 and exceeds most open-source and commercial models such as GPT-4o Audio.

B.1 Automatic speech recognition

The ASR comparison table evaluates Step-Audio 2 and related systems across multiple test sets using character and word error rates. Results may vary depending on the inference backend used.

  • Table 7 compares six ASR systems across multiple test sets using character error rates for several languages and word error rates for English and Arabian.
  • The compared systems are Doubao LLM ASR, GPT-4o Transcribe, Kimi-Audio, Qwen-Omni, Step-Audio 2, and Step-Audio 2 mini.
  • Evaluation results obtained with the vLLM backend may differ from results obtained with the transformers backend.

B.2 Paralinguistic information understanding

Table 8 compares Step-Audio 2 and other open-source and commercial systems on the StepEval-Audio-Paralinguistic benchmark.

  • Table 8 evaluates GPT-4o Audio, Kimi-Audio, Qwen-Omni, Step-Audio-AQAA, Step-Audio 2, and Step-Audio 2 mini.
  • The table focuses on the StepEval-Audio-Paralinguistic benchmark.
  • The comparison includes both open-source and commercial audio systems.

B.3 Audio understanding

Table 9 compares Step-Audio 2 and Step-Audio 2 mini with multiple open-source and commercial systems on MMAU.

  • The evaluation compares nine audio models on MMAU.
  • The comparison includes Audio Flamingo 3, Gemini 2.5 Pro, GPT-4o Audio, Kimi-Audio, Omni-R1, and Qwen2.5-Omni.
  • Step-Audio-AQAA, Step-Audio 2, and Step-Audio 2 mini are also evaluated on MMAU.

B.4 Speech translation

Table 10 compares BLEU scores for speech-to-text and speech-to-speech translation across six audio systems.

  • The evaluation uses BLEU scores for speech-to-text and speech-to-speech translation.
  • The comparison includes GPT-4o Audio, Qwen2.5-Omni, Qwen-Omni, and Step-Audio-AQAA.
  • Step-Audio 2 and Step-Audio 2 mini are included as additional systems in the translation comparison.

B.5 Speech-to-speech conversation

Table 11 compares six systems on URO-Bench, covering understanding, reasoning, and oral conversation.

  • The evaluation benchmark is URO-Bench.
  • URO denotes understanding, reasoning, and oral conversation, respectively.
  • The comparison includes GPT-4o Audio, Kimi-Audio, Qwen-Omni, and Step-Audio-AQAA.
  • Step-Audio 2 and Step-Audio 2 mini are also compared on this benchmark.
Loading 2507.16632v3…