Source-linked AI summary
StepAudio 3 Gen Technical Report
Bin Lin, Bo Zhao, Boyang Wang, Boyang Zhang, Boyong Wu, Chao Yan, Chen Geng, Chen Wu, Cheng Yi, Chengli Feng, Chenglin Zhu, DanNi Wan, Daxin Jiang, Dongqing Pang, Fei Tian, Feng Tian, Future Li, Gang Yu, Guanglong Yang, Jia Peng, Jiahao Song, Jiamin Fan, Jiangjie Zhen, Jianzheng Gao, Jun Chen, Li Xie, Lifang Zhang, Lingli Ji, Liying Shi, Lun Cai, Min Xu, Na Wang, Peilin Li, Peng Yang, Pengfei Tan, Qingjian Lin, Ruijie Xiong, Runze Li, Shenghua Hu, Shi Qiu, Siqi Tu, Siyi Zhou, Tianjiao Deng, Wanying Lu, Weiming Niu, Wen Sun, WenWen Qu, Xiangyu Zhang, Xianwei Zhang, XiaoSu Su, Xing Chen, Xinyu Liu, Xuerui Yang, Yang Li, Yang Yang, Yechang Huang, Yibo Zhu, Yifan Zhang, Yiyang Xu, Yu Fu, Yu Luo, Yu Zhou, Yumang Wang, Yunzhou Ju, Yuxiang Yang, Zekai Liu, Zengwei Yao, Zhenwei Mou, Zheqi Dai, Zhiyue Wu, Zichao Zhou
TL;DR
General-purpose audio generation requires one framework that can cover diverse domains without sacrificing language-model capabilities. StepAudio 3 Gen uses shared discrete RVQ audio tokens, an RVQ Adaptor, and interference-aware progressive pretraining; it achieves state-of-the-art TTS and voice-design performance while retaining strong generation across other audio types.
Problem
Existing audio systems specialize in speech, environmental sound, music, or singing with incompatible representations, conditioning formats, and generation pipelines.
Method
StepAudio 3 Gen combines a shared 12.5-Hz, 16-codebook RVQ representation, an RVQ Adaptor, and interference-aware progressive pretraining in a discrete autoregressive LLM.
Results
StepAudio 3 Gen achieves state-of-the-art results on TTS and voice design while demonstrating strong generation across speech, vocals, music, sound effects, and mixed audio.
Takeaways & Limitations
A shared semantic–acoustic representation and discrete autoregressive modeling can support diverse audio generation tasks within one LLM-based architecture.
Abstract
from arXiv · showhide
We introduce StepAudio 3 Gen, a general-purpose audio generation model that supports zero-shot text-to-speech (TTS), voice design, vocal generation, sound effects, music, vibe speech, and mixtures of multiple audio types within a unified framework. At its core, StepAudio 3 Gen is a discrete autoregressive generator that models audio directly over residual vector quantization (RVQ) tokens, departing from the diffusion Transformer-based continuous generation paradigm prevalent in recent general audio models. Its StepAudio Tokenizer represents general audio at 12.5 Hz in a shared $16 \times 2048$ residual code space, jointly quantizing semantic and waveform-level acoustic features so that each code layer preserves both types of information. For generation, the backbone predicts the first codebook along the time axis using autoregressive modeling, while a lightweight causal Transformer completes the remaining fifteen codebooks along the codebook axis. Our study further identifies three key design principles: (1) interference-aware progressive pretraining for acquiring audio capabilities while preserving the textual abilities of the large language model, (2) RVQ Adaptor for effectively incorporating multi-codebook acoustic representations, and (3) discrete autoregressive modeling over a shared representation across general audio domains. With progressive pretraining, multi-task instruction training, and supervised fine-tuning, StepAudio 3 Gen achieves state-of-the-art performance on both TTS and voice design, while retaining strong generation capabilities across speech, vocals, sound effects, and music. Audio samples are available at https://stepaudiollm.github.io/step-audio-3-gen/.
1 Introduction
StepAudio 3 Gen unifies speech, environmental sound, music, singing, and mixed audio generation through discrete language-model-style modeling. Its design addresses representation and optimization interference by combining a shared RVQ representation, an RVQ Adaptor, and progressive pretraining.
- Motivation: Specialized audio systems use incompatible representations, conditioning formats, and pipelines, motivating a unified model for applications combining multiple audio types.Prior systems separately optimize speech, environmental sound, music, or singing objectives.
- Representation and factorization: High-fidelity RVQ requires multiple codebooks, so time–depth modeling predicts temporal structure separately from within-frame codebook structure.This avoids prohibitively long flattened sequences while preserving acoustic detail beyond a coarse semantic token.
- Progressive pretraining: Four-stage interference-aware pretraining uses frozen-backbone alignment, 50% text replay, and temporary residual-predictor gradient detachment to add audio capabilities while retaining textual abilities.Generation is introduced after audio understanding, with end-to-end gradients restored only after the residual predictor converges.
- RVQ Adaptor: The RVQ Adaptor maps summed multi-codebook audio embeddings into the pretrained LLM input space before adding them to ordinary token embeddings.This lets the model consume complete acoustic representations without inserting a separate sequence encoder or replacing the shared backbone.
- Unified audio generation: A shared 12.5-Hz, 16-codebook RVQ representation supports speech, singing, music, sound effects, and mixed audio in one discrete autoregressive generator.The time–depth design separates temporal prediction from within-frame codebook prediction without a continuous acoustic renderer.
2 Model Architecture
The model combines a shared semantic–acoustic tokenizer with an LLM that predicts audio in a single text–audio stream. The main LLM predicts the first RVQ code over time, while a lightweight causal predictor completes the remaining codes within each frame.
- 2.1 StepAudio Tokenizer: StepAudio Tokenizer discretizes general audio at 12.5 Hz into a shared multi-codebook space and reconstructs 24 kHz waveforms.Semantic and acoustic features are jointly quantized so every code layer carries both semantic content and acoustic detail.
- 2.2 LLM Backbone: The LLM treats codebook 0 as 2,048 added vocabulary tokens, allowing text and audio to share one autoregressive stream and softmax.The first audio code is mapped to a contiguous range after the text vocabulary.
- 2.2 LLM Backbone: The main LM predicts c0 along time, while a causal RVQ Code Predictor uses the hidden state and c0 to autoregressively generate c1 through c15.All sixteen codes together form the RVQ frame used for waveform decoding.
- 2.2 LLM Backbone: The generation factorization is exact for each frame because the backbone hidden state already encodes prior-frame codebooks, but predictor outputs do not feed back into the backbone.Joint end-to-end training is therefore delayed until the residual predictor has converged.
3 Data
Training data combines broad text and audio pretraining with task-oriented post-training across speech, vocals, music, environmental sound, and related understanding tasks. Specialized corpora and filtering procedures target naturalness, controllability, coverage, and paired-data quality.
- 3 Data: The data pipeline uses broad-coverage pretraining followed by task-oriented post-training to establish and refine unified audio understanding and generation.Post-training corpora are curated for generation quality, naturalness, and controllability across audio domains.
- 3 Data: Pretraining mixes text with speech, singing, music, environmental sound, TTS, speech-to-speech translation, ASR, translation, captioning, and paralinguistic tasks.The mixture is designed to introduce general-audio generation while retaining the text corpus.
- Text to Speech: TTS data emphasizes natural conversational speech, including tone, rhythm, spontaneous vocal behavior, recording context, and acoustic conditions.Additional TTS-oriented captioning describes properties such as noise level and speaking characteristics.
- Speech to Vocal: Speech-to-vocal training constructs pseudo-paired speech–singing data through TTS voice cloning and singing voice conversion to match speaker timbre.Samples are retained using both audio-quality and timbre-similarity criteria.
- Music and Sound: Music and sound-effect corpora combine generated or recorded audio with diverse textual, event, source, acoustic, scene, and temporal annotations.Sound mixtures are filtered, normalized, aligned, and gain-adjusted to preserve foreground–background structure.
4 Training
StepAudio 3 Gen trains a unified audio language model through progressive pretraining, supervised fine-tuning, and reinforcement learning. The curriculum integrates audio capabilities while preserving textual competence by controlling backbone updates and retaining text replay.
- LLM Pretraining: A four-stage pretraining curriculum progressively adds audio alignment, understanding, generation, and joint acoustic conditioning while preserving the pretrained LLM’s textual ability.Stage 1 freezes the backbone; later stages retain text replay, detach the residual-code predictor during generation training, and restore gradients only during a low-learning-rate cool-down.
- LLM Pretraining: Stage 1 trains only the audio embedding and adaptor against textual supervision with a frozen backbone, making textual-ability drift strictly zero.The adaptor is initialized as an identity map, while newly added audio-code rows receive a reduced learning-rate multiplier.
- LLM Pretraining: Stage 3 detaches the residual code predictor from backbone hidden states because its fifteen-codebook loss can otherwise overwhelm the token-level objective and reshape learned representations.The predictor learns acoustic detail independently while the backbone remains driven by the token-level objective; detachment is removed only after predictor convergence.
- LLM Pretraining: Stage 4 restores end-to-end gradients during a low-learning-rate, long-context cool-down and reduces the speech-loss weight from λ = 1.0 to λ = 0.1.This delayed joint optimization lets the backbone acquire useful acoustic conditioning after the residual predictor has converged while limiting gradient-induced changes.
- Post-training: Post-training combines full-parameter supervised fine-tuning with GRPO across TTS, voice design, vocals, music, sound generation, and mixtures.GRPO samples 16 candidates per instruction, uses instruction-consistency and transcript-recognition rewards, and discards groups with reward standard deviation below 0.02.
5 Evaluation
The evaluation tests RVQ Adaptor integration, preservation of textual capabilities, TTS human-likeness, and voice-design instruction following. StepAudio 3 Gen consistently outperforms comparison systems on audio-language, text, TTS, and voice-design evaluations.
- 5.1 RVQ Adaptor: The RVQ Adaptor system consistently outperforms its counterpart across ASR, audio reasoning, speech translation, and speech question-answering benchmarks.The comparison uses CER, WER, BLEU, and accuracy across AISHELL-1, LibriSpeech, MMAU, CoVoST, and SpeechMMLU.
- 5.2 Interference-aware progressive pretraining: The interference-aware progressive pretraining strategy consistently outperforms a three-stage baseline across text, reasoning, and code-generation benchmarks.The evaluated metrics include accuracy, exact match, and Pass@1 on FinEval, C-Eval, MMLU, CMMLU, MATH, GSM8K, BBH, and HumanEval.
- 5.3 TTS: 1755.33 Elo and an 82.0% aggregate win rate show StepAudio 3 Gen leading five commercial TTS systems in perceived human-likeness.Its head-to-head win rates range from 73.0% to 90.0%, with a majority against every opponent.
- 5.3 TTS: StepAudio 3 Gen achieves the best style-consistency performance among evaluated systems, averaging 85.2% in Chinese and 77.7% in English.The comparison covers APS, DSD, and RP conditions on the full InstructTTSEval benchmark.
- 5.4 Voice Design: 1668.5 Elo and a 75.5% aggregate win rate place StepAudio 3 Gen first in blind human preference evaluation for voice design.Its individual-opponent win rates range from 65.3% to 85.7% across Chinese and English comparisons.
6 Extensions
StepAudio 3 Gen extends its shared discrete audio framework beyond speech to vocals, long-form music, sound effects, acoustic scenes, and vibe speech. These capabilities support natural-language control of vocal attributes, musical structure, temporal sound arrangement, and multi-speaker scenes.
- 6 Extensions: The shared RVQ token space supports speech, voice design, vocals, music, sound effects, and complex audio scenes without task-specific tokenizers or diffusion-based renderers.The same discrete autoregressive framework is used across the evaluated and extended audio capabilities.
- 6 Extensions: Text-to-vocal and speech-to-vocal generation control voice identity, lyrics, musical key, pitch characteristics, and singing style through prompts or reference speech.Text prompts specify the target voice and vocal attributes, while speech input provides a reference voice for speech-to-vocal generation.
- 6 Extensions: The model generates instrumental music exceeding 60 seconds while maintaining overall musical structure and stylistic consistency.Prompts can specify genre, mood, instrumentation, rhythm, production style, and intended use.
- 6 Extensions: Sound generation covers isolated effects and layered acoustic scenes with controllable event ordering, timing, and co-occurrence.Prompts can describe multiple sources, ambience, interactions, and temporal arrangement.
- 6 Extensions: Vibe speech combines spoken content, speaking style, atmosphere, background sound, and contextual cues in multi-speaker acoustic scenes.The model supports coherent conversational environments involving multiple speakers and sound sources.
7 Conclusion
StepAudio 3 Gen presents unified general-purpose audio generation through a discrete autoregressive framework and shared semantic–acoustic representation. It achieves state-of-the-art TTS and voice-design results while supporting vocals, music, sound, and vibe speech.
- 7 Conclusion: State-of-the-art results on TTS and voice design accompany solid generation across vocals, music, sound, and vibe speech.The reported capabilities span multiple audio domains within one model.
- 7 Conclusion: A shared semantic–acoustic representation supports diverse audio generation tasks within a single LLM-based architecture.The conclusion identifies this shared representation as the basis for unifying domains.
- 7 Conclusion: Progressive audio introduction helps mitigate interference with the pretrained LLM during capability acquisition.The report presents progressive training as a practical lesson for preserving the pretrained model while adding audio abilities.
- 7 Conclusion: A lightweight RVQ Adaptor effectively integrates multi-codebook acoustic representations for discrete autoregressive generation.This is one of the report’s three identified design principles.
- 7 Conclusion: Discrete autoregressive modeling offers a practical and scalable alternative to continuous-generation approaches for unified audio models.The conclusion frames this as the broader implication of the reported findings.