Source-linked AI summary
Step-Audio-EditX Technical Report
Chao Yan, Boyong Wu, Peng Yang, Pengfei Tan, Guoqiang Hu, Li Xie, Yuxin Zhang, Xiangyu, Zhang, Fei Tian, Xuerui Yang, Xiangyu Zhang, Daxin Jiang, Shuchang Zhou, Gang Yu
TL;DR
Zero-shot TTS can reproduce reference-voice attributes but offers limited independent control over emotion and style. Step-Audio-EditX uses large-margin data for iterative editing and zero-shot synthesis, and the report describes improved editing accuracy and control across repeated edits.
Problem
Zero-shot TTS attributes are derived from reference audio, and prepended emotion or style instructions can fail to control the cloned voice effectively.
Source: §1 · p. 2Method
Step-Audio-EditX post-trains an LLM-based audio model with large-margin data for iterative emotion, speaking-style, and paralinguistic editing alongside zero-shot TTS.
Source: §1 · p. 2Results
Emotion and speaking-style accuracy increase after the first edit and improve further over successive iterations, while Step-Audio-EditX shows better emotional accuracy in zero-shot cloning than the other two models.
Source: §5.2.1 · pp. 8–9Takeaways & Limitations
The framework supports iterative editing and can extend to tasks including dialect, accent, vocal editing, and imitation.
Source: §1 · p. 2, §7 · p. 11The dual-codebook tokenizer retains emotional, prosodic, and other non-linguistic information, so its disentanglement is suboptimal.
Source: §2.2 · p. 3
Abstract
from arXiv · showhide
We present Step-Audio-EditX, the first open-source LLM-based audio model excelling at expressive and iterative audio editing encompassing emotion, speaking style, and paralinguistics alongside robust zero-shot text-to-speech (TTS) capabilities. Our core innovation lies in leveraging only large-margin synthetic data, which circumvents the need for embedding-based priors or auxiliary modules. This large-margin learning approach enables both iterative control and high expressivity across voices, and represents a fundamental pivot from the conventional focus on representation-level disentanglement. Evaluation results demonstrate that Step-Audio-EditX surpasses both MiniMax-2.6-hd and Doubao-Seed-TTS-2.0 in emotion editing and other fine-grained control tasks.
1 Introduction
Zero-shot TTS can clone a reference voice, but its emotion, style, accent, and timbre remain tied to the reference, limiting independent control. Step-Audio-EditX uses large-margin data to support iterative editing across speech attributes and voices, alongside zero-shot TTS.
1 Introduction
- Attributes in zero-shot TTS are derived from the reference audio, and prepended style or emotion instructions can fail to guide the cloned voice effectively.Instruction-based control often performs well for in-domain speakers but faces challenges disentangling speech attributes.Source: §1 · p. 2
- Large-margin training pairs preserve linguistic content while varying one or a few speech attributes, enabling attribute editing and iterative increases or decreases in target intensity.The approach also extends to speed adjustment, denoising, and silence trimming.Source: §1 · p. 2
- Step-Audio-EditX is an open-source LLM-based audio model for expressive, iterative emotion, speaking-style, and paralinguistic editing, with zero-shot TTS capabilities.The reported approach uses large-margin data alone for emotion and speaking-style control, without extra presentation modeling or adapter modules.Source: §1 · p. 2
2 Architecture
Step-Audio-EditX combines a dual-codebook tokenizer, an audio LLM, and a flow-matching decoder in a unified framework for zero-shot TTS and editing. Its retained audio representations carry emotion and prosody information, providing a test of post-training rather than tokenizer disentanglement.
2.1 Overview
- The unified system uses a dual-codebook tokenizer, an audio LLM to generate token sequences, and a flow-matching decoder to produce waveforms, supporting zero-shot TTS and diverse editing tasks.The report adds zero-shot TTS and paralinguistic editing to the prior model and reduces model size from 130B to 3B.Source: §2.1 · p. 3
2.2 Audio Tokenizer
- The dual-codebook tokenizer retains emotional, prosodic, and other non-linguistic information, making it suitable for testing the post-training strategy despite suboptimal disentanglement.It uses parallel linguistic and semantic tokenizers interleaved at a 2:3 ratio.Source: §2.2 · p. 3
2.3 Audio LLM
- The 3B audio LLM is initialized from a text LLM, trained on a 1:1 blend of text and audio dual-codebook tokens, and generates audio tokens as its sole output.Source: §2.3 · p. 3
2.4 Audio Decoder
- The decoder's flow-matching module generates Mel spectrograms from audio tokens and reference-audio and speaker-embedding conditions, then BigVGANv2 converts them into waveforms.The flow-matching module uses a DiT backbone trained on 200,000 hours of speech, improving Mel reconstruction, pronunciation accuracy, and timbre similarity.Source: §2.4 · pp. 3–4
3 Data
The report focuses on post-training data for zero-shot TTS and editing, including large-margin emotion and speaking-style pairs and a semi-synthetic strategy for paralinguistic editing. It also constructs human- and LLM-judged preference pairs for reinforcement learning.
3 Data; 3.1 SFT Data
- Post-training data covers zero-shot TTS and emotion, speaking-style, and paralinguistic editing, with large-margin data targeting emotion and speaking-style tasks.Source: §3 · p. 4, §3.1 · p. 4
3.1.1 Zero-shot Text to Speech
- The zero-shot TTS dataset is professionally annotated, primarily Chinese and English, with a small amount of Cantonese and Sichuanese data and about 60,000 speakers.It captures vocal variation within speakers and across a broad speaker population.Source: §3.1.1 · p. 4
3.1.2 Emotion and Speaking Style Editing
- Emotion and speaking-style examples use same-speaker neutral and target-attribute clips to create zero-shot cloning triplets, with large-margin synthetic comparisons requiring only one prompt segment per attribute.Voice actors recorded an approximately 10-second clip for each emotion and style combination; a human-annotated scoring model assessed audio pairs, with a universal minimum margin score of 6.Source: §3.1.2 · pp. 4–5
3.1.3 Paralinguistic Editing
- Paralinguistic editing uses NVSpeech annotations to form quadruplets pairing tag-removed cloned audio with original transcripts and audio; its time-domain task uses intrinsic margins, so selection needs no margin-scoring model.The report says a small quadruplet set is sufficient to elicit this capability.Source: §3.1.3 · p. 5
3.2 Reinforcement Learning Data
- The reinforcement-learning preference data combines human ratings of correctness, prosody, and naturalness with LLM-as-a-judge scores for emotion and speaking-style editing.Human-selected pairs required a score margin greater than 3 on a 5-point scale; LLM-judged pairs retained margins greater than 8 on a 1–10 scale for reward-model and PPO training.Source: §3.2 · p. 5
4 Training
Training aligns the model for zero-shot TTS, editing, and human preferences through supervised fine-tuning followed by proximal policy optimization. The report attributes improved stability and instruction-following expressiveness to reinforcement learning, particularly for edits with substantial source–target divergence.
4 Training
- The post-training pipeline uses supervised fine-tuning followed by proximal policy optimization to align outputs with zero-shot TTS, editing tasks, and human preferences.Source: §4 · p. 5
4.1 Supervised Fine-tuning
- Supervised fine-tuning uses chat prompts for TTS and editing, encoding the reference waveform into dual-codebook tokens for TTS and pairing source audio with an editing command for editing.The model is fine-tuned for one epoch, with learning rate from 1 × 10^-5 to 1 × 10^-6.Source: §4.1 · p. 6
4.2 Reinforcement Learning
- Reinforcement learning improves zero-shot TTS stability and editing-instruction capability and expressiveness, especially when source and target emotions or styles diverge substantially.Examples include generating sad speech from a happy prompt and converting loud speech into a whisper.Source: §4.2 · p. 6
- The reward model starts from a 3B SFT model and learns directly from large-margin dual-codebook token pairs using Bradley–Terry loss, avoiding waveform decoding during reward computation.Source: §4.2 · p. 6
- PPO trains on the most challenging SFT prompts after warming the critic for 80 steps before the actor, using a 0.2 clip threshold and a 0.05 KL penalty coefficient.Source: §4.2 · p. 6
5 Evaluation
Step-Audio-EditX is evaluated on a benchmark covering emotion, speaking style, and paralinguistics, with results reporting improved editing accuracy and generalization across closed-source TTS voices. Editing gains increase across iterations, while a single iteration also improves emotion and paralinguistic editing outcomes.
5.1 Evaluation Benchmark
- The benchmark evaluates emotion, speaking style, and paralinguistics using zero-shot-cloned audio scored by Gemini-2.5-Pro.Emotion and style are classified against predefined categories; paralinguistic editing receives an overall average on a 1–3 scale.Source: §5.1 · p. 7
5.2.1 Emotion and Speaking Style Editing Results
- Emotion and speaking-style accuracy rises after the initial edit and improves further across successive iterations, including when prompt audio is held constant.The prompt-fixed ablation reports continued improvement as editing iterations increase.Source: §5.2.1 · p. 8
- Across closed-source TTS voice models, one Step-Audio-EditX editing round significantly improves emotion and style accuracy, with further gains over two more rounds.The evaluation applies three editing iterations to outputs from the systems’ built-in voices.Source: §5.2.1 · p. 9
- One emotional-editing iteration outperforms closed-source models’ native emotion controls, and Step-Audio-EditX has higher zero-shot-cloning emotion accuracy than the other two models.Emotional accuracy improves for all audio samples after one editing iteration.Source: §5.2.1 · p. 9
5.2.2 Paralinguistic Results
- A single editing iteration yields a significant gain from adding paralinguistic tags; on closed-source voices, edited reproduction is comparable to native paralinguistic synthesis.The generalization comparison uses one female and one male built-in voice per closed-source model.Source: §5.2.2 · p. 10
5.2 Evaluation Results
- Across emotion, speaking style, and paralinguistic editing, large-margin learning with reinforcement learning enhancement delivers high accuracy and strong generalization.The evaluation also reports superior editing accuracy and scalability on audio generated by various closed-source TTS systems.Source: §5.2 · p. 7, §5.2.2 · p. 10
6 Extensions
Large-margin paired data supports rapid acquisition of editing capabilities through supervised fine-tuning, with reinforcement learning available to improve challenging cases. The section extends this approach to speech-rate control, denoising, and silence trimming.
6 Extensions
- Large-margin paired samples let the model rapidly acquire target editing capabilities through SFT, with reinforcement learning available to improve challenging cases.The section identifies speed editing and denoising and silence trimming as practical extensions.Source: §6 · p. 11
6.1 Speed Editing
- Speed editing uses SoX-generated faster and slower versions of a speaker’s audio, and SFT alone suffices despite substantial token-length differences.Training triplets pair text and source audio with faster and slower versions.Source: §6.1 · p. 11
6.2 Denoising and Silence Trimming
- Prompt noise and silence can be mistaken for speaker characteristics and reproduced, so generative editing targets both prompt and synthesized audio.Imitating those features may be desirable in some use cases but undesirable in others.Source: §6.2 · p. 11
- Denoising pairs noisy, reverberant audio with its clean reference, while silence trimming pairs source audio with speech segments extracted using Silero-VAD [32].The denoising augmentation uses additive noise and reverberation simulation; trimming concatenates speech segments by VAD timestamps.Source: §6.2 · p. 11
7 Conclusion
Step-Audio-EditX combines large-margin training and reinforcement learning for zero-shot TTS and expressive editing. The conclusion frames editing as conditional regeneration or transfer, with mask-based editing for partial changes that preserve other content.
7 Conclusion
- Step-Audio-EditX uses large-margin training and reinforcement learning for zero-shot TTS, iterative emotion and speaking-style editing, and paralinguistic editing.The framework is described as extendable to dialect, accent, vocal editing, and imitation.Source: §7 · p. 11
- The authors distinguish their editing from conventional editing, describing it as conditional regeneration or transfer.For partial changes that preserve remaining content, they describe a mask-based method that reconstructs paired data so only selected edited tokens differ.Source: §7 · pp. 11–12