PaperlayerView exact revision on arXiv

Source-linked AI summary

StepAudio 3 Music Technical Report

Chengli Feng, Zhiyue Wu, Jiahao Song, Zheqi Dai, Boyang Wang, Ruibin Yuan, Junming Gong, Wenxiao Zhao, Jing Guo, Gang Yu, Xiangyu Zhang, Xuerui Yang, Chao Yan

arXiv:2609.16034v1eess.AScs.SD

TL;DR

Long-form music generation requires coordinating global musical decisions with detailed acoustic realization, while controllable creation also needs structured, actionable conditions beyond global text prompts. The system combines a generation-oriented music tokenizer, Mixture-of-Experts autoregressive generation with optional ABC-CoT planning, and a flow-matching DiT renderer. StepAudio 3 Music achieves the highest reported AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and highest MuQ-MuLan similarity among evaluated systems, while remaining competitive on SongBench.

  • Problem

    Long-form music generation requires coordinating global musical decisions with detailed acoustic realization, while controllable creation also needs structured, actionable conditions beyond global text prompts.

  • Method

    The system combines a generation-oriented music tokenizer, Mixture-of-Experts autoregressive generation with optional ABC-CoT planning, and a flow-matching DiT renderer.

  • Results

    StepAudio 3 Music achieves the highest reported AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and highest MuQ-MuLan similarity among evaluated systems, while remaining competitive on SongBench.

  • Takeaways & Limitations

    Explicit musical planning and separation of sequence modeling from acoustic rendering provide complementary control through text, notation, and task-specific references.

  • Takeaways & Limitations

    The reported subsets primarily evaluate vocal and lyrics-conditioned generation and do not establish comparative performance for instrumental, cover-song, or vocal-to-mix generation.

Abstract

from arXiv · show

We introduce StepAudio 3 Music, a large-scale, long-form music generation model that supports explicit musical planning and open-domain text-controlled generation. The StepAudio Music Tokenizer represents audio as a 50-Hz stream from a 65536-entry single codebook, using semantically informed self-supervised and multi-task training to preserve musical structure and reconstruction-relevant information. A flow-matching diffusion Transformer (DiT) predicts continuous StepAudio VAE latents, which our VAE decoder converts into 48-kHz audio. This discrete-continuous design is guided by comparisons of single-codebook VQ, Semantic and Acoustic RVQ, and different DiT configurations. For explicit planning, a Mixture-of-Experts autoregressive model uses ABC notation to produce an intermediate arrangement plan (ABC-CoT) before predicting music tokens, making harmony, rhythm, and melodic structure part of the generation context. A progressive training curriculum and supervised fine-tuning support song and instrumental generation, accompaniment generation from dry vocals, and cover-song synthesis for up to 5 minutes and 30 seconds. With reinforcement learning via direct preference optimization (DPO), the final model achieves the highest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and the highest MuQ-MuLan similarity among the evaluated systems, with competitive SongBench results. On the preliminary Artificial Analysis Music Arena Vocals leaderboard, it obtains a Quality Elo of 1105, behind only Suno V5.5 and Mureka and ahead of Suno V5, MiniMax models, and other systems. Audio demonstrations are available at https://stepaudiollm.github.io/step-audio-3-music.

1 Introduction

StepAudio 3 Music addresses long-form music generation by combining open-domain text control with explicit, readable musical planning. Its generation-oriented representation and progressive training support structured musical development while separating sequence organization from acoustic realization.

  • Motivation: Prior work motivates structured musical interfaces because global prompts alone do not directly specify local phrase development, chord changes, or relationships between song sections.ABC notation extends this line of work with a readable representation of musical content and form.
  • System overview: StepAudio 3 Music combines a generation-oriented tokenizer, MoE autoregressive model, and flow-matching DiT renderer for long-form, text-controlled music generation.Lyrics, prompts, and task-specific references condition generation, while the renderer handles acoustic realization.
  • Representation design: The tokenizer adopts a 50-Hz, 65536-entry single-codebook representation because predictable autoregressive targets can support musical consistency better than reconstruction-optimized bottlenecks.The design reflects a trade-off between acoustic reconstruction and stable long-sequence generation.
  • Explicit musical planning: ABC-CoT first organizes harmony, rhythm, melody, and form into an intermediate arrangement plan before predicting music tokens.The plan provides temporally structured context and can be inspected or revised before synthesis.
  • Training strategy: A progressive curriculum moves from broad music generation, recognition, and understanding to planning, task-specific conditioning, high-quality annealing, supervised fine-tuning, and DPO alignment.DPO aligns outputs with expert judgments of condition adherence and musical quality.

2 Model Architecture

The architecture separates symbolic sequence generation from continuous acoustic rendering, with ABC-CoT adding a two-pass planning interface and codec studies selecting a predictable single-token bottleneck. Renderer experiments indicate that tokenizer information limits gains from simply scaling DiT capacity.

  • Architecture: Lyrics, prompts, and references condition an MoE model that can generate an ABC-CoT plan before producing music tokens, while a separate renderer converts those tokens into waveform audio.The autoregressive model and renderer specialize in musical organization and acoustic realization, respectively.
  • Music codec and renderer: The codec emits one token per frame at 50 Hz from a 65536-entry codebook; a flow-matching DiT predicts aligned 50-Hz VAE latents decoded into 48-kHz audio.The VAE decoder is frozen, and the codec is held fixed while the autoregressive model is optimized.
  • Explicit Musical Planning with ABC Notation: The two-pass factorization produces arrangement a from conditions c, then predicts music tokens m1:T conditioned on c, a, and prior tokens.Both passes use the same autoregressive backbone, after which the renderer realizes the sequence acoustically.
  • Long-form rendering: The renderer processes long-form audio in 30-second chunks, carrying the preceding chunk’s final two seconds of VAE latents as local continuity context.The first chunk uses an all-zero two-second latent context.
  • Detokenizer study: At fixed 50-Hz representation, scaling the DiT from 0.9B to 4B or 8B parameters yields no consistent gain, with the 0.9B model best on most reported metrics and the 4B model best on SDR.The results indicate an effective information ceiling imposed by the tokenizer.
  • Discrete Bottleneck Design: At 25 Hz, Semantic and Acoustic RVQ reconstruct better from ground-truth tokens, but Acoustic RVQ generations are poor in fidelity and musicality, while single-codebook VQ preserves better generation behavior.The comparison separates reconstruction quality from generation quality under predicted-token conditions.

3 Data Pipeline

StepAudio 3 Music builds structured training records by progressively filtering audio and annotating retained songs with vocal, language, lyric, quality, and section information. The pipeline combines automated quality checks with task-specific confidence signals.

  • Pipeline overview: The data pipeline filters low-quality material and converts retained songs into structured records using bandwidth analysis, event detection, source separation, language identification, transcription, and song-structure analysis.The process targets both acoustic quality and structural metadata quality.
  • Quality and content filtering: Bandwidth analysis removes recordings with severely degraded or insufficient spectral information, while audio-event detection excludes samples lacking sufficient musical content.The event detector identifies speech, singing, and music before downstream annotation.
  • Source separation: Demucs separates vocals and accompaniment so isolated vocals can support language identification and lyric transcription with reduced instrumental interference.The original mixture and accompaniment remain available for structural and music-related analysis.
  • Multilingual transcription: Language identification routes reliable samples to language-appropriate ASR systems, using FireRed ASR for Chinese and English and WhisperX for other supported languages.Uncertain language predictions are excluded from downstream lyrics annotation.
  • Structured annotation: Timestamped lyrics retain text, boundaries, and confidence scores, enabling quality filtering and alignment with predicted song sections such as verses, choruses, bridges, and outros.The final records include language, lyrics, section labels, frequency-band measurements, and ASR confidence.

4 Training

Training builds StepAudio 3 Music in stages: it learns a semantically informed discrete tokenizer and fixed flow-matching renderer, then trains an MoE language model with progressive musical-planning supervision.

  • Tokenizer Training: The tokenizer progresses from self-supervised BEST-RQ learning to full-song CTC, Mel, and Chroma supervision, then inserts a 65536-entry single-codebook bottleneck producing 50-Hz music tokens.Each stage initializes from the preceding checkpoint.
  • Tokenizer Training: The joint tokenizer objective combines lyric recognition, Mel reconstruction, Chroma reconstruction, and vector quantization to preserve musical structure and acoustic detail.
  • Renderer Training: The tokenizer is frozen before flow-matching DiT training, which predicts continuous VAE latents from aligned music-token conditions for fixed-decoder waveform rendering.Two-second latent contexts support continuity when rendering 30-second target segments.
  • Renderer Training: The final renderer uses a 0.9B DiT with 50-Hz StepAudio VAE latents, after which the tokenizer and renderer remain fixed during language-model training.
  • LLM Pretraining: The MoE language model uses large-scale pretraining, multi-task mid-pretraining, and high-quality annealing to learn generation, understanding, ABC-CoT planning, and reference-conditioned tasks.The corpus contains more than 100 million songs, over 5 million hours of audio, and approximately one trillion tokens, with additional music-to-ABC and ABC-to-music data.

5 Evaluation

Evaluation combines objective musical-quality and caption-alignment metrics with an independent blind-comparison vocals leaderboard, while also testing rendering, preference alignment, and planning contributions.

  • Evaluation: The evaluation uses SongBench, AudioBox-Aesthetics, MuQ-MuLan similarity, and the Artificial Analysis Music Arena Vocals leaderboard to assess complementary quality and alignment dimensions.The study also examines acoustic rendering, preference alignment, and planning contributions.

5.1 Evaluation Setup

The main comparison uses common benchmark subsets, commercial-system baselines, and a fixed DPO checkpoint with shared decoding and metric-specific aggregation procedures.

  • Evaluation Setup: AudioBox-Aesthetics and MuQ-MuLan use 339 common lyrics-to-song conditions, while SongBench uses 316 common vocal conditions from the mixed prompt-generation track.
  • Evaluation Setup: The comparison includes Suno V5.5, Suno V5, Mureka V9, and MiniMax Music 3 using public-interface outputs collected within a common evaluation window.Baseline service-side inference configurations are not assumed reproducible locally, and default settings are used without system-specific prompt tuning.
  • Evaluation Setup: StepAudio 3 Music is evaluated with the post-SFT, post-DPO checkpoint at temperature 0.9 and top-p 0.95, using the same decoding and preprocessing pipeline for objective evaluation.
  • Evaluation Setup: Scores are arithmetic means over each evaluator’s common subset, and comparisons are made within metrics because evaluators use different subsets and scales.MuQ-MuLan scores the English prompt field, using the original prompt only as fallback.

5.2 Objective Evaluation

StepAudio 3 Music leads the objective comparison on AudioBox enjoyment, usefulness, and production quality plus MuQ-MuLan caption similarity, but not across every evaluator family.

  • Objective Evaluation: AudioBox-Aesthetics measures Content Enjoyment, Content Usefulness, Production Quality, and descriptive Production Complexity, while MuQ-MuLan similarity proxies caption adherence.
  • Objective Evaluation: 7.7086 Content Enjoyment, 8.0052 Content Usefulness, 8.3868 Production Quality, and 0.4465 MuQ-MuLan similarity are the highest reported values for StepAudio 3 Music.These metrics use common benchmark subsets summarized in Table 3.
  • Objective Evaluation: SongBench ranks Mureka V9 first on its individual dimensions, so StepAudio 3 Music does not lead uniformly across evaluator families.

5.3 Subjective Evaluation

The preliminary Artificial Analysis Music Arena Vocals benchmark places StepAudio 3 Music fourth with a Quality Elo of 1105. Because the arena and internal evaluation differ in samples and task format, their results provide complementary rather than directly replicable evidence.

  • Artificial Analysis uses blind pairwise comparisons of tracks generated from the same prompt, with Quality Elo fitted using a Bradley–Terry model.Tracks are normalized toward −16 LUFS and capped at −1 dBTP before comparison.
  • The arena and internal objective evaluation differ in sample composition and task format, so they should be interpreted as complementary evidence rather than direct replications.The internal set supplies fixed lyrics and captions, whereas the arena asks systems to generate their own lyrics from a curated prompt pool and reviewed user submissions.
  • 1105 Quality Elo places StepAudio 3 Music fourth among the systems shown in the preliminary Artificial Analysis Music Arena Vocals leaderboard.The score comes from direct evaluation without adaptation to AA prompts.

5.4 Ablation Studies

The ablations evaluate rendering, preference optimization, and inference-time planning. DPO improves all reported metrics, while ABC-CoT planning and especially LLM plan editing further raise SongBench performance.

  • 5.4 Ablation Studies: The ablation framework separately examines acoustic rendering, preference optimization, and inference-time musical planning, including VAE and DiT reconstruction analyses.The DiT comparison holds the tokenizer fixed and uses ground-truth music tokens.
  • 5.4.2 Preference Optimization: DPO improves all five reported metrics over SFT, including SongBench Musicality, SongBench mean, AudioBox-Aesthetics axes, and MuQ-MuLan similarity.SongBench mean rises from 6.5445 to 6.6438 (+0.0993), while MuQ-MuLan similarity rises from 0.4157 to 0.4465.
  • 5.4.3 ABC-CoT Planning and LLM-Based Plan Editing: ABC-CoT alone improves all seven SongBench dimensions, increasing the mean from 6.6438 to 6.6713 (+0.0275).The gains are modest, particularly for Instrumental and Vocal, and depend on the quality of the generated plan.
  • 5.4.3 ABC-CoT Planning and LLM-Based Plan Editing: LLM-edited ABC-CoT raises the SongBench mean from 6.6713 to 6.7220 (+0.0506), achieving a total improvement of 0.0781 over Direct generation.Compared with Direct, the largest gains are in Mixing (+0.1199), Arrangement (+0.1089), and Melody (+0.1060).
  • 5.4.3 ABC-CoT Planning and LLM-Based Plan Editing: Refining ABC notation gives a text-based LLM an editable score representation for improving rhythmic organization, chord progressions, and instrumentation before acoustic synthesis.The revised plan is supplied as a prefix to the same DPO model for music-token generation.

5.5 Evaluation Limitations

The evaluation has important scope and reproducibility boundaries. Reported comparisons are affected by data selection, single-sample conditions, hosted baselines, task coverage, and incomplete isolation of planning components.

  • SongBench scores are not fully independent of data selection because SongBench is used to filter the SFT data.The third-party blind arena offers complementary external evidence but does not directly validate the ablation results.
  • The main objective comparison uses one generation per condition, so it does not measure variation across sampling seeds.Common-subset filtering excludes conditions where any system failed to produce valid audio.
  • Hosted baseline services lack complete service-side inference configurations, limiting local reproduction of the comparisons.
  • The reported subsets primarily evaluate vocal and lyrics-conditioned generation, leaving instrumental, cover-song, and vocal-to-mix comparative performance unestablished.Separate task-specific evaluations are required for those capabilities.
  • Caption similarity and overall music quality do not establish note-level or bar-level adherence to ABC notation, and the planning comparison does not isolate each component's contribution.The proposed role of music-theoretic knowledge remains an interpretation of the observed results.

6 Conclusion

StepAudio 3 Music combines a predictable music-token representation, MoE sequence modeling, and continuous acoustic rendering for text- and notation-controlled long-form generation. It achieves strong objective comparisons while retaining clear limitations in complex arrangements, long temporal dependencies, vocal naturalness, and high-frequency reconstruction.

  • The system uses a 50-Hz, 65536-entry single-codebook tokenizer with semantically informed training, followed by flow-matching DiT prediction of continuous VAE latents.The VAE decoder renders the resulting acoustic detail into 48-kHz waveform audio.
  • StepAudio 3 Music obtains the strongest AudioBox Content Enjoyment, Content Usefulness, and Production Quality scores and highest MuQ-MuLan similarity, while remaining competitive on SongBench.
  • ABC-CoT expresses an intermediate arrangement before music-token prediction, providing readable context for melody, harmony, rhythm, and form.A progressive curriculum, SFT, and DPO support generation, task execution, and alignment with expert listening preferences.
  • Complex arrangements, very long temporal dependencies, vocal naturalness, and high-frequency reconstruction remain areas for improvement.Future work targets stronger adherence to symbolic plans, extended ABC-based control, long-form coherence, and acoustic quality.
Loading 2609.16034v1…