PaperlayerView exact revision on arXiv

Source-linked AI summary

ACE-Step: A Step Towards Music Generation Foundation Model

Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, Joe Guo

arXiv:2506.00045v1cs.SDeess.AS

TL;DR

Existing music-generation methods trade off speed, coherence, and controllability. ACE-Step combines diffusion with DCAE compression, a linear transformer, and REPA semantic guidance to build a flexible open-source foundation model. The authors report a robust proof of concept with competitive open-source performance and downstream control, while noting remaining evaluation and generation-quality limitations.

  • Problem

    Text-to-music systems face trade-offs among generation speed, musical coherence, lyric alignment, and controllability, while open-source models lag commercial counterparts.

  • Method

    ACE-Step integrates a diffusion generator with Sana’s DCAE, a lightweight linear transformer, semantic guidance from MERT and mHuBERT through REPA, and flow-based creative controls.

  • Results

    ACE-Step demonstrates the viability of a versatile music-generation foundation model and achieves state-of-the-art performance among open-source models on SongEval.

  • Takeaways & Limitations

    The model serves as a robust proof of concept and stepping stone toward narrowing the gap between open-source and closed-source music generation.

  • Takeaways & Limitations

    Evaluation excluded newer systems including Suno v3.5 and v4, and ACE-Step remains surpassed by the latest commercial offerings.

Abstract

from arXiv · show

We introduce ACE-Step, a novel open-source foundation model for music generation that overcomes key limitations of existing approaches and achieves state-of-the-art performance through a holistic architectural design. Current methods face inherent trade-offs between generation speed, musical coherence, and controllability. For example, LLM-based models (e.g. Yue, SongGen) excel at lyric alignment but suffer from slow inference and structural artifacts. Diffusion models (e.g. DiffRhythm), on the other hand, enable faster synthesis but often lack long-range structural coherence. ACE-Step bridges this gap by integrating diffusion-based generation with Sana's Deep Compression AutoEncoder (DCAE) and a lightweight linear transformer. It also leverages MERT and m-hubert to align semantic representations (REPA) during training, allowing rapid convergence. As a result, our model synthesizes up to 4 minutes of music in just 20 seconds on an A100 GPU-15x faster than LLM-based baselines-while achieving superior musical coherence and lyric alignment across melody, harmony, and rhythm metrics. Moreover, ACE-Step preserves fine-grained acoustic details, enabling advanced control mechanisms such as voice cloning, lyric editing, remixing, and track generation (e.g. lyric2vocal, singing2accompaniment). Rather than building yet another end-to-end text-to-music pipeline, our vision is to establish a foundation model for music AI: a fast, general-purpose, efficient yet flexible architecture that makes it easy to train subtasks on top of it. This paves the way for the development of powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. In short, our goal is to build a stable diffusion moment for music. The code, the model weights and the demo are available at: https://ace-step.github.io/.

1 Introduction

Text-to-music generation must coordinate melody, harmony, rhythm, timbre, long-range structure, and lyrics, while existing open-source systems lag in quality, speed, alignment, and usability. ACE-Step addresses this gap with an open-source foundation model combining compressed latent diffusion, semantic alignment, and controllable generation.

  • Text-to-music generation must jointly model melody, harmony, rhythm, timbre, long-range structure, and lyrical content.
  • Open-source models often trail commercial systems in musical quality, style generalization, lyric-audio alignment, inference speed, and input flexibility.
  • Two-stage systems suffer error propagation and slow sequential generation, whereas latent diffusion offers higher fidelity but needs added techniques for coherence and controllability.
  • ACE-Step is introduced as an efficient, versatile open-source foundation model intended to empower creators and democratize musical creation.
  • Its design integrates DCAE mel-spectrogram compression, flow matching with a linear transformer, REPA semantic guidance, and flow-based creative controls.

2 Related Work

Prior music-generation systems span autoregressive, two-stage, and latent-diffusion paradigms, each balancing fidelity, speed, structure, and controllability differently. ACE-Step builds on diffusion, flow matching, transformer backbones, semantic alignment, and deep audio compression to target a more flexible foundation model.

  • Generative audio systems commonly use two-stage acoustic pipelines or diffusion models to produce high-fidelity outputs.
  • Music generation uniquely requires modeling harmony, rhythm, long-range structure, timbre, and lyric alignment.
  • Two-stage discrete-token systems model long dependencies but can lose information and generate slowly, while latent diffusion improves efficiency and fidelity but requires structural control.
  • Open-source work progresses from slow hierarchical autoregression toward faster autoregressive and diffusion systems for full-song synthesis and lyric coherence.
  • Closed-source systems increasingly use sophisticated multi-stage architectures for coherent, high-fidelity, long-form songs, but their creative intervention may be limited.
  • ACE-Step combines flow matching, diffusion transformers, REPA, and DCAE-based compression as components for efficient conditional music generation.

3 Method

ACE-Step is designed as a fast, general-purpose, flexible music-generation foundation model using diffusion, efficient architecture, and semantic alignment. The method is organized around its data foundation, architecture, and training procedure.

  • ACE-Step adopts diffusion-based generation with efficient architectural choices and advanced semantic alignment techniques.
  • The method is presented through three components: data foundation, model architecture, and training methodology.
  • The model is intended to be fast, general-purpose, and flexible for music generation.

3.1 Data Foundation

ACE-Step relies on a large, diverse, quality-filtered music corpus enriched with textual, lyrical, structural, and musical annotations. Training strategies address missing lyrics, multilingual tokenization, variable durations, prompt diversity, and the difficulty of reliable lyric alignment.

  • The data foundation is a large-scale, diverse audio dataset curated, processed, and annotated to provide conditioning signals.
  • The corpus contains approximately 1.8 million musical pieces, roughly 100,000 hours, across 19 languages, with low-fidelity recordings and live performances removed.
  • Annotations include descriptive captions, lyric transcriptions aligned through IPA and lyric databases, structural segments, BPM, key, and stylistic tags.
  • Robustness strategies cover conditional lyric handling, multilingual phonemic preprocessing, variable-length batching, and diverse textual and lyric prompting.
  • Full-song lyric alignment is difficult because of musical timing, pauses, annotation inaccuracies, backing vocals, and transcription errors.
  • Direct IPA lyric representation proved impractical for editing and cross-lingual phoneme management, while linking lyric semantics directly to pronunciation introduced noise.
  • REPA uses MERT and mHuBERT semantic guidance because diffusion models learn low-frequency musical structure before high-frequency vocal details.
  • Ablations found severe alignment degradation without REPA, and removing it during fine-tuning caused notable performance decay.

3.2 Model Architecture

ACE-Step uses a diffusion model over compressed mel-spectrogram latents, conditioned on text, lyrics, and speaker information. Its architecture combines DCAE compression with a Linear DiT and specialized encoder conditioning.

  • Core architecture: ACE-Step operates on compressed mel-spectrogram latents and integrates text, lyric, and speaker embeddings through cross-attention.The three encoder outputs are concatenated before conditioning the diffusion model.
  • Audio autoencoder: The Music-DCAE uses 8x compression in time and frequency, producing approximately 10.77Hz temporal resolution in latent space.A 32x configuration caused unacceptable audio-quality degradation, motivating the f8c8, channel=8 setting.
  • Core diffusion model: The denoising network is a Linear Diffusion Transformer adapted from Sana for ACE-Step’s sequential audio representation.The provided architecture description identifies the Linear DiT as the core denoising network.
  • Core diffusion model: AdaLN-single shares one adaptive-normalization layer’s parameters across DiT blocks, reducing model size and memory consumption.The same modification replaces standard AdaLN in the DiT blocks.
  • Core diffusion model: The DiT feed-forward layers use 1D rather than 2D convolutions to better match the temporal, sequential structure of the audio representation.This is the second architectural modification described for the adapted Linear DiT.
  • Conditioning encoders: Conditioning uses a frozen multilingual mT5 text encoder, a trainable lyric encoder, and a speaker encoder processing a 10-second unaccompanied vocal segment.The text encoder produces 768-dimensional embeddings, while the lyric encoder follows SongGen’s design.

3.3 Training Objectives

ACE-Step combines flow matching for latent generation with semantic alignment losses that guide intermediate DiT representations toward pretrained audio representations. The final objective is a weighted sum of these generative and semantic terms.

  • Flow matching loss: Flow matching trains the model to reconstruct clean audio latents from noisy interpolations between Gaussian noise and data latents.The model predicts a velocity-related vector from the noisy latent, timestep, and text, speaker, and lyric conditions.
  • Flow matching loss: Noisy latents are formed by interpolating clean latents x0 and Gaussian noise z with coefficient σt.The interpolation is xnoisy = (1 −σt)x0 + σtz.
  • Flow matching loss: The flow matching loss compares the reconstructed latent prediction with the ground-truth clean latent using mean squared error.The reconstruction uses the noisy latent and the model’s predicted output.
  • Semantic alignment loss: The auxiliary semantic alignment loss aligns intermediate Linear DiT features with representations from pretrained audio self-supervised models.The aligned representations are intended to enhance musical coherence, lyric alignment, and intelligibility.
  • Semantic alignment loss: MERT supplies general musical features, while mHuBERT supplies representations associated with lyric intelligibility and speech-like characteristics.MERT processes 5-second chunks at 24kHz, and mHuBERT processes 30-second chunks at 16kHz.
  • Semantic alignment loss: The SSL loss temporally aligns DiT, MERT, and mHuBERT representations before averaging their one-minus-cosine distances.Interpolation or pooling matches the temporal dimensions to a common length T′.
  • Combined objective: The final training objective is the weighted sum LTotal = LFM + λSSL · LSSL, with λSSL empirically set to 1.0.This combines the generative flow-matching term with semantic SSL guidance.

4 Experiments

Training used DeepSpeed ZeRO Stage 2 and manual listening assessments because no automated evaluation pipeline was available. The mHuBERT loss weight was reduced during final fine-tuning to preserve instrumental sounds and musicality.

  • Training setup: DeepSpeed ZeRO Stage 2 was used to train both Music-DCAE and ACE-Step.
  • Training setup: The mHuBERT SSL loss component was reduced to 0.01 during the final 100,000 fine-tuning steps.The reduction followed an initial SSL loss weight of 1.0.
  • Evaluation and monitoring: Manual listening assessments every 2,000 steps monitored training because an automated evaluation pipeline was unavailable.This limited the ability to precisely track training progress.

5 Evaluation

ACE-Step is evaluated through blind human judgments, waveform reconstruction, automatic song-generation metrics, and generation speed. It performs strongly against closed-source baselines in human evaluation and achieves competitive or leading results across several automatic metrics, while remaining subject to metric and comparison-scope caveats.

  • Human Evaluation: 32 participants rated generated samples on musicality, emotional expression, innovativeness, and sound quality using a 5-point Likert scale.The cohort included music professionals, enthusiasts, and general listeners, with randomized sample presentation.
  • Human Evaluation: Approximately 85 in Emotional Expression, 82 in Innovativeness, 80 in Sound Quality, and 78 in Musicality were achieved by ACE-Step, exceeding the reported baseline ranges.Suno v3 scored around 70-75, Hailuo 65-72, and Udio v1 below 15 across categories.
  • Song Generation Metrics: Udio v1 achieved the highest combined style alignment, followed by Hailuo and ACE-Step, with Suno v3, Yue, and DiffRhythm ranking subsequently.Style alignment was quantified using CLAP and Mulan scores.
  • Song Generation Metrics: Hailuo achieved the best lyric alignment, while DiffRhythm and ACE-Step also performed well and Udio v1, Suno v3, and Yue scored lower.The metric used average confidence from Whisper forced alignment, which may favor genres with sparse instrumentation.
  • Song Generation Metrics: Hailuo performed strongly on Audiobox aesthetics, Suno v3 led SongEval Coherence, and ACE-Step was competitive, especially in Memorability and Clarity.Audiobox scores were described as style-dependent; SongEval was reported to have over 90% agreement with human preferences at utterance and system levels.
  • Generation Speed: 15.63x RTF made ACE-Step the fastest evaluated system on an RTX 4090, ahead of DiffRhythm at 10.03x and Yue at 0.083x.The reported comparison makes ACE-Step approximately 188 times faster than Yue.
  • Metrics Interpretation: Human judgments diverged from objective metrics, reflecting subjective musical perception and limitations in metrics’ coverage of nuanced qualities.The authors identify ACE-Step as state-of-the-art among open-source models on SongEval, which they describe as the most reliable objective metric in the evaluation.
  • Metrics Interpretation: ACE-Step showed higher genre fidelity and SongEval musicality than competing open-source approaches, while DiffRhythm’s lyric-alignment advantage was associated with lower SongEval scores.The paper attributes DiffRhythm’s lyric result to pre-aligned lyric integration and describes a possible trade-off with contextual vocal placement.

6 Application

ACE-Step provides fine-grained control over generated and existing audio, while fine-tuned modules support specialized vocal, instrumental, stem, and genre-focused tasks.

  • Controllable Features: ACE-Step offers controllable audio synthesis and direct editing for targeted modifications to musical content.Its capabilities include variations generation, audio repainting, and lyric editing.
  • Controllable Features: Variations generation adjusts the mixing ratio between original and new noise to control the extent of modifications.The method uses training-free, inference-time optimization with additional Gaussian noise based on trigFlow’s formulation.
  • Controllable Features: Audio repainting modifies selected audio regions while mask constraints preserve the remainder of the input.It can target instrumental lines and combine with variations generation for localized changes in style, lyrics, or vocals.
  • Controllable Features: Lyric editing makes localized lyrical changes while preserving melody, vocal characteristics, and accompaniment.The feature works on generated and uploaded audio and is currently optimized for small segment edits.
  • Specialized Generation Tasks: Fine-tuned LoRA and ControlNet-LoRA modules enable lyric-to-vocal, text-to-sample, stem generation, rap specialization, and singing-to-accompaniment.These modules target vocal synthesis, production samples, coherent instrument stems, Chinese rap performance, and instrumental backing from vocals.

7 Discussion

ACE-Step is presented as an open-source foundation model that combines diffusion generation with DCAE and a lightweight linear transformer. The authors identify audio fidelity, style and lyric adherence, output variability, and future data and architecture improvements as ongoing concerns.

  • Discussion: ACE-Step integrates a diffusion-based generator with Sana’s DCAE and a lightweight linear transformer as an open-source music foundation model.The authors describe the system as a proof-of-concept and stepping stone toward narrowing the gap between open-source and closed-source music generation.
  • Limitations: Current audio quality is limited by mel-spectrogram DCAE processing and a 32kHz monophonic vocoder rather than direct end-to-end audio-to-audio generation.The authors also identify uniform 44.1kHz resampling during training as a factor limiting fidelity.
  • Limitations: ACE-Step still has room to improve adherence to specified musical styles and lyrical content.The authors specifically identify more accurate and granular style tagging as a route to better stylistic control.
  • Limitations: Generation quality can vary across attempts, producing a lottery-like experience that the authors aim to reduce.This variability means optimal results may require multiple generations.
  • Future Work: Future work targets larger and longer training, improved multilingual and multi-style support, and more advanced 1D VAE integration.The stated goals include stronger generalization, better text-audio alignment, and improved continuation and inpainting quality.
  • Discussion: The authors position continued development as a path toward a foundational tool for artists, producers, and researchers.This conclusion frames ACE-Step’s role within ongoing open-source music-generation development.

A Training Details

The supplied material identifies tables for Music DCAE training details and ACE-Step model training details, without listing their parameter specifications.

  • Table 3 is titled “Music DCAE Training Details.”
  • The training-detail tables are organized around parameters and specifications.
  • Table 4 is titled “ACE-Step Model Training Details.”

B Evaluation Prompts

The evaluation prompts cover diverse musical styles and include English and Chinese lyrics. They pair style tags with structured song lyrics spanning verses, choruses, and bridges.

  • 20 prompts cover diverse musical styles and include both English and Chinese lyrics.
  • The lyrics are organized into song sections including verses, choruses, and bridges.
  • The prompts use style tags such as rock, hip-hop, folk, R&B, electronic, jazz, reggae, blues, classical, country, pop, and disco.
Loading 2506.00045v1…