Source-linked AI summary
ACE-Step 1.5: Pushing the Boundaries of Open-Source Music Generation
Junmin Gong, Yulin Song, Wenxiao Zhao, Sen Wang, Shengyuan Xu, Jing Guo, Xuerui Yang
TL;DR
Open-source music generation has lagged proprietary systems in fidelity, semantic consistency, and complex editing. ACE-Step v1.5 addresses this with a hybrid planner-renderer architecture, distilled inference, intrinsic alignment, and a unified editing framework, achieving sub-second consumer-hardware generation while supporting versatile creative workflows. Its capabilities remain bounded by parameter constraints required for consumer accessibility.
Problem
Open-source music systems remained limited in acoustic fidelity, semantic consistency, and complex editing compared with proprietary capabilities.
Method
ACE-Step v1.5 decouples Language Model planning from DiT acoustic rendering and combines distilled inference, intrinsic reinforcement learning, and an evolving annotation pipeline.
Results
Sub-second inference on consumer hardware is achieved without compromising fidelity, while one toolkit supports synthesis, cover generation, and precise editing.
Takeaways & Limitations
The framework offers a versatile foundation for creative workflows spanning music synthesis, cover generation, and precise editing.
Takeaways & Limitations
Current capabilities remain bounded by parameter constraints necessary for consumer-grade accessibility.
Abstract
from arXiv · showhide
We present ACE-Step v1.5, a highly efficient open-source music foundation model that brings commercial-grade generation to consumer hardware. On commonly used evaluation metrics, ACE-Step v1.5 achieves quality beyond most commercial music models while remaining extremely fast -- under 2 seconds per full song on an A100 and under 10 seconds on an RTX 3090. The model runs locally with less than 4GB of VRAM, and supports lightweight personalization: users can train a LoRA from just a few songs to capture their own style. At its core lies a novel hybrid architecture where the Language Model (LM) functions as an omni-capable planner: it transforms simple user queries into comprehensive song blueprints -- scaling from short loops to 10-minute compositions -- while synthesizing metadata, lyrics, and captions via Chain-of-Thought to guide the Diffusion Transformer (DiT). Uniquely, this alignment is achieved through intrinsic reinforcement learning relying solely on the model's internal mechanisms, thereby eliminating the biases inherent in external reward models or human preferences. Beyond standard synthesis, ACE-Step v1.5 unifies precise stylistic control with versatile editing capabilities -- such as cover generation, repainting, and vocal-to-BGM conversion -- while maintaining strict adherence to prompts across 50+ languages. This paves the way for powerful tools that seamlessly integrate into the creative workflows of music artists, producers, and content creators. The code, the model weights and the demo are available at: https://ace-step.github.io/ace-step-v1.5.github.io/
1 Introduction
ACE-Step v1.5 targets the gap between open-source prototypes and production-ready music generation by separating semantic planning from acoustic rendering. It combines distilled inference, intrinsic alignment, and unified editing capabilities to improve efficiency, control, and scope.
- Hybrid architecture: ACE-Step v1.5 separates structural planning in the Language Model from acoustic rendering in the Diffusion Transformer.The LM expands vague prompts into blueprints, while the DiT focuses on acoustic richness and instrument separation.
- Efficiency: Over 100× inference speedup compresses generation from 50 steps to 4–8 steps.The distillation algorithm is reported to improve signal-to-noise performance simultaneously.
- Efficiency: Sub-second generation of high-fidelity audio is enabled on consumer GPUs using less than 4GB of VRAM.The contribution claims instrumental clarity is retained during this optimization.
- Control: Intrinsic reinforcement learning derives rewards from the model’s own comprehension tasks to align generation across over 50 languages.The framework is designed to enforce stylistic and lyrical constraints without external reward-model bias.
- Omni-task capabilities: A unified toolkit supports cover generation, repainting, track extraction, and vocal-to-BGM conversion beyond standard text-to-music synthesis.The model is presented as addressing diverse, long-tail music-production demands.
2 Data Infrastructure
ACE-Step v1.5 uses an alignment-first data infrastructure that evolves annotation models through seeded supervision, reward modeling, reinforcement learning, and corpus filtration. The pipeline targets fine-grained semantic-acoustic descriptions across large-scale, multilingual, and multi-task data.
- Data infrastructure: An alignment-first data infrastructure maximizes information density per training sample.The authors frame this data-engineering shift as the source of the model’s fidelity and controllability.
- Self-evolving annotation: A Self-Evolving Annotation System uses 5 million Gemini-annotated samples to train specialist captioning and transcription models.Reward models trained on 4 million synthesized negative pairs refine the annotators before they process the 27-million-sample corpus.
- Semantic-acoustic annotation: The multi-stage pipeline describes more than 2,000 musical styles and 50+ languages at fine granularity.It combines foundation supervised fine-tuning with contrastive reward modeling and semantic hard negatives.
- Filtering and augmentation: Metric-based filtration discards samples with low audio-text correlation, while query rewriting addresses the shift from dense captions to sparse user prompts.Qwen3-30B-A3B generates diverse input variations for augmentation.
- Multilingual processing: Stochastic Romanization converts 50% of non-Roman-script lyrics into phonemic representations during training.The strategy is intended to share phonological representations across languages without expanding vocabulary size.
- Curriculum learning: A three-phase curriculum uses over 27 million processed samples for foundation pre-training, omni-task fine-tuning, and later capability unlocking.The fine-tuning phase retains the top 50% of quality samples and adds 6 million stem-separated tracks for editing tasks.
3 Model Architecture
The model architecture combines a waveform-domain acoustic representation, a roughly 2B-parameter Diffusion Transformer, and a Qwen-based planning model. A masked latent framework lets one system perform synthesis and multiple musical editing operations.
- Acoustic representation: A waveform-domain 1D VAE compresses 48kHz stereo audio into a 64-dimensional latent space at 25Hz.The representation achieves a reported 1920× compression rate while preserving acoustic information.
- Acoustic representation: Adversarial tuning over 600k steps sharpens acoustic textures, with the final 100k steps removing the KL penalty and increasing adversarial loss weight from 0.1 to 0.5.The VAE is reported to outperform 2D DCAE baselines in reconstruction metrics.
- DiT backbone: A roughly 2B-parameter DiT uses alternating local Sliding Window Attention and global GQA to balance acoustic detail with long-term musical coherence.The hybrid attention design assigns odd layers to local nuances and even layers to rhythmic and melodic consistency.
- Conditioning and tokenization: Qwen3-0.6B caption embeddings combine with timbre and lyric encoders through Cross-Attention, while FSQ converts 25Hz latents into 5Hz discrete structural codes.The FSQ tokenizer uses an approximately 64k codebook as the structural Source Latent.
- Masked generation: A composite Source, Noised Target, and Mask tensor is patchified to halve the sequence rate to 12.5Hz for throughput.The masked input formulation supports the model’s unified generative and editing setup.
- Omni-task formulation: Manipulating Source Latent and Mask configurations supports six modalities, including text-to-music, cover generation, repainting, extraction, layering, and completion.Quantized latents preserve musical elements such as melody and rhythm during transformations.
- Language-model planning: The Composer Agent generates YAML Chain-of-Thought metadata such as BPM, key, duration, and structure before content output.The LM supports Planner and Listener interaction paradigms among four distinct modes.
4 Training Strategy
ACE-Step v1.5 uses progressive training, distillation, and intrinsic alignment to coordinate efficient generation across its LM and DiT components.
- 20M-sample foundation pre-training establishes priors for musical structure and acoustic distributions before omni-task training.
- Intrinsic reinforcement learning aligns the DiT and LM through internal geometric and probabilistic consistency rather than external rewards.
- The Attention Alignment Score combines coverage, monotonicity, and path confidence, achieving > 95% correlation with human judgments for lyric-audio synchronization.
- Table 1 compares commercial and open-source models using a 1.7B LM with a dynamic distilled DiT and no RL.
5 Evaluation
ACE-Step v1.5 is evaluated with objective metrics, human preference tests, and a usability framework intended to reflect real-world creative workflows.
- The evaluation spans objective metrics, subjective human preference, and a usability assessment for deployment scenarios.
- The test set covers 20 musical styles and bilingual Chinese/English prompts, with Gemini 3 Pro generating structured prompts and lyrics.
- AudioBox CU reaches 8.09, while PQ reaches 8.35 and Coherence reaches 4.72, with Mem 4.72 and Nat 4.59 for temporal consistency.
- Blind A/B testing places ACE-Step 1.5’s subjective quality between Suno-v4.5 and Suno-v5, indicating sub-second generation did not compromise aesthetic fidelity.
- The proposed Usability metric uses a 17-point checklist across six dimensions to assess practical utility in production workflows.
- The evaluation framework shifts attention from isolated quality scores toward deployment barriers and workflow integration in professional creative ecosystems.
6 Conclusion
ACE-Step v1.5 combines hybrid planning and acoustic rendering to provide fast, versatile music generation and editing on consumer hardware. The authors note that consumer-accessibility constraints still bound current capabilities.
- ACE-Step v1.5 decouples structural planning from acoustic rendering to achieve sub-second inference on consumer hardware without compromising fidelity.
- The unified toolkit supports synthesis, cover generation, and precise editing for creative workflows.
- Current capabilities remain bounded by parameter constraints required for consumer-grade accessibility.