Source-linked AI summary
HeartMuLa: A Family of Open Sourced Music Foundation Models
Dongchao Yang, Yuxin Xie, Yuguo Yin, Zheyu Wang, Xiaoyu Yi, Gongxi Zhu, Xiaolong Weng, Zihan Xiong, Yingzhe Ma, Dading Cong, Jingliang Liu, Zihang Huang, Jinghan Ru, Rongjie Huang, Haoran Wan, Peixu Wang, Kuoxi Yu, Helin Wang, Liming Liang, Xianwei Zhuang, Yuanyuan Wang, Dingdong Wang, Haohan Guo, Junjie Cao, Zeqian Ju, Songxiang Liu, Yuewen Cao, Heming Weng, Yuexian Zou
TL;DR
Music foundation models remain limited by closed pipelines and coarse control, while jointly conditioning song generation on style, lyrics, and reference audio remains challenging. HeartMuLa presents an open-source family combining alignment, lyric recognition, low-frame-rate tokenization, and controllable generation. The authors report consistent improvements over existing codecs and song-generation baselines in reconstruction quality, generation performance, and modeling efficiency.
Problem
Existing music systems have reproducibility and control limitations, and jointly conditioning song generation on style descriptions, lyrics, and reference audio remains an open challenge.
Method
HeartMuLa unifies open-source music-text alignment, lyric recognition, hierarchical low-frame-rate audio tokenization, semantic-rich encoding, and multi-condition song generation.
Results
HeartMuLa shows consistent improvements over existing codecs and song-generation baselines in reconstruction quality, generation performance, and modeling efficiency.
Takeaways & Limitations
The released model family is intended as a foundation for future music-understanding and generation research and creative music production.
Takeaways & Limitations
Whole-graph compilation is constrained in cascaded decoding because dynamic control flow, including temperature = 0 or variable CFG scales, cannot be safely captured.
Abstract
from arXiv · showhide
We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.
1 Introduction
HeartMuLa addresses limitations in reproducibility and controllability by unifying open-source music understanding, alignment, tokenization, lyric recognition, and song generation. Its components support rich conditional inputs, efficient long-sequence modeling, and fine-grained musical control.
- Existing music systems often rely on proprietary datasets or closed-source pipelines, limiting reproducibility and downstream research.
- The framework unifies open-source models for music–text alignment, lyric recognition, music tokenization, and controllable song generation.
- HeartCLAP learns shared music–text embeddings for music tagging, cross-modal retrieval, and downstream generative tasks.
- HeartTranscriptor recognizes lyrics in complex musical signals, while HeartCodec uses 12.5 Hz high-fidelity tokens to capture long-range structure and acoustic detail efficiently.
- HeartMuLa accepts style descriptions, detailed lyrics, and reference audio while controlling genre, mood, rhythm, and expressive variation.
- The song-generation framework supports up to six minutes of long-form music, short-video background music, and natural-language style control across song sections.
- Releasing model weights and evaluation protocols is intended to support reproducibility, extensibility, community adoption, and future music-intelligence research.
2 HeartCodec
HeartCodec combines multi-level semantic encoding, ultra-low-frame-rate tokenization, and high-fidelity reconstruction to represent music efficiently while preserving musical and acoustic information. Experiments show strong reconstruction, subjective quality, and downstream generation performance after targeted training stages.
- Semantic-Rich Encoder: HeartCodec extracts complementary phonetic, semantic, and acoustic representations from Whisper, WavLM, and MuEncoder features.MuEncoder features capture attributes such as timbre, phrasing, and melodic structure, while lower-level features encode fine-grained timbral and spectral details.
- Ultra-Low Frame Rate Compressor: 12.5 Hz tokenization reduces sequence length while retaining multi-level information through resampling, query-based compression, and residual vector quantization.The compressed representation uses 8 codebooks with vocabulary size 8192 for downstream language-model-based generation.
- High-Fidelity Reconstruction Decoder: HeartCodec reconstructs waveforms by mapping discrete representations into a continuous latent space before decoding, mitigating quantization-induced fidelity loss.A generative flow-matching module predicts continuous latents, and reflow distillation reduces sampling from 50 to 10 steps.
- Experimental Results: SQ-Finetune substantially improves reconstruction metrics, with HeartCodec achieving higher VISQOL and lower FAD and FD than competing codecs.HeartCodec also remains top-tier for vocal fidelity, with STOI of 0.66 and PESQ of 1.52/2.10, while achieving the lowest WER and competitive SPK_SIM.
- Experimental Results: A guidance scale of 1.25 provides a more natural and balanced auditory experience despite stronger objective gains at 1.5.The authors therefore select 1.25 for the model configuration.
- Experimental Results: SQ-Codec is selected as the flow-matching target latent because it combines strong objective and subjective reconstruction quality with favorable efficiency.SQ-Codec outperforms alternatives across seven subjective dimensions, including Vocal Similarity of 3.90 and Melody Naturalness of 4.25; additional reflow and SQ-Codec fine-tuning improve downstream generation metrics.
3 HeartMuLa
HeartMuLa is a hierarchical, discrete-token song-generation framework that combines global structural modeling with local acoustic-detail prediction. It conditions generation on lyrics, musical tags, and optional reference audio for controllable synthesis.
- 3.1 Hierarchical Architecture: HeartMuLa operates on HeartCodec’s RVQ audio tokens, using a hierarchical process that models coarse musical structure before fine-grained acoustic details.The token sequence contains L frames, each with K RVQ tokens.
- 3.1 Hierarchical Architecture: The global transformer predicts base layer-0 tokens, while a local transformer predicts residual codes, reducing the cost of multi-stream sequence modeling.Sequence factorization assigns long-range modeling to the global backbone and acoustic refinement to the local decoder.
- 3.2 Conditions: Lyrics, structural markers, musical tags, and optional reference audio form conditioning signals for precise control over generated music.Lyrics can include markers such as [intro], [verse], and [chorus], while reference audio supplies a global stylistic cue.
- 3.2 Conditions: Reference audio is represented with MuQ-MuLan embeddings and randomly dropped in half of training examples to support unconditional modeling.The embedding characterizes musical style without including speaker timbre information.
3.3 Training
HeartMuLa is trained progressively from short-context acoustic learning to full-song modeling, supervised refinement, and preference optimization. Its hierarchical objectives separately weight global semantic structure and residual acoustic detail.
- 3.3 Training: HeartMuLa uses four training stages: warmup, pretraining, supervised fine-tuning, and Direct Preference Optimization.Warmup uses 30-second lyric-containing segments; pretraining uses full songs and all three condition types.
- 3.3.2 Optimization Objectives: The total objective combines a global loss for long-term semantic modeling with residual losses for acoustic-detail reconstruction across RVQ layers.The global layer receives greater emphasis because layer 0 captures coarse semantic information.
- 3.3.2 Optimization Objectives: During supervised fine-tuning, the global-loss coefficient increases to λ0 = 2.0 while residual-layer weights are attenuated.Warmup and pretraining instead use λ0 = 1.0 and uniform residual weights.
- 3.3.3 Direct Preference Optimization: DPO uses preference pairs to optimize the policy directly, avoiding explicit reward modeling and online generation during training.Preference data is based on tag similarity, Phonemes Error Rate, and AudioBox-related scores.
- 3.3.2 Optimization Objectives: The hierarchical DPO loss separates preference influence on global semantic coherence from local acoustic details.This decomposition allows the two code groups to be optimized through distinct additive terms.
3.4 Evaluation
HeartMuLa is evaluated with objective and subjective measures covering musical quality, style adherence, and lyric intelligibility. Across five languages, it shows competitive musical performance and particularly strong lyric clarity, while DPO improves targeted dimensions.
- 3.4 Evaluation: HeartMuLa is evaluated across musical quality, style adherence, and lyric intelligibility using objective metrics and blind-listening MOS ratings.Objective measures include SongEval and AudioBox, while subjective evaluation covers perceptual dimensions such as musicality, harmony, structure, and fidelity.
- 3.4.2 Objective Evaluation: HeartMuLa maintains consistent SongEval performance, style adherence, and audio quality across English, Chinese, Japanese, Korean, and Spanish.The evaluation restricts inputs to lyrics and textual style descriptions for fair comparison with baselines lacking audio-reference support.
- 3.4.2 Objective Evaluation: HeartMuLa achieves the lowest Phoneme Error Rate in every tested language, including 0.09 in English and 0.12 in Chinese.The reported comparisons include Suno-v5 and MiniMax-Music-2.0.
- 3.4.3 Subjective Evaluation: The subjective study uses a double-blind procedure with nine listeners, randomized trials, trimmed means, and 95% confidence intervals.Each listener evaluates 20 samples from each model, including English and Chinese samples.
- 3.4.4 Ablation Study: PER-DPO reduces Phoneme Error Rate to 0.0683, Muq-DPO reaches a Tag-Sim score of 0.2839, and Audiobox & Songeval-DPO reaches an average score of 4.448.The final model is obtained by linearly merging models trained on the three preference dimensions.
3.5 Inference Acceleration
HeartMuLa accelerates long-form autoregressive inference through coordinated KV-cache alignment, FlashAttention, CUDA Graphs, and tensor-only control flow. These optimizations target interactive batch-size-1 generation, reducing latency while preserving musical quality and supporting streaming.
- Optimization Results: 5.4× overall speedup reduces end-to-end generation time from 398.3s to 73.4s in the optimized configuration.GPU kernel launches also decrease from 1,561,161 to 979,149, alleviating dispatch and launch overhead.
- KV-Cache Alignment: KV-cache alignment keeps token indices, positional encodings, attention masks, and cache writes synchronized for reliable cache reuse.Separate positional buckets are used when module execution orders differ.
- FlashAttention: FlashAttention exposes only valid cached prefixes and uses precomputed causal masks to reduce kernel launches during long-sequence decoding.The method maintains stable speedups as sequence length increases.
- CUDA Graph: CUDA Graphs capture static transformer execution while dynamic tokens, positions, and control decisions remain external to preserve shape invariance and correctness.This separation enables the latency reduction reported in Table 17.
- Engineering Constraints: Whole-graph compilation is constrained by warm-up cache pollution, frozen random states from in-graph sampling, and dynamic decoding control flow.These constraints require careful separation of static and dynamic pipeline elements.
- Batching: As batch size increases, KV-cache reuse and FlashAttention remain effective but provide smaller relative benefits because inter-module synchronization overhead grows.The optimizations primarily target batch size = 1, where latency and temporal stability matter more than throughput.
- Streaming: Streaming inference emits acoustic tokens or audio chunks frame by frame, with cache alignment and tensor-only control flow maintaining temporal consistency without quality loss.Streaming achieves the lowest end-to-end latency and best intelligibility among evaluated settings.
4 HeartCLAP
HeartCLAP aligns musical audio and text in a shared embedding space using pretrained encoders, contrastive learning, and diverse annotations. It significantly outperforms Laion-CLAP and MuQ-MuLan across retrieval metrics and directions.
- Architecture: HeartCLAP projects music and text encoder features into a shared 1024-dimensional embedding space for cross-modal alignment.Both encoders initialize from pretrained MuQ-MuLan weights.
- Training Objective: InfoNCE contrastive learning pulls matched music-text embeddings together and pushes mismatched pairs apart using cosine similarity and a learnable temperature.The objective operates on embeddings from the paired music and text encoders.
- Training Data: Training annotations cover genre, mood, instrumentation, singer timbre, gender, scene, and topic in tag-based and natural-language formats.Attribute-level and tag-level masking randomly removes categories or individual tags.
- Experiment Results: HeartCLAP significantly outperforms Laion-CLAP and MuQ-MuLan across all Recall@K and mAP@10 evaluation metrics on WikiMT-X.The evaluation covers both Text-to-Music and Music-to-Text retrieval tasks.
5 HeartTranscriptor
HeartTranscriptor adapts Whisper for robust lyric recognition in complex, multilingual music by training on refined musical audio data and evaluating across song benchmarks.
- HeartTranscriptor fine-tunes Whisper for lyric recognition in music, where instrumental interference and singing phonetics challenge standard speech ASR.The training objective uses cross-entropy with teacher forcing on musical audio.
- The dataset pipeline separates vocals from accompaniment, filters recognition outputs by WER or CER, and constructs verified song examples.It uses Demucs for source separation and Whisper-Medium for recognition before quality refinement.
- The evaluation covers full-length Chinese and English songs plus short multilingual slices in English, Chinese, Korean, Japanese, and Spanish.The benchmarks include SSLD-200 and the internal HeartBeats-ASR-Bench.
- HeartTranscriptor achieves the lowest error rates across the evaluated datasets and languages, outperforming baseline and domain-specific models.Table 20 compares WER and CER across models and datasets.
6 Overall Training Datasets
The training resources combine filtered music, lyrics, instrumental audio, and speech data with structural and fine-grained style annotations. A human-validated benchmark evaluates music understanding across acoustic, semantic, and contextual dimensions.
- Training data: The overall training data includes lyric music, instrumental music, and text-to-speech datasets filtered for musicality and lyric alignment.HeartTranscriptor removes mismatched music-lyrics pairs during curation.
- Training data: Style inputs use tags and descriptions covering attributes such as genre, mood, instruments, scenes, vocal timbre, topics, and region.The tag-based approach standardizes varied user style inputs.
- Annotations: Structural annotations mark song sections such as intro, verse, prechorus, chorus, bridge, and outro to support coherent generation.SongFormer provides the structural analysis used during pretraining.
- Annotations: Fine-grained style annotations describe each song section along dynamics and energy, vocal technique, and style and vibe.The pipeline grounds multimodal analysis in raw audio while incorporating textual structure.
- Evaluation benchmark: HeartBeats-Benchmark uses expert curation to map acoustic signals to high-level semantics through a verified evaluation taxonomy.Its dimensions cover musical style, instrumentation, vocal texture, narrative theme, emotional valence, and usage scenario.
- Evaluation benchmark: Random Dimension Dropout masks two of six dimensions per sample, leaving four dimensions to form the final input prompt.This strategy increases input diversity during training.
7 Related Works
Related work spans semantic and acoustic audio tokenizers, language-model and diffusion approaches, and song-generation systems targeting long-range structure and vocal-accompaniment coherence. HeartMuLa positions itself around low-bitrate semantic richness and high-fidelity music reconstruction.
- Audio tokenizers: Semantic tokenizers capture high-level linguistic and phonetic abstractions that are compatible with language-model objectives.They commonly derive representations from self-supervised models such as HuBERT and WavLM.
- Audio tokenizers: Acoustic tokenizers prioritize waveform reconstruction through an encoder, discrete quantizer, and decoder with low inference overhead.EnCodec, HiFi-Codec, and DAC demonstrate this approach across speech, music, and general audio.
- Audio tokenizers: MimiCodec focuses on speech, while MuCodec has limited reconstruction performance relative to the targeted low-bitrate music setting.HeartMuLa instead targets semantic richness and high-fidelity music reconstruction.
- Generation paradigms: Language-model music generation typically converts audio into discrete VQ-VAE or RVQ token sequences before autoregressive modeling.This paradigm builds on the scaling and reasoning capabilities of large language models.
- Generation paradigms: Diffusion models have also been introduced for modeling audio and music data.The related-work discussion presents diffusion as a separate generation paradigm.
- Song generation: Song-generation systems address coherent vocals and accompaniment using hierarchical, dual-sequence, or coarse-grained style-conditioning approaches.Prior systems improve structure and vocal-accompaniment relationships but retain reported limitations.
8 Conclusion
HeartMuLa unifies open-source models for music understanding, tokenization, lyric recognition, and controllable generation. Its ultra-low-frame-rate token framework supports efficient, coherent long-form generation with fine-grained control.
- HeartMuLa is an open-source family covering music-text alignment, music tokenization, lyric recognition, and controllable music generation.
- The framework uses hierarchical audio language modeling built on ultra-low-frame-rate music tokens.
- Experiments report consistent improvements over existing codecs and song-generation baselines in reconstruction quality, generation performance, and modeling efficiency.
- The authors position HeartMuLa as a foundation for future music understanding and generation research and creative music production.
9 Ethics and Responsibility
The HeartMuLa initiative frames responsible music generation around creating novel compositions from learned acoustic representations rather than reproducing copyrighted source material. It also emphasizes labeling AI-generated content and uses watermarking for audio security and authentication.
- HeartMuLa generates novel musical compositions from learned statistical acoustic representations rather than reproducing copyrighted source material.The model is described as operating within an open-sourced initiative for music intelligence research.
- The training dataset is enriched with culturally and linguistically diverse music content to support creation across various musical styles.
- The initiative advocates labeling AI-generated or AI-assisted content to support accountability for creators and listeners.
- HeartMuLa implements watermarking to improve audio security and facilitate content authentication.
10 Contributions
The contributions section lists collaborators from Peking University, The Chinese University of Hong Kong, Scale global holding, and Airo music technology.
- The listed contributors include Yuxin Xie, Yuguo Yin, Xiaoyu Yi, Dading Chong, Yuexian Zou, and Yingzhe Ma from Peking University.
- Zheyu Wang, Gongxi Zhu, Xiaolong Weng, Zihan Xiong, Heming Weng, and Dading Chong are listed with Scale global holding and Airo music technology.
- Dongchao Yang is listed as affiliated with The Chinese University of Hong Kong.