PaperlayerView exact revision on arXiv

Source-linked AI summary

Pushing the Frontier of Full-Song Generation: Hierarchical Autoregressive Planning Meets Flow-Matching Rendering

Junyu Dai, Xinyue Fan, Weiqin Li, Xiangang Li, Yunjia Li, Bin Ma, Yukun Ma, Chongjia Ni, Yufei Shi, Biao Tian, Haoxu Wang, Menglin Wu, Jianwei Yu, Huaicheng Zhang, Han Zhao, Shengkui Zhao, Haina Zhu

arXiv:2607.20253v3cs.SDcs.AIeess.AS

TL;DR

Full-song generation must coordinate complex musical content while maintaining long-range audio quality and preserving melody in covers. The report combines discrete token planning, melody guidance, and full-song latent rendering in one framework; it reports competitive results in its evaluated settings, while dedicated instrumental and cover evaluation remains future work.

  • Problem

    Full songs combine interacting musical elements, while single-codebook representation and chunkwise synthesis can limit fine detail or long-range quality.

    Source: §1 · pp. 1–2

  • Method

    A unified system combines semantic-aware RVQ tokens and hierarchical autoregressive planning with melody-guided cover conditions and FullDiT full-song flow matching in a continuous VAE latent.

    Source: §1 · p. 3, §4 · pp. 15–16

  • Results

    The system has the highest point estimate in most dimensions on the 500-example multilingual benchmark and earned Elo 1,129 with official rank range 2–3 on the vocal leaderboard.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The framework brings discrete music composition, melody control, and full-length high-fidelity synthesis together in a practical song-generation system.

    Source: §4 · p. 16

    Dedicated quantitative evaluation of instrumental and cover song generation remains future work.

    Source: §1 · p. 3

Abstract

from arXiv · show

In this report, we present a unified song generation framework capable of producing high-quality full-length music from lyrics, text descriptions, and musical attributes. The proposed framework supports three tasks: Lyrics-to-Song Generation, which generates complete songs from text descriptions, lyrics, and musical attributes; Instrumental Music Generation, which creates music without vocals; and Cover Song Generation, which reinterprets existing songs with different styles while preserving their melodic content. Architecturally, our system consists of four main components: a semantic-aware tokenizer, hybird-LM, FullDiT, and a two-level melody module. The tokenizer encodes audio into 8-codebook RVQ tokens for efficient discrete music representation. Based on these tokens, hybird-LM performs hierarchical autoregressive audio-token modeling for full-song generation. To improve audio fidelity, FullDiT performs full-song flow matching in a continuous VAE latent space conditioned on codec tokens, lyrics, and text captions. For cover song generation, the melody module extracts and discretizes melody cues from reference audio to guide generation while preserving the original melodic content. Finally, we investigate DPO, GRPO, and OPD as reward-based post-training strategies for hybird-LM and apply flow-based GRPO to FullDiT to improve musicality and rendering quality. Experimental results on a multilingual automatic benchmark, complemented by the Artificial Analysis Music with Vocals leaderboard, show that the proposed framework achieves competitive performance in the evaluated settings.

1 Introduction

The report presents a unified framework for controllable full-song generation and reports competitive performance in its evaluated settings. It addresses the challenges of representing complex music, maintaining long-range audio quality, and preserving melody in covers.

1 Introduction

  • Full-song generation must coordinate lyrics, vocal melody, accompaniment, instruments, rhythm, and long-range structure while producing clear vocals and natural accompaniment.The report describes full-length music with these interacting elements as considerably more demanding than general audio generation.Source: §1 · p. 1
  • The 500-example benchmark has the highest point estimate in most dimensions; Lucky Dolphin ranks 2–3 with Elo 1,129 on the vocal leaderboard.The leaderboard result is based on 2,289 evaluation samples; the gap to Mureka V8 is 12 Elo points, with overlapping 95% confidence intervals.Source: Figure 1 · p. 2, §1 · p. 3
  • Single-codebook representations can inadequately capture musical diversity, while predicting multiple RVQ codebooks makes audio-token modeling harder.The passage contrasts single-codebook limitations with RVQ-based multi-codebook modeling, which captures finer audio detail but makes higher-codebook prediction more difficult.Source: §1 · p. 2
  • Chunk-wise diffusion or DiT decoding can limit global consistency and ultimate audio quality in long-form music.These approaches are effective for local generation, but the passage identifies long-form music as the setting where the limitation can arise.Source: §1 · p. 2
  • Cover generation must preserve a reference song’s main melody while balancing song-defining note structure against fine-grained singing pitch variation.The passage describes this balance as an open challenge despite prior melody-extraction work.Source: §1 · p. 2
  • The unified system supports lyrics-to-song, instrumental, and cover generation, combining an 8-codebook semantic-aware tokenizer, hybrid-LM, FullDiT, and two-level melody modeling.FullDiT performs full-song flow matching in a continuous VAE latent space conditioned on codec tokens, lyrics, and caption, without chunkwise synthesis.Source: §1 · p. 3
  • DPO and GRPO post-training are used to align the system with human preferences and improve musicality, controllability, and listening quality.The report also investigates OPD and applies flow-based GRPO to FullDiT, as described in its abstract.Source: §1 · p. 3

2.1 Overview

The overview describes a pipeline from text and optional reference inputs to autoregressively planned audio tokens and a rendered full-song waveform. Cover generation adds discretized melody cues to the token-planning stage.

2.1 Overview

  • The tokenizer maps music to discrete sequences using 8-codebook RVQ, trained through BEST-RQ-style pretraining, multitask finetuning, and RVQ token training.The overview identifies these tokens as the discrete representation used downstream by the framework.Source: §2.1 · p. 3
  • Hybrid-LM autoregressively predicts audio tokens from text conditions, optional lyrics, and musical attributes, supporting both vocal and instrumental generation.It uses an 8B global LLM for first-level tokens and a 0.4B local LLM for residual tokens.Source: §2.1 · pp. 3–4
  • For covers, the melody module extracts note-level and frame-level pitch cues, discretizes them into two token scales, and conditions hybrid-LM.The coarse and fine cues guide overall vocal melody and local pitch details to help preserve the reference melody.Source: §2.1 · pp. 3–4
  • FullDiT performs full-song flow matching in a continuous VAE latent with non-causal self-attention, conditioned on codec sequence, caption, and available lyrics.The generated latent is decoded into a high-fidelity waveform.Source: §2.1 · pp. 3–4, Figure 2 · p. 4
  • The report’s post-training strategies combine constructed preference signals, on-policy rewards, and teacher guidance to improve musicality.The overview names DPO, GRPO, and OPD as the strategies for hybrid-LM post-training.Source: §2.1 · p. 4

2.2 Semantic-Aware RVQ Tokenizer

The tokenizer converts audio into discrete music tokens through a three-stage training pipeline. Its Conformer encoder and eight RVQ codebooks yield a compact representation for downstream language modeling.

2.2 Semantic-Aware RVQ Tokenizer

  • The tokenizer training pipeline proceeds through BEST-RQ-style pretraining, multi-task finetuning, and RVQ token training.BEST-RQ-style pretraining uses masked audio features to predict random-projection-quantized targets for contextual and semantic representations.Source: §2.2 · p. 5, Figure 3 · p. 5
  • The tokenizer uses a 24-layer Conformer encoder followed by eight RVQ codebooks, each with 8,192 entries, to produce discrete token sequences.The resulting tokens serve as the target space for hybrid-LM.Source: §2.2 · p. 5
  • Multi-task finetuning uses ASR, reconstruction, and chroma prediction objectives to improve representation of lyrics, acoustic detail, and pitch-related structure.The passage places this as the second stage of tokenizer training.Source: §2.2 · p. 5
  • In the final stage, the bottom 12 Conformer layers are frozen while the top 12 layers and eight-codebook RVQ module are optimized for tokenization.The resulting discrete tokens are described as preserving semantic information while maintaining high reconstruction fidelity.Source: §2.2 · p. 5

2.3 hybrid-LM

hybrid-LM training combines a curated, staged song-data pipeline with hierarchical autoregressive modeling. Its global branch handles temporal planning while a local branch predicts residual codebooks, reducing global sequence length while retaining multi-codebook detail.

2.3.1 Data and Training Pipeline

  • Structured annotations cover style, mood, instrumentation, vocals, song sections, usage, and musical attributes, and join lyrics and section markers as hybrid-LM's textual condition.The corpus is curated from diverse music sources using metadata completeness and perceptual and musical-quality estimates.Source: §2.3.1 · pp. 5–6
  • The three training stages progress from broad multilingual pretraining to style-balanced mid-training and then supervised fine-tuning on a compact, high-quality subset.The stages target general text-to-music correspondence and long-range dependencies, broader style coverage, and final improvements in coherence, condition following, stability, and perceptual quality.Source: §2.3.1 · pp. 5–6, Figure 4 · p. 6

2.3.2 Hierarchical Autoregressive Architecture

  • An 8B global LLM models temporal structure and a 0.4B local LLM predicts residual codebooks; this factorization shortens the global sequence while retaining multi-codebook acoustic capacity.The global model runs once per frame, while the local model predicts the remaining seven codebooks along a fixed-length depth axis.Source: §2.3.2 · pp. 6–7, Figure 5 · p. 7
  • The frame-level global model supports long-range dependencies and complete-song generation up to five minutes.It models melody development, rhythmic continuity, lyric progression, section transitions, repetition, and overall song structure.Source: §2.3.2 · p. 7
  • Music descriptions and section-marked lyrics condition generation, with optional frame-aligned melody tokens for cover generation; classifier-free guidance strengthens adherence to requested attributes and lyrics.The textual condition is dropped in 10% of training examples to enable classifier-free guidance at inference.Source: §2.3.2 · pp. 7–8
  • The training objective averages residual-codebook losses and weights the global first-codebook objective 5:1 against the aggregated local objective.This weighting addresses the residual branch's larger number of prediction targets and reflects the first codebook's role in temporal planning.Source: §2.3.2 · p. 8

2.4 FullDiT

FullDiT renders hybrid-LM's discrete music plan as a continuous acoustic latent through conditional, full-song flow matching. It uses frame-aligned codec conditioning, full-song context, and distractor conditioning to address plausible upstream token errors.

2.4 FullDiT

  • The renderer's separate training corpus filters for acoustic quality, including effective bandwidth, stereo width, quality estimates, and source bitrate.The resulting lossless-audio subset is used to train 48-kHz stereo rendering.Source: §2.4 · p. 8
  • Frame-aligned codec embeddings are added to the corresponding audio states, preserving temporal correspondence between the discrete plan and generated acoustics.The codec condition fuses embeddings from all eight RVQ codebooks at each frame.Source: §2.4 · p. 8
  • FullDiT converts the full-song RVQ sequence, lyrics, and caption into a continuous acoustic latent using non-causal full-song flow matching.A fixed VAE decoder converts the latent into a 48-kHz stereo waveform.Source: §2.4 · p. 8, Figure 6 · p. 9
  • FullDiT attends non-causally across the entire song while conditioning directly on encoded lyrics and captions, enabling modeling of section transitions and long-range timbral consistency.It processes up to 8192 frames, or 327.68 seconds at 25 Hz.Source: §2.4 · p. 9
  • Error-Matched Distractor Conditioning replaces codec tokens at rates matched to hybrid-LM's top-1 error rates while keeping the target acoustic latent unchanged.Near-miss tokens are sampled from cosine top-K neighborhoods, training FullDiT to correct plausible codec errors using remaining codec information, text, song context, and its acoustic prior.Source: §2.4 · p. 9
  • Four-way classifier-free guidance independently controls adherence to the codec plan, lyrics, and global musical description without separate models.The branches provide unconditional, codec-only, codec-plus-lyrics, and fully conditioned predictions.Source: §2.4 · pp. 9–10

2.5 Two-Level Melody Modeling

The two-level melody module supplies cover generation with coarse note-level and fine pitch-level cues. The parallel cues represent both the reference melody's overall contour and its local expressive variation.

2.5 Two-Level Melody Modeling

  • The melody module represents vocal melody with coarse note-level and fine pitch-level features for cover-song guidance.The two levels capture overall contour and local pitch variation, respectively.Source: §2.5 · p. 10
  • A separator extracts the vocal track; MIDI transcription estimates note sequences, while frame-level F0 captures more detailed pitch variation and expressive dynamics.MIDI notes describe the melody's overall contour and progression.Source: §2.5 · p. 10
  • MIDI and F0 are quantized into parallel frame-aligned token sequences at 25 Hz and supplied to hybrid-LM as a two-codebook melody representation.MIDI pitch IDs and log-pitch F0 are each represented with 128 levels, with zero reserved for non-vocal frames.Source: §2.5 · p. 10

2.6 Reward-Based Post-Training

The section describes post-training for hybrid-LM and FullDiT through supervised fine-tuning, preference and reward optimization, distillation, and flow-based GRPO. It reports improvements in musicality and rendering quality.

2.6.1 Supervised Fine-Tuning

  • Supervised fine-tuning trains hybrid-LM on a compact curated subset of high-quality songs, shifting its distribution toward higher-quality samples and initializing preference optimization.Source: §2.6.1 · p. 11

2.6.2 Direct Preference Optimization

  • DPO uses SongBench-scored candidate songs to retain sufficiently different preference pairs, then optimizes their relative likelihoods with an auxiliary SFT loss on preferred samples.The score-gap threshold filters ambiguous comparisons; the combined objective is described as improving stability and preserving generation capabilities.Source: §2.6.2 · pp. 11–12

2.6.3 Group Relative Policy Optimization

  • GRPO samples candidate sequences under the current policy, scores rendered audio with SongBench, normalizes rewards within each group, and adds online fine-tuning on its highest-reward sample.Groups with reward variance below a minimum threshold are excluded from the policy-gradient objective.Source: §2.6.3 · p. 12

2.6 Reward-Based Post-Training

  • Post-training is reported to improve musicality, global structure, and listenability, OPD quality and stylistic consistency, and FullDiT vocal-accompaniment balance and stereo rendering.Source: §2.6.3 · p. 12, §2.6.4 · p. 13, §2.6.5 · p. 13

2.6.4 On-Policy Distillation

  • OPD retains GRPO while a frozen teacher provides soft distributional guidance on student-generated prefixes, transferring its high-quality prior without direct imitation of teacher trajectories.The authors report that a moderate OPD weight supports quality and stylistic consistency while retaining on-policy diversity.Source: §2.6.4 · pp. 12–13

2.6.5 Post-Training for FullDiT

  • FullDiT post-training uses FlowTTS-GRPO, converting deterministic ODE sampling to SDE exploration and optimizing group-relative terminal rewards combining ViSQOL audio quality with an internal music reward.SDE exploration is restricted to an early-step window, and classifier-free guidance is omitted during rollouts.Source: §2.6.5 · p. 13

3 Results

The controlled ablations identify benefits from EMDC, full-song training context, and decoder-side text conditioning, while multilingual evaluations show strong results across several evaluator families.

3.1 Controlled FullDiT Ablations

  • Removing EMDC worsens metrics under synthetic codec corruption, while random 30-second-crop training and removing decoder-side text conditioning each underperform M1.M3a has the largest Production Quality drop; M3b receives no wins in non-tied song-level comparisons.Source: §3.1 · p. 14

3.2 Fine-grained Automatic Evaluation

  • The system has the highest mean score in 15 of 18 reported dimensions on a 500-case multilingual benchmark.The benchmark covers eight genres and five languages.Source: §3.2 · p. 14, Table 3 · p. 15
  • The system leads on five SongBench dimensions, while its Vocal score nearly matches Mureka V8 and Structure remains an improvement area.The Vocal scores are 7.6234 for this system and 7.6248 for Mureka V8.Source: §3.2 · p. 15
  • The system has the highest mean score in every SongEval dimension and leads on three AudioBox-Aesthetic dimensions and CMI-Reward Quality.Production Complexity is descriptive rather than a direct quality measure, and Alignment remains below Mureka V8.Source: §3.2 · p. 15

4 Conclusion

The conclusion presents a unified framework for three song-generation tasks, combining discrete token generation, melody control, and full-length synthesis.

4 Conclusion

  • The framework supports Lyrics-to-Song, Instrumental Music, and Cover Song Generation using a semantic-aware 8-codebook RVQ tokenizer and hybrid-LM.The two-level melody representation captures both main melodic contour and fine-grained pitch variation for melody-guided cover generation.Source: §4 · pp. 15–16
  • FullDiT performs full-song flow matching with non-causal self-attention over a continuous VAE latent, conditioned on full-length codec tokens and available text and lyrics.Source: §4 · p. 16
  • The report investigates DPO, GRPO, and OPD for hybrid-LM post-training and flow-based GRPO for FullDiT, with competitive results on the reported benchmark and leaderboard.Source: §4 · p. 16
Loading 2607.20253v3…