SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation

1. Introduction

• Song generation is challenging because it requires precise vocal–instrumental alignment, while existing multi-stage pipelines can accumulate errors and make training and inference cumbersome.The paper notes that vocal rhythm and instrumental beats are tightly coupled in rap, while vocals often guide ballad emotion.Source: §1 · pp. 1–2
• SongGen is a fully open-source, single-stage autoregressive transformer with controls for lyrics and musical attributes, optional reference-voice cloning, and mixed or dual-track output modes.Mixed mode blends vocals and accompaniment into one output; dual-track mode synthesizes them separately for editing.Source: §1 · p. 2
• In mixed mode, accompaniment’s faster convergence can bias learning against vocals; an auxiliary vocal-token target improves vocal clarity.Vocals pose greater modeling challenges because of semantic density and overlap-related signal-to-noise ratio.Source: §1 · p. 2
• In dual-track mode, token-pattern variants aim to preserve frame-level alignment, and the optimal pattern coordinates tracks at quality on par with mixed-mode generation.The tracks are generated in sync by a single transformer decoder.Source: §1 · p. 2
• An automated cleaning, processing, and quality-filtering pipeline produces a dataset of 540K song clips spanning over 2,000 hours.The pipeline addresses the scarcity of open-source song data.Source: §1 · p. 2
• SongGen outperforms the multi-stage baseline on MusicCaps, with musicality and vocal–instrument harmony competitive with ground truth and expressive vibrato in generated songs.The evaluation reports expressive vocal techniques such as vibrato as contributing to naturalness and authenticity.Source: §1 · p. 2

2.1. Text-to-Music Generation

• Text-to-music work models codec-derived audio tokens with transformer language models or diffusion, but realistic vocals remain challenging for systems focused on instrumental music.The cited work includes Agostinelli et al. (2023) and Copet et al. (2024) for text-to-music models.Source: §2.1 · pp. 2–3

2.2. Song Generation

• Jukebox uses a multi-scale VQ-VAE and transformer cascade but has limited style control and long inference; Melodist and MelodyLM use multi-stage pipelines with Mandarin-pop corpora.MelodyLM uses text-to-MIDI, text-to-vocal, and vocal-to-accompaniment stages.Source: §2.2 · p. 3
• SongCreator lacks text-based control and has limited vocal clarity, while Freestyle focuses on rap and is constrained to one style (Lei et al., 2024; Ning et al., 2024).Rap typically features simpler melodies in the cited description.Source: §2.2 · p. 3
• The work scales up both data and model size, adopting track-decoupled next-token prediction and a two-stage causal LM, while exploring more diverse, efficient token patterns in a single-stage transformer.SeedMusic (Bai et al., 2024) combines auto-regressive language modeling and diffusion; because it is not open-source and relies on a large proprietary dataset, fair comparison with the fully open model is unfeasible.Source: §2.2 · p. 3

3.1. Overview

• SongGen’s autoregressive decoder predicts audio tokens conditioned on user inputs, and a codec decoder synthesizes the final song.The framework supports mixed and dual-track generation modes.Source: §3.1 · pp. 3–4

3.2.1. AUDIO TOKENIZATION

• X-Codec represents audio as discrete codebook tokens, with a one-step delay between adjacent codebook sequences for decoder modeling.The experiments use eight codebooks, each with 1,024 entries, at a 50 Hz frame rate.Source: §3.2.1 · p. 4

3.2.2. MIXED MODE

• Mixed mode predicts tokens from the combined audio, while Mixed Pro adds aligned vocal-token prediction to encourage clearer vocals.The auxiliary vocal heads contribute to training loss only and are not used at inference.Source: §3.2.2 · pp. 4–5

3.2.3. DUAL-TRACK MODE

• Dual-track mode synchronously generates vocal and accompaniment sequences using Parallel codebook concatenation or Interleaving along time.Interleaving uses longer sequences but models track interactions in lower transformer layers; Parallel cannot separate track information before the output heads.Source: Figure 3 · p. 4, §3.2.3 · p. 5

Lyrics Conditioning

• VoiceBPE converts lyrics into phoneme-like tokens, which a small transformer encoder maps to pronunciation-related embeddings.The 6,681-token vocabulary is intended to generalize to unseen words and sung variations in phoneme duration and pitch.Source: §3.3 · p. 5

Voice Conditioning

• A frozen MERT encoder processes a three-second vocal clip to produce voice embeddings for controlling vocal timbre and singing techniques.A 1D convolution aggregates outputs from MERT’s 24 hidden layers and output layer.Source: §3.3 · pp. 5–6

Text Conditioning

• A frozen FLAN-T5 encoder converts descriptive text about attributes such as instruments, emotion, tempo, genre, and singer gender into text embeddings.Source: §3.3 · p. 6

Condition Integration

• The lyric, voice, and text embeddings are separately projected, concatenated along time, and supplied through cross-attention to control song generation.Source: §3.3 · p. 6

3.4. Automated Data Preprocessing Pipeline

• The pipeline gathers 8,000 hours of music, separates vocals and accompaniment, segments clips using voice activity detection, and removes clips with low track energy.The resulting clips average 15 seconds in duration.Source: §3.4 · p. 6
• Two Whisper models transcribe vocals, and clips with transcription edit distance above 20% are excluded; caption alignment is screened with CLAP scores.After preprocessing, the dataset contains about 540K English-voiced clips totaling around 2,000 hours.Source: §3.4 · p. 6

3.5. Training Scheme

• Mixed-mode training aligns modalities, randomly drops the reference voice in half of examples, then fine-tunes on 100K pairs meeting stricter quality thresholds.The final subset uses edit distance ≤5%, CLAPsrc ≥25%, and energy > 1000.Source: §3.5 · p. 6
• Dual-track training initializes from the mixed-mode model after Step 1, adapts the decoder to the new token pattern, and then fine-tunes all weights.Its subsequent stages mirror mixed-mode Steps 2 and 3.Source: §3.5 · p. 6
• Curriculum learning starts with larger loss weights for the first three codebooks, then gradually balances weights to refine finer audio details.The initial weights are 0.25 for the first three codebooks and 0.05 for the rest.Source: §3.5 · pp. 6–7

4.1. Experimental setup

• The evaluation uses 326 English-voiced MusicCaps samples, combining automatic metrics averaged over five seeds with MOS ratings of quality, relevance, vocals, harmony, and singer similarity.SongGen and the multi-stage baseline use a ground-truth three-second vocal clip as reference in the reported Tables 1 and 2.Source: §4.1 · p. 7, §4.2 · p. 7

4.2. Results of Text-to-song Generation

• Across automatic and human evaluations, the single-stage model beats the multi-stage baseline except on PER and SECS; CU rises +9.4% and OVL. +0.57.Other reported gains are CE (+5.9%), PC (+4.7%), PQ (+7.5%), and HAM. (+1.04) on a five-point MOS scale; generation takes 18.04 seconds versus 42.85 seconds for a 30-second sample on an A800 GPU.Source: §4.2 · p. 7, Table 1 · p. 8, Table 2 · p. 8
• SongGen outperforms Suno in text relevance and vocal control, while fine-tuned Parler-TTS is ineffective for text-to-song generation.The paper reports gaps against ground truth and Suno, while noting that training used 2,000 hours of labeled public data.Source: §4.2 · p. 7
• Mixed Pro outperforms Mixed across all metrics, while Interleaving (A-V) leads dual-track patterns and offers better vocal quality but slightly inferior harmony to Mixed Pro.Interleaving has a slightly worse FAD result than Mixed Pro; accompaniment-first ordering performs better than vocals-first across parallel and interleaving patterns.Source: §4.2 · p. 8
• Without a reference voice, performance declines slightly, but listening tests still find enjoyable songs with coherent vocals.Table 3 reports results without voice input.Source: §4.2 · p. 9, Table 3 · p. 9

4.3. Ablation Studies

• High-Quality Finetuning improves all metrics, and curriculum learning outperforms fixed, equal codebook loss weights.The paper attributes the first result to its quality-filtering criteria and describes curriculum learning as prioritizing important tasks before progressively refining details.Source: §4.3 · p. 9, Table 4 · p. 9
• VoiceBPE with a lyrics encoder and cross-attention achieves the best results across lyric-integration variants; cross-attention is reported as more effective and stable than prepending lyrics.The paper argues that phoneme-like VoiceBPE suits song lyrics better than word-level T5 and that the lyrics encoder learns pronunciation patterns from token combinations.Source: §4.3 · p. 9, Table 5 · p. 9

5. Limitations and Future Work

• SongGen generates songs only up to 30 seconds, which the authors say is insufficient for complete song structures.The paper attributes this limit to the scarcity of open-source song data.Source: §5 · p. 9
• The current X-Codec operates at 16kHz, and the authors identify higher-fidelity rendering as future work.They propose training a renderer to upsample the audio.Source: §5 · p. 9
• The proposed renderer is intended to upsample audio for higher-quality output.This is future work, not a capability reported for the current model.Source: §5 · p. 9

• SongGen is a fully open-source, single-stage auto-regressive transformer for text-to-song generation.Source: §6 · p. 9
• Its token-pattern designs support two generation modes: mixed mode and dual-track mode.Source: §6 · p. 9
• Experiments demonstrate the efficacy of the token-pattern design and strong song-generation capabilities in both modes.Source: §6 · p. 9

• SongGen could help novices express creativity with a low entry barrier and streamline experienced music producers’ workflows.Source: Impact Statement · p. 10
• Autonomous song generation and voice cloning carry risks of copyright infringement, intellectual-property misuse, and deepfake audio creation.Source: Impact Statement · p. 10
• The authors say SongGen’s development should include careful consideration of ethical and societal implications.Source: Impact Statement · p. 10

A. Implementation and Training Details

• The lyrics encoder has 6 transformer layers with hidden size 1024; the 24-layer decoder has hidden size 1024 and causal self-attention and cross-attention in every layer.Source: A. Implementation and Training Details · p. 13
• Mixed Pro mode uses vocal loss weight λ = 0.1, and SongGen trains for about 400K steps on 16 Nvidia A100 (80GB) GPUs with batch size 16 per GPU.Source: A. Implementation and Training Details · p. 13
• Training uses AdamW with β1 = 0.9, β2 = 0.99, weight decay 10^-4, and cosine scheduling; learning rates are 10^-4 initially and 5 × 10^-5 for fine-tuning.Source: A. Implementation and Training Details · p. 13
• The multi-stage baseline trains its two models separately for about 200K steps each; both losses plateau around 60K steps, and training configurations are publicly available.Source: A. Implementation and Training Details · p. 13

B. Details in Evaluations

• The evaluation set contains 326 English-voiced MusicCaps samples with no overlap with training, filtered from a 2.8K-sample test set that also includes instrumentals, sound effects, and speech.Source: B. Details in Evaluations · p. 13
• Objective evaluation includes FAD, KL, CLAP and CLaMP3 for audio fidelity or text alignment, plus PER for lyric adherence and SECS for speaker identity.The authors report unexpectedly low CLAP scores and say PER is not ideal for vocal quality because current ASR models struggle with sung vocals.Source: B. Details in Evaluations · p. 13
• Subjective MOS evaluation randomly samples 36 generated clips, with 20 listeners rating each on a 1-to-5 scale across overall quality, text relevance, vocal quality, harmony, and speaker similarity.Source: B. Details in Evaluations · p. 13

C. The Impact of Different Audio Codecs.

• X-Codec surpasses Encodec and DAC on all reported metrics and shows more stable training and faster convergence.These comparisons follow Step 1 training.Source: C. The Impact of Different Audio Codecs · p. 14
• Despite better reconstruction perceptual quality from Encodec and DAC, both produced more invalid song outputs, including lyric-following failures, noise, and silence.X-Codec showed higher success rates in generating coherent vocals.Source: C. The Impact of Different Audio Codecs · p. 14
• The authors speculate that X-Codec’s music exposure in pretraining and its semantic features may contribute to its performance.They selected it despite its 16 kHz sampling rate; high-fidelity, song-specific neural codecs for generative modeling remain an open challenge.Source: C. The Impact of Different Audio Codecs · p. 14