PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Qwen-Music: An Auditable Framework for LLM-Based Music Composition and Diffusion Rendering

Yangbin Yu, Mingyu Yang

arXiv:2609.31652v1cs.SDcs.AIeess.AS

TL;DR

Inference-only access is insufficient to study how representation loss and planning errors interact with acoustic rendering. Open-Qwen-Music reconstructs the modular text-to-music pipeline with inspectable interfaces and released training resources; the authors describe it as the first fully open system of this kind, while not claiming quality parity with Qwen-Music.

  • Problem

    Inference-only access is insufficient for studying how representation loss and planning errors interact across semantic composition and acoustic rendering.

    Source: §1 · p. 1

  • Method

    Open-Qwen-Music combines a semantic tokenizer, autoregressive planner, and diffusion renderer under explicit cross-module contracts.

    Source: §1 · p. 2

  • Results

    The authors present Open-Qwen-Music as the first fully open text-to-music system combining LLM-based semantic composition with diffusion rendering, without establishing quality parity with Qwen-Music.

    Source: §5 · p. 10

  • Takeaways & Limitations

    Its executable pipeline and released artifacts support reproducible analysis of the system and inspection of interfaces among representation, generation, and rendering.

    Source: §1 · p. 2

    Normalization and quantization measurably increase PER, but whether this information loss makes downstream sequence modeling harder remains unestablished.

    Source: §5 · p. 9

Abstract

from arXiv · show

We present Open-Qwen-Music, an open reconstruction of Qwen-Music and a fully specified research system for text-to-music generation that couples LLM-based semantic composition with diffusion-based acoustic rendering. The system comprises a 25 Hz single-codebook music tokenizer, a 3B-parameter autoregressive Music LLM, and a diffusion renderer producing 48 kHz stereo audio, following the cross-module interfaces reported by Qwen-Music. The strongest systems of this design remain closed, and prominent open music-generation projects release weights and inference code without their training corpora or end-to-end training implementations. This limits independent and controlled study of how information loss and prediction errors propagate from semantic representation through autoregressive planning to acoustic rendering. To our knowledge, Open-Qwen-Music is the first fully open release of an LLM-composition-plus-diffusion-rendering text-to-music system. Beyond model weights and inference code, the release includes the training datasets and provenance manifests, complete data-processing, annotation, training, inference, and evaluation pipelines, configurations, and pretrained weights for every learned module. Artifact manifests bind the identities of these artifacts across the complete workflow. Together, these artifacts establish a reproducible implementation of the modular architecture and provide an empirical basis for component-level analysis and future evaluation. We present the system as a transparent, executable research baseline and a starting point for the community, not as evidence of quality parity with Qwen-Music. Open-Qwen-Music is an ongoing effort, and we will continue to improve its generation quality, controllability, and robustness. All release artifacts are available at https://github.com/biang15343100-source/Open-Qwen-Music.

1 Introduction

Open-Qwen-Music reconstructs a modular semantic-composition and acoustic-rendering system while addressing the limited openness of existing systems for reproducible component study. It is presented as an executable research baseline, not evidence of quality parity with Qwen-Music.

1 Introduction

  • Many open projects release pretrained weights and inference code without training corpora or complete training implementations, making module-level effects difficult to measure.The decomposition also couples representation information loss with planner errors that can create out-of-distribution renderer inputs.Source: §1 · p. 1
  • The system is an open reconstruction of Qwen-Music, whose broad design separates autoregressive semantic composition from diffusion-based audio rendering.The paper frames the work around independently reproducible interfaces and component-level diagnostics.Source: §1 · pp. 1–2
  • The framework connects 25 Hz single-codebook Semantic Tokens, 6.25 Hz Melody Tokens, and 48 kHz stereo output through explicit cross-module contracts.Its modules include a tokenizer, autoregressive planner, and diffusion-based acoustic renderer.Source: §1 · p. 2
  • The release includes datasets, provenance manifests, complete processing, training, inference, and evaluation pipelines, configurations, and pretrained weights for every learned component.Artifact manifests bind the identities of artifacts across the workflow.Source: §1 · p. 2
  • The paper reports fixed-protocol evaluations of automatic supervision and semantic discretization, qualitative end-to-end observations, and auditable intermediate artifacts.The reported measurements characterize this implementation rather than general properties of the architecture.Source: §1 · p. 2

2 Dataset and Data Pipeline

The data pipeline builds nested corpora from a broad audio inventory, applies automated preprocessing and annotation, and releases selected corpora with access-aware artifact policies.

2 Dataset and Data Pipeline

  • Dataset-Inventory indexes 26 heterogeneous collections without a unified annotation schema, while Dataset-Annotated and Dataset-Annotation-Filtered form nested subsets and MiniMax is an independent synthetic set.The inventory includes research, public-download, and synthetic collections; annotations are absent, inconsistent, or source-specific.Source: §2 · p. 2, §2.1 · p. 3

2.1 Corpus Construction

  • Dataset-Annotated caps each source at 20,000 records for unified annotation, while Dataset-Annotation-Filtered applies quality filters and balancing, including a 4:1 vocal-to-instrumental ratio.The filtered corpus contains 26,044 synthetic vocal, 9,440 non-synthetic vocal, and 8,871 non-synthetic instrumental records; it has no separate validation or test split.Source: §2.1 · p. 3
  • The MiniMax set uses deterministic generation seeds, duration and audio-quality checks, and the same ASR, alignment, and sectioning pipeline as other corpora.Lyrics supplied for generation requests are not treated as transcripts; only style conditions are retained directly.Source: §2.1 · p. 3

2.2 Preprocessing and Annotation

  • Preprocessing validates audio and deduplicates records, then annotation estimates stems, structure, transcripts, timestamps, audio features, and tags through ordered stages.Annotations are automatic estimates rather than human ground truth; confidence thresholds filter records but do not estimate accuracy, and the filtered set is not an acoustic-fidelity set.Source: §2.2 · pp. 3–4

2.3 Research Data Release

  • The public research package releases Dataset-Annotation-Filtered and the MiniMax set after permission-aware desensitization, with artifact manifests recording access and redistribution decisions.The package combines 36,811 access-screened filtered-corpus audio records with all 25,606 MiniMax records; only artifacts covered by release policy are included.Source: §2.3 · p. 4

3 Model and Training

Open-Qwen-Music reconstructs Qwen-Music as a modular system with fixed interfaces among a semantic tokenizer, autoregressive planner, and diffusion renderer. The section specifies their architectures, training choices, and reconstruction scope.

3 Model and Training

  • The tokenizer, planner, and renderer share fixed interfaces; changing an interface invalidates downstream assets and checkpoints.Table 2 fixes those cross-module interfaces.Source: §3 · p. 4, Table 2 · p. 4

3.1 Music Tokenizer

  • A causal encoder maps 24 kHz mono audio to 25 Hz tokens from a single 32,768-entry codebook.Its 128-bin log-Mel frontend feeds a 24-layer Conformer, with a 16-dimensional cosine vector quantizer after layer 13.Source: §3.1 · p. 4
  • Tokenizer training progresses from broad masked-prediction learning through domain and causal adaptation to semantic supervision and quantization.Domain adaptation uses 175,204 records (8,509 hours); later phases train CTC, Mel, and Chroma objectives, then add the cosine quantizer.Source: §3.1 · p. 5, Table 3 · p. 5

3.2 Music LLM and Melody-CoT

  • The Music LLM reconstruction uses a public Qwen2.5-Omni-3B thinker rather than Qwen-Music’s unavailable initialization, with a deterministic serializer and limited training scope.It uses fixed English section-based windows of at most 30 seconds, no curriculum learning, and no supervised fine-tuning, DPO, or GSPO.Source: §3.2 · p. 5
  • The implementation retains direct text-to-semantic and two Melody-CoT layouts, but final inference uses the direct layout.Section mode pairs each lyric-bearing section with a melody segment; unique-section mode samples a representative segment per section label each epoch.Source: §3.2 · p. 6
  • Planner training predicts Semantic Tokens from English section-based windows, optimizing next-token cross-entropy on Semantic and optional Melody positions.The corpus contains 291,569 windows, each limited to 30 seconds; training uses a 4,096-token per-GPU budget.Source: §3.2 · p. 6, Table 4 · p. 6

3.3 Acoustic Render

  • The renderer combines a Spec-VAE with a Stable Audio 3-compatible conditional DiT, whose Semantic Tokens stay aligned at 25 Hz to the 128-dimensional spectral latent.The VAE takes 48 kHz stereo audio through an STFT and produces [2, 480, T]; training uses conditional flow matching with the VAE encoder frozen.Source: §3.3 · p. 7
  • Renderer training also adapts to noisy semantic conditions, then applies band-specific refinement before inverse STFT.EMDC replaces semantic frames with nearby codebook entries; the Refiner uses low-band phase, mid-band magnitude and phase, and high-band magnitude, while inference uses an 8-step Euler solver.Source: §3.3 · p. 7

4 Evaluation

The evaluation audits automatic annotation and semantic-token quantization with fixed protocols, then qualitatively assesses reconstruction and end-to-end generation. It reports measurable quantization loss and distinct listening limitations across the two paths.

4 Evaluation

  • The evaluation combines held-out quantitative audits of automatic supervision and semantic discretization with qualitative listening observations of the complete pipeline.Each quantitative table specifies its evaluation set, sample count, aggregation, and metric scale.Source: §4 · p. 8

4.1 Annotation Quality

  • The negative confidence–error correlation indicates that agreement between transcription paths is informative for filtering, and task-specific rows should be interpreted individually.The audit retains separate task denominators rather than combining them into an overall annotation score.Source: §4.1 · p. 8, Table 6 · p. 8

4.2 Semantic Tokenizer Quantization

  • 98.2% of the 32,768-entry codebook is active, while nearest-code assignment raises PER by 2.15–2.88 percentage points relative to projected continuous features.The utilization shows no evident codebook collapse on the evaluated panels; the PER increase measures additional phonetic prediction error, not a cross-system ranking.Source: §4.2 · p. 8

4.3 Qualitative Listening Observations

  • With reference-audio Semantic Tokens, rendered audio generally preserves intelligible lyrics, but vocal timbre sounds electronic and stereo width and depth are limited.Source: §4.3 · p. 9
  • End-to-end generations omit or obscure some lyrics; relative to reference-token reconstruction, this suggests Music-LLM sequence errors contribute to additional degradation.Source: §4.3 · p. 9

5 Discussion

Open-Qwen-Music completes the text-to-audio path under substantially constrained data, compute, and supervision conditions, with reproducibility and inspectability as its aim rather than quality parity. Qualitative observations identify limitations in both generated Semantic Tokens and acoustic rendering, while the discussion outlines targeted improvements.

Discussion

  • About 10,100 training hours, one 64-GPU budget per module, automatic annotations, and public component replacements constrain the reconstruction, which nevertheless completes text-and-lyrics-to-48-kHz-stereo generation.Qwen-Music reports more than five million training hours; the reconstruction is presented as reproducible and inspectable, not as evidence of quality parity.Source: §5 · p. 9

Perceptual characterization

  • With reference-audio Semantic Tokens, lyrics remain mostly intelligible, but generated-token end-to-end vocals are often missing or indistinct; the qualitative comparison does not isolate the stages’ causal contributions.The renderer also retains electronic vocal coloration and limited stereo width and depth.Source: §5 · p. 9

Main problems

  • The annotation audit reports lyric LCS recall of 0.889 and attribute accuracies of 75.2%–88.3%, with annotation errors able to propagate into model inputs and renderer conditions.The planned response includes improved annotation models and human-verified calibration subsets.Source: §5 · pp. 9–10
  • Nearest-code assignment raises PER by 2.15–2.88 percentage points versus projected continuous features despite 98.2% codebook utilization; whether this makes downstream sequence modeling harder remains unestablished.The authors identify controlled tokenizer comparisons or direct sequence-modeling measurements as needed evidence.Source: §5 · p. 9
  • The Music LLM uses English section windows no longer than 30 seconds, and its generated Semantic Tokens accompany unstable output and frequent missing or indistinct vocals without an independent held-out evaluation.Planned work targets token stability and completeness, held-out evaluation, broader language coverage, and longer contexts.Source: §5 · pp. 9–10
  • Reference-token rendering generally preserves lyrics but has electronic vocal coloration and weak stereo width and depth; planned work targets these fidelity limits and robustness to generated tokens.The authors aim to retain the observed lyric preservation with reference-audio Semantic Tokens.Source: §5 · p. 10
  • Evaluation currently combines annotation and tokenizer measurements with qualitative system listening, but lacks matched text-to-music comparisons; planned additions include multi-seed generation and controlled listening.No system-level matched comparison is reported.Source: §5 · p. 10

Planned improvements

  • Planned work expands data, reduces semantic-bottleneck information loss, and investigates multi-objective alignment across musicality, lyric intelligibility, and control adherence.The proposed alignment direction uses rewards computed from rendered audio rather than token sequences alone.Source: §5 · p. 10

A.1 Dataset-Inventory Source Breakdown

Table 8 inventories 26 retained Dataset-Inventory sources and provides rounded hours and source citations; accompanying notes clarify that ingested artifacts may differ from official releases.

A.1 Dataset-Inventory Source Breakdown

  • Table 8 covers 26 retained sources in Dataset-Inventory, with hours rounded to two decimals and row citations to primary papers, ingested release pages, or both.Derivative and mirror caveats are stated below the table.Source: Table 8 · p. 15
  • Source labels identify artifacts ingested by the pipeline, not necessarily official upstream distributions; noted examples include a Jamendo-QA mirror and a MedleyDB derivative.The MusicCaps audio came from a local third-party mirror, distinct from the official metadata release.Source: §A.1 · p. 15

A.2 Dataset-Annotation-Filtered Eligibility Rules

The eligibility-rules table summarizes Dataset-Annotation-Filtered criteria, while the accompanying thresholds specify fixed filtering cutoffs rather than estimates of human-label accuracy.

A.2 Dataset-Annotation-Filtered Eligibility Rules

  • Table 9 summarizes the main Dataset-Annotation-Filtered eligibility rules and notes that silence, timeline, and annotation-consistency checks also apply.The supplied table passage does not enumerate individual rule values.Source: Table 9 · p. 15
  • Lyric-confidence thresholds are 0.723 for pretraining, 0.905 for Melody-CoT, and 0.945 for SFT; tags are retained at confidence 0.40 or above.These are fixed filtering thresholds, not estimates of human-label accuracy; the SFT threshold is reserved for future data curation.Source: §A.2 · p. 15

A.3 Annotation Pipeline

The released annotation pipeline is a 12-stage dependency-aware directed acyclic graph, with eligible stages able to run in parallel and independent evidence streams retained until fusion.

A.3 Annotation Pipeline

  • The released annotation pipeline is a dependency-aware directed acyclic graph with 12 stages; stages whose prerequisites are satisfied may run in parallel.The table summarizes each stage’s input-output role.Source: §A.3 · p. 16, Table 10 · p. 16
  • Separate evidence streams are preserved until fusion: the two transcription passes and generative tag predictions remain separate from acoustic voice evidence.These separations are part of the pipeline’s stated design.Source: §A.3 · p. 16

A.4 Annotation Output Schema

Exported annotation rows use one versioned schema as the input to downstream corpus construction. Table 11 lists the public field groups and their semantics.

A.4 Annotation Output Schema

  • Each exported annotation row follows a single versioned record schema.Downstream corpus construction reads this record rather than stage-specific intermediate files.Source: §A.4 · p. 16
  • Table 11 lists the public field groups in each exported annotation record.Source: §A.4 · p. 16, Table 11 · p. 16
  • The schema table also describes the semantics of those field groups.Source: §A.4 · p. 16, Table 11 · p. 16

B Training and Inference Configuration

The configuration section consolidates optimization and decoding settings, separating training phases and reporting effective batch sizes after gradient accumulation. It specifies distinct settings for the semantic tokenizer, Music LLM, and acoustic models, with inference settings listed separately.

B Training and Inference Configuration

  • The configuration tables separate sequential training phases and report effective global batch sizes after gradient accumulation.The section consolidates optimization and decoding settings for the reported system.Source: B Training and Inference Configuration · p. 16

B.1 Semantic Tokenizer

  • The tokenizer uses six optimization phases with shared distributed settings and phase-specific masking, CTC, loss, and codebook procedures.All six phases use 64 GPU processes, BF16, one accumulation step, gradient clipping at 1.0, AdamW, and linear warmup followed by cosine decay; weight decay is 0.01 except during supervised-head warmup. Masked pretraining uses 0.4-second waveform spans, probability 0.3, and −20-dB relative-RMS noise. The final three phases use a 4,096-unit multilingual CTC vocabulary; Mel and Chroma use signed L1 losses. Before the last phase, a 32,768 × 16 cosine codebook is initialized from 1,048,576 projected frames using PCA whitening and five spherical k-means iterations; quantizer commitment weight is 0.25, with 5,000-step gate warmup and frozen-lower-encoder interval.Source: §B.1 · p. 17, Table 12 · p. 17

B.2 Music LLM

  • The Music LLM trains only on the direct text-to-semantic sequence layout in two stages.Each stage uses 64 GPU processes, one sequence per process, two accumulation steps, BF16, and effective global batch size 128; Stage 2 continues the data cursor but starts a new optimizer. Both use AdamW at 1 × 10−4, betas 0.9/0.95, weight decay 0.01, constant schedule without warmup, clipping at 2.0, and a 4,096-token per-GPU budget. Training uses 291,569 English windows of at most 30 seconds, sampled from three strata at probabilities 0.31479, 0.12719, and 0.55802.Source: §B.2 · p. 17, Table 13 · p. 17

B.3 Acoustic Models

  • The Renderer predicts 128-dimensional VAE latents at 25 Hz from 48-kHz stereo audio using linear rectified flow.It uses a Gaussian source, uniformly sampled timesteps, velocity prediction, uniform loss weighting, AdamW, 2,000-step warmup, constant rate, and FP32 EMA decay 0.999. Training begins with 25,000 clean-Semantic-Token updates, then a 1,000-update corruption ramp, followed by the full mixture; 25% of examples receive codebook-neighbor corruption with K = 384.Source: §B.3 · pp. 17–18, Table 14 · p. 18
  • The Spec-VAE uses a two-stage optimization schedule with adversarial and feature-matching losses added in Stage 2.Its common STFT uses a 960-sample Hann window, 480-sample hop, no centering, and 240-sample left pad; DC is retained and Nyquist removed. Stage 1 combines multi-resolution STFT loss, KL regularization, and mixed-scale spectral loss. Stage 2 freezes the encoder, separately optimizes decoder and discriminator, and adds waveform adversarial and feature-matching losses.Source: §B.3 · p. 18
  • The Refiner keeps the VAE fixed and alternates four generator updates per discriminator update.It uses separate AdamW learning rates for the generator, waveform discriminator, and spectral discriminator; its objective combines STFT and instantaneous-frequency/group-delay reconstruction with waveform and spectral adversarial and feature-matching losses.Source: §B.3 · p. 18

B.4 Inference

  • Table 15 reports the inference settings used for text-to-music generation.Source: Table 15 · p. 18
Loading 2609.31652v1…