PaperlayerView exact revision on arXiv

Source-linked AI summary

A Tutorial on Generative Models for Music Audio

Mateusz Modrzejewski, Joanna Kurkowska, Dorien Herremans

arXiv:2610.38252v1eess.AScs.AIcs.ETcs.MMcs.SD

TL;DR

Music audio generation now spans varied musical tasks, but the methods and evidence behind its systems are difficult to assess together. This monograph connects system design to musical requirements and reviews the evidence, while noting that data, scale, and evaluation limit cross-system comparisons.

  • Problem

    Music generation systems serve distinct needs, while their methods and the evidence behind comparisons are difficult to follow together.

    Source: §1 · p. 6

  • Method

    The monograph connects musical requirements to representations, objectives, sampling procedures, system comparisons, evaluation, and release evidence.

    Source: §14 · p. 96

  • Results

    Representation, plans, generation schedules, model size, hardware, and implementation shape sequence length, reconstruction, control, and runtime across systems.

    Source: §14 · p. 96

  • Takeaways & Limitations

    The six questions offer a way to read system reports across representation, generation, controls, planning, evaluation, and release documentation.

    Source: §14 · p. 96

    Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.

    Source: §7.4 · p. 52

Abstract

from arXiv · show

Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.

1 Introduction

Music audio generation has moved from low-fidelity, slow proofs of concept toward varied systems for songs, editing, and live interaction. This monograph connects their methods and musical requirements to the evidence available for comparing systems.

1 Introduction

  • Music generation must coordinate lyrics, singing, and accompaniment in songs, preserve unrequested content during edits, and respond quickly enough for live performance.Earlier systems established neural audio synthesis, intermediate-representation pipelines, and long-range symbolic modeling, but Jukebox remained low-fidelity and took hours to sample.Source: §1 · p. 6
  • MusicLM, latent diffusion systems, and MusicGen established codec-token language models and latent diffusion as the two architectural lineages organizing most reviewed systems.MusicGen also showed that a single-stage token language model with codebook interleaving could serve as a strong open baseline.Source: §1 · p. 6
  • Later systems expanded toward full songs, planner-renderer hybrids, faster flow and diffusion sampling, preference alignment, fine-grained control, and instruction-based editing.These developments moved systems from one-shot generators toward production tools.Source: §1 · pp. 6–7
  • Automatic evaluation has lagged generation: Fréchet Audio Distance depends on embeddings and reference sets, correlates weakly with music preferences, and full-song evaluation lacks a standard protocol.Responses include preference-validated metrics, expert benchmarks, human-preference arenas, and learned aesthetics predictors.Source: §1 · p. 7
  • Training and release conditions remain contested, and commercial agreements alone do not establish complete provenance, settle other claims, or demonstrate musicians’ consent or compensation.The monograph treats provenance, memorization, attribution, and output identification as distinct questions and separates announcements, allegations, and court findings.Source: §1 · p. 7
  • The monograph teaches shared foundations and system comparisons so readers can assess design choices, evaluation evidence, and release documentation for their intended use.It also identifies six recurring questions about representation, cost, conditioning, planning, measurement, and artifacts, and states that differences in data, scale, and evaluation limit cross-system conclusions about reported gains.Source: §1 · pp. 8–9

2 Background and Scope

The monograph sets a post-MusicLM review scope, situates current systems in earlier technical foundations, and documents how its literature and system comparisons were assembled. It also explains how codec design shapes sequence cost, semantic content, and reconstruction.

2 Background and Scope

  • The review covers music audio generation from MusicLM onward, including earlier work that establishes methods, and includes general text-to-audio models when evaluated on music or used by music systems.Its system comparison has an end-of-September 2026 eligibility cutoff; later contextual work does not enter the tables or counts, which describe documented versions rather than continuously updated specifications.Source: §2 · pp. 11, 13

2.1 Earlier Foundations

  • Current systems recombine earlier lines including waveform generation, intermediate-representation pipelines, symbolic-token transformers, neural codecs, and adversarial synthesis.Codec RVQ and AudioLM’s semantic-acoustic split made hierarchical audio tokens fast enough to use; differentiable signal processing and RAVE began a parallel instrument line.Source: §2.1 · p. 11

2.2 Literature Collection

  • The literature collection used citation chasing, structured searches, screening, and venue monitoring, with explicit inclusion criteria and source-priority rules.The searches returned 1,115 unique candidates; later abstract rereading found additional candidates that were not included in the finalized classification or counts.Source: §2.2 · p. 12, §2 · p. 13

3.1 Codecs, Tokenizers, and Autoencoders

  • Codec token rate sets sequence length and affects cost and reachable duration, while semantic content and graceful degradation describe what tokens encode and what dropping codebooks preserves.The review describes developments including low-rate and music-specific codecs, semantic supervision, source-aware tracks, and a semantically supervised 25 Hz tokenizer.Source: §3.1 · p. 14

3 Building Blocks

The building blocks translate audio into compressed representations for generation and back into sound. Their design balances sequence cost, reconstruction fidelity, semantic structure, and useful conditioning signals.

3 Building Blocks

  • Discrete representations select vocabulary entries, while continuous representations use real-valued vectors; in both, a generator produces compressed representations that a decoder turns into audio.The decoder is trained on real-recording reconstructions, while a separate generator learns plausible sequences in that space.Source: §3 · p. 15
  • Continuous-latent models use diffusion or flow over VAE latents or spectrograms, while compact and low-frame-rate autoencoders can support long contexts or tractable block-wise generation.Other designs structure latents for generation and editing or use consistency training for single-step reconstruction; high-compression decoders still face high-frequency, phase, and stereo challenges.Source: §3 · p. 15

3.2 Audio-Text Embeddings and Music Encoders

  • Conditioning supplies information such as text, melody, lyrics, or reference audio, and text reaches generation through text encoders or joint audio-text embeddings.When an embedding also scores prompt adherence, the score inherits the distinctions learned by that embedding.Source: §3.2 · p. 15
  • Music encoders provide features for understanding, control, and evaluation, while joint music-text embeddings also support prompt conditioning and adherence scoring.The passage identifies MuLan, CLAP, MERT, MuQ, and CLaMP 3 as examples of these representation families.Source: §3.2 · p. 15

3 Building Blocks

Generation systems depend on representations, data, and evaluators whose properties constrain what they can preserve or measure. The review emphasizes that shared encoders can carry shared blind spots and that training data documentation remains limited.

3 Building Blocks

  • Metrics, reward models, and planning schemes inherit biases from the encoders they use, so representation benchmarks matter to generation research.The passage notes MERT improvements reported by MuQ and CLaMP 3’s cross-modal, multilingual contrastive pretraining.Source: §3 · p. 16

3.3 Data

  • Paired music-text data is scarce, so systems use pseudo-captions or parse musical metadata, while evaluation relies on small human-written caption sets.Open caption corpora include freely licensed music captioned at scale and rewrites of expert album reviews.Source: §3.3 · p. 16
  • Open corpora can be noisy, and some systems condition on per-clip quality or filter and reward outputs with learned aesthetics predictors.Quality conditioning can steer inference toward higher-quality output.Source: §3.3 · p. 16
  • Codec token rate sets sequence length and thus generator cost, while bitrate and reconstruction quality affect preserved detail.Semantic content and graceful degradation are additional properties later tokenizers target.Source: §3.3 · p. 16
  • Continuous latents trade compression against high-frequency, phase, and stereo losses, while an embedding reused for prompt adherence can share the generator’s blind spot.Newer autoencoders aim to combine compactness, reconstruction accuracy, and semantic structure.Source: §3.3 · p. 16
  • Training corpora are mostly undisclosed, and documented datasets under-represent most of the world’s music.The monograph separately flags the scarcity and synthetic nature of captions.Source: §3.3 · p. 16

4 Objectives, Sampling, and Metrics

This section establishes shared notation for generation objectives, sampling, and evaluation, connecting worked examples to system comparisons while organizing design choices around representations, generation patterns, conditioning, preference alignment, and metrics. It also notes that distribution-level distances depend on the embedding encoder and reference set, and that finite-sample estimates can be biased or noisy; KAD uses an unbiased MMD estimator, but variance remains and estimates can be negative.

4.1 The Generic Problem

  • Generation systems can be compared through four choices: representation, distribution factorization, condition routing, and sampler; these choices create trade-offs such as lower token rates reducing cost but potentially reducing reconstruction quality.The framework also connects sampler choices to latency and treats application and release conditions as context for reading results.Source: §4.1 · p. 17

4.2 Representations

  • MusicGen’s EnCodec example produces 200 tokens per second and uses a 2,200-bit-per-second codec; token rate affects sequence length and cost, while codec design constrains reconstruction quality.Residual vector quantization refines frames through ordered codebooks; semantic streams preserve supervised structure for a renderer, while continuous autoencoders use compressed latents and may add a Gaussian-prior penalty.Source: §4.2 · pp. 18–19

4.3 Generative Paradigms

  • MusicGen’s delayed autoregressive schedule generates N frames with Q streams in N + Q − 1 steps rather than NQ, while its 30-second example requires 1,503 steps versus 6,000 flattened steps.Step count is not wall time: it also depends on model, hardware, context growth, and codec decoding; the cited comparison table reports no faster-than-real-time code for its autoregressive token models.Source: §4.3 · pp. 21–22
  • Masked models iteratively unmask confident tokens, while diffusion predicts noise and flow matching regresses to a velocity field; their sampling costs and few-step procedures depend on their distinct objectives and learned paths.Consistency models map trajectory points to an endpoint, whereas flow matching, rectified flow, and reflow are distinct objective or path procedures; plans can separate long-range structure from acoustic rendering.Source: §4.3 · pp. 22–24
  • Conditions can enter as global embeddings, time-aligned controls, or reference audio; inference-time optimization and inversion also enable control or editing without further training, with inversion errors accumulating during reverse integration.Control branches or adapters require training; inference-time optimization costs a sampling run and backward pass per gradient step.Source: §4.3 · pp. 24–25

4.4 Sampling Acceleration

  • Acceleration includes step, consistency, adversarial, and layer distillation, but fast batch generation is not equivalent to streaming: chunk computation must stay below chunk duration, and latency depends on future visibility.Real-time factor measures generation time relative to generated audio duration; it does not indicate how soon the first sound arrives.Source: §4.4 · pp. 25–26

4.5 Preference Alignment

  • Preference alignment uses human ratings or a learned reward with reinforcement learning, or pairwise preferences with DPO; diffusion-DPO uses a denoising-loss surrogate, and preference optimization differs from adversarial speed post-training.Preference methods target outputs a judge or listener prefers; adversarial post-training uses data and a discriminator to move few-step samples toward the data distribution.Source: §4.5 · pp. 26–27

4.6 Distribution-Level Metrics

  • Fréchet Audio Distance embeds generated and reference clips with a fixed audio encoder, fits a Gaussian to each set, and reports the distance between the Gaussians.Source: §4.6 · p. 27
  • KAD applies an unbiased MMD U-statistic to audio embeddings, avoiding the Gaussian assumption but retaining sampling variance and allowing negative finite-sample estimates (Chung et al., 2025).The Fréchet plug-in estimate is nonlinear in estimated means and covariances, so it is biased, generally upward, with small samples.Source: §4.6 · p. 27
  • Fréchet and kernel distances cannot be compared across different encoders or reference sets; small samples bias Fréchet estimates upward and make unbiased MMD estimates noisy.A reported distance needs its encoder, reference set, and number of clips to support a ranking.Source: §4.6 · p. 27

5 Comparing Music Generation Systems

The section organizes music generators along five complementary dimensions, separating model design from the task and release context in which capabilities and results should be read. It then defines representation, method, conditioning, application, and artifact categories and explains how to use the comparison tables.

5 Comparing Music Generation Systems

  • Five dimensions—representation, generation method, conditioning, application task, and release attributes—keep model design distinct from the contexts for interpreting capabilities and results.The dimensions interact, particularly in systems that use different representations and methods at separate stages.Source: §5 · p. 29

5.1 Audio Representations

  • Four representation classes are symbolic, discrete acoustic, continuous acoustic, and hybrid or sketch representations.Examples include MIDI or ABC, codec tokens, VAE latents or waveforms, and intermediate plans or sketches.Source: §5.1 · pp. 30–31

5.2 Generative Methods

  • Autoregressive, masked or parallel, diffusion, flow-matching, and planner-renderer methods differ in factorization and the capabilities or costs that follow.Autoregression supports continuation and streaming but scales with token count; parallel decoding reduces latency, while diffusion supports editing and inference-time control. Flow variants target fewer solver steps, and planner-renderer hybrids separate planning from audio rendering.Source: §5.2 · pp. 31–32

5.3 Conditioning and Control

  • Conditions range from global text and reference audio to time-aligned signals, structural plans, multimodal inputs, and interactive control.The condition may apply to a whole output, align with the audio, or enter as a reference recording; controls include chords, rhythm, dynamics, gestures, and live input.Source: §5.3 · pp. 32–33

5.4 Application Classes

  • Six unordered application classes cover instrumental text-to-music, lyrics-to-song, production co-creation, live interaction, cross-modal scoring, and unified understanding and generation.Systems are tabulated once in the class matching their main evaluation; output, condition, and constraint rules resolve overlaps, with the stricter class taking precedence in ties.Source: §5.4 · pp. 33–34

5.5 Reported Attributes and Release Conditions

  • Release tables record system attributes such as duration, output format, latency, openness, provenance, and post-training, while unsupported entries remain unknown.The tables omit model scale because reporting is inconsistent, document memorization or watermark information for only ten of 105 systems, and do not score perceived output quality.Source: §5.5 · p. 34

5.6 Reading the Comparison Tables

  • Table 5.2 links values of the five comparison dimensions to the sections explaining their mechanisms and reviewing systems that use them.Readers can start from a system description to follow design choices, or from a goal and its constraints to trace backward to relevant choices.Source: §5.6 · p. 35, Table 5.2 · p. 36

5.7 Relation to Earlier Surveys

  • The taxonomy adapts earlier surveys by retaining useful dimensions while emphasizing conditioning, application task, and released artifacts for music-audio systems.Earlier taxonomies organize symbolic generation, musical dimensions, audio levels, or modalities; this framework connects those views to its five dimensions.Source: §5.7 · pp. 36–37

6 Instrumental Text-to-Music

Instrumental text-to-music generates seconds-to-minutes of studio-style audio from a global text condition, allowing direct examination of representations and sampling cost. The reviewed evidence distinguishes runtime more clearly than quality: no paradigm dominates when data, representations, scale, and metrics vary.

6.1 Autoregressive Token Language Models

  • MusicLM predicts semantic, coarse, and fine tokens in three stages, whereas MusicGen replaces that hierarchy with one delayed-codebook transformer and adds chromagram melody conditioning.Source: §6.1 · p. 38
  • Autoregressive systems either delegate acoustic detail to a renderer or scale a single acoustic-token space; a controlled comparison found quality, speed, and control trade-offs rather than one dominant paradigm.Source: §6.1 · pp. 40–41

6.2 Masked and Parallel Decoding

  • Masked token models generate positions iteratively: VampNet changes masks for multiple tasks, while MAGNeT reports near-autoregressive quality at substantially lower latency.Source: §6.2 · p. 41

6.3 Diffusion and Flow Matching

  • Diffusion spans waveform cascades and compressed latents; Stable Audio uses timing-conditioned latents for structured 95-second stereo pieces and coherent full-length tracks.Source: §6.3 · pp. 41–42
  • Flow matching regresses velocity along a chosen path; reflow or distillation reaches few solver steps, while Presto distills steps and network layers for faster-than-real-time latent diffusion.Source: §6.3 · p. 42

6.4 Lessons and Open Trade-offs

  • Six Table 6.1 systems report faster-than-real-time generation, but quality comparisons cannot isolate paradigm effects because data, representations, scale, and metrics vary.Source: §6.4 · p. 43
  • MusicGen is the common open adaptation backbone and Stable Audio offers documented-source examples, but latent compression losses have not been balanced against sampling cost in a controlled study.Source: §6.4 · p. 43

7 Song Generation

Song-generation systems divide lyric, vocal, accompaniment, and structural responsibilities across token models, diffusion or flow models, and planner-renderer hybrids. Their reported differences are difficult to compare directly because training data and evaluation protocols vary.

7 Song Generation

  • Full-song generation must coordinate lyrics, vocals, accompaniment, and song sections over outputs that may last several minutes.Length alone is not the challenge: the words must align with singing, vocals stay consistent, and accompaniment and sections support the same song.Source: §7 · p. 47

7.1 Commercial Systems and Compositional Precursors

  • Early commercial systems disclosed little architecture, while academic precursors decomposed song generation into accompaniment, vocals, melody, or chord-conditioned stages.Later systems moved some of these decompositions inside a model, including interacting vocal and accompaniment token sequences in SongCreator (Lei et al., 2024).Source: §7.1 · p. 47

7.2 Open Systems

  • Open systems span track-decoupled or interleaved token models, whole-song flow matching, and preference-aligned generators.Examples include YuE’s five-minute songs (Yuan et al., 2025), SongGen (Liu et al., 2025d), DiffRhythm (Ning et al., 2025), and LeVo’s direct preference optimization (Lei et al., 2025).Source: §7.2 · p. 48

7.3 Planning, Scale, and Post-Training

  • Planner-renderer hybrids use a token model to sketch a song and a diffusion or flow model to render acoustic detail, but imperfect predicted plans create exposure bias.The full-context renderer is trained against this mismatch; systems also differ in plan format and post-training, including YuE2’s editable score and semantic tokens conditioning a flow renderer (Multimodal Art Projection, 2026).Source: §7.3 · pp. 48–49, Figure 7.1 · p. 50

7.4 Lessons and Open Trade-offs

  • The section groups song methods into token models, whole-song diffusion or flow, and planner-renderer hybrids, with lyric alignment and sampling cost among their comparison axes.The review also notes that commercial systems are closed and their quality is not compared with open systems.Source: §7.4 · pp. 51–52
  • Song systems differ in lyric alignment, sampling cost, and explicit structure, but corpus and evaluation differences prevent clean cross-system quality comparisons.Token models carry lyrics in their sequences; diffusion models need alignment supplied or recovered, while reported flow systems generate full songs in seconds.Source: §7.4 · pp. 51–52

7.3 Planning, Scale, and Post-Training

  • Preference alignment can reduce output diversity, making diversity preservation a separate design concern, while which signals alignment rewards should contain remains an open question.Systems choose plans such as melody token streams, editable scores, or semantic sketches, without a comparison of what each gives the renderer and user.Source: §7.4 · p. 52

8 Control, Editing, and Co-Creation

Control, editing, and co-creation methods trade training effort, inference-time computation, and preservation of the source recording. The section organizes approaches by how controls enter generation, how edits use source audio, and how stems are represented.

8.1 Time-Varying Control

  • Time-varying controls enter through trained mechanisms, inference-time optimization, or activation steering, with costs ranging from model updates to per-output computation.Adapters can keep a pretrained backbone frozen but still require adapter training; lightweight conditioners reduce that added training burden.Source: §8.1 · pp. 56–57

8.4 Lessons and Open Trade-offs

  • Explicit plans give users a way to specify and revise musical structure over minutes, while adapters, control branches, and editors place costs at different stages.A plan may encode sections and textures or a multilevel song program; the section concludes that multiple controls still lack a common training and evaluation framework.Source: §8.1 · p. 57, §8.4 · p. 60

8.2 Editing and Instruction Following

  • Editing methods learn from instruction triplets, invert a recording into a pretrained generator, or construct an editing path without inversion.Inpainting also supports targeted regeneration, while separate evaluation work addresses what an edit should leave unchanged.Source: §8.2 · pp. 57–58

8.3 Stems and Accompaniment

  • Stem systems generate named tracks, treat stems as parallel channels of a shared distribution, or generate accompaniment conditioned on the remaining mix.The parallel-channel systems in Table 8.1 use public research datasets, whereas named-stem and mix-conditioned systems use in-house or licensed catalogs, affecting reproducibility.Source: §8.3 · pp. 58–59

8.4 Lessons and Open Trade-offs

  • Adding control trades additional training against per-edit computation and source preservation, and the reviewed routes have not been compared under one protocol.The papers use different training and inference costs and preservation measures; how to support multiple controls within a common framework remains open.Source: §8.4 · p. 60

9 Real-Time and Interactive Generation

Real-time music generation must meet both throughput and response-delay constraints while receiving input. The reviewed approaches either reduce the sampling cost of larger models or build compact, controllable instruments for tight hardware budgets.

9.1 Streaming Generation and Live Accompaniment

  • Streaming generation either accelerates offline models or changes generation to operate chunk by chunk, while live accompaniment additionally requires causal input handling.Block-wise diffusion with caching has been reported to match and improve on discrete autoregressive live-model inference cost on consumer hardware.Source: §9.1 · pp. 66–67

9.2 Neural Audio Synthesis as an Instrument

  • Neural instruments use compact latent or synthesizer controls for real-time performance, with systems designed for laptops or embedded hardware.RAVE reports 48 kHz synthesis at twenty times real time on a laptop CPU; its non-causal streaming companion incurs additional buffering latency.Source: §9.2 · pp. 68–70
  • The instrument and streaming lines approach latency from opposite directions: one starts with small models, while the other reduces the cost of larger offline models.Both need evaluation of response latency and musical interaction as well as output quality, within the available computation budget.Source: §9.2 · p. 70

9.3 Lessons and Open Trade-offs

  • The streaming text-to-music models typically run on GPUs with text prompts, while neural instruments typically run on CPUs with audio or gesture control.Latency figures use different hardware and lack a shared procedure, and the two lines are evaluated differently.Source: §9.3 · pp. 70–71
  • Future visibility and chunk duration trade perceived latency against coherence, and live systems must satisfy response delay as well as sustained throughput.A chunk may use input through t + ϕ and takes computation time to produce; sustained streaming requires computation no longer than chunk duration τ.Source: Figure 9.1 · p. 68, §9.3 · p. 71

10 Boundaries and Adjacent Areas

The adjacent areas connect to music audio generation through expanded conditioning, unified language-model interfaces, and symbolic planning or co-creation. The section marks these connections as selected examples rather than full surveys.

10 Boundaries and Adjacent Areas

  • The section covers selected examples connecting adjacent areas to music audio generation; its structured searches did not include a topic group for symbolic generation.The cited branch surveys provide fuller treatments.Source: §10 · p. 72

10.1 Cross-Modal Scoring

  • Video-to-music systems add temporal alignment alongside semantic alignment, while video and dance conditioning leave the underlying generative paradigms unchanged.Examples include event curves without paired training data and diffusion systems conditioned on motion or visual cues.Source: §10.1 · pp. 72–73, §10.3 · p. 75

10.2 Unified Understanding and Generation

  • Unified systems either route language-model requests to specialized generators or use one model for both understanding and generation.A third pattern distributes composition across agents, mostly in the symbolic domain; one reported decentralized swarm outperformed a centralized system across quality, diversity, and structural variety.Source: §10.2 · p. 73, §10.3 · p. 75
  • Audio-side agent frameworks orchestrate existing generators, but whether coordination rules can substitute for parameter updates remains open for audio output.The claim has been tested on symbolic output.Source: §10.2 · p. 74

10.3 Symbolic Generation at the Boundary

  • Symbolic generation supports audio systems as an intermediate control layer and a site for co-creation research.Hybrid sketch representations connect symbolic and audio generation by placing an intermediate plan or learned sketch between condition and sound.Source: §10.3 · pp. 74–75

11 Evaluation

Evaluation instruments answer different questions, and automatic metrics do not provide interchangeable evidence of music quality. The section reviews human evaluation and learned judges, recommends transparent listening-test reporting, and identifies gaps that constrain system claims.

11.1 Distribution-Level Metrics

  • FAD varies with its embedding and reference set and correlates weakly with human preferences on modern text-to-music output, while MAD rankings agreed with listeners better than FAD’s in MusicPrefs validation.MAD uses self-supervised MERT embeddings; results across metrics and human ratings can still be inconsistent.Source: §11.1 · p. 76
  • Distribution metrics compare collections in a representation, prompt-adherence scores assess description matching, and listening tests or task-specific measures address other questions rather than interchangeable measures of quality.The MusicGen example combines FAD, label-distribution KL divergence, CLAP prompt adherence, and listening ratings; the listening test is the result to weigh most when raters and items are reported.Source: §11.1 · pp. 76–77

11.2 Human Evaluation and Learned Judges

  • Human evaluation includes pairwise arena votes, controlled preference studies, workflow-fit studies, musician-centered assessment, and co-design with music therapists.Learned evaluators predict ratings from audio using expert-annotated corpora, but aesthetic scorers can learn genre as a shortcut and overestimate pop music.Source: §11.2 · pp. 78, 81

11.3 A Minimum Listening-Test Protocol

  • A listening-test report should state prompt and item sampling, listener recruitment and training, randomization, repeated measures, intervals, rater effects, and multiple-comparison handling.Playback conditions should also be specified and held consistent across conditions.Source: §11.3 · p. 82
  • Use preference tests for unconstrained outputs without a meaningful common reference, and MUSHRA when a common reference exists, such as reconstruction or processing.If a learned judge was used for post-training, report a human test on the same items rather than the judge score alone.Source: §11.3 · p. 83

11.4 Gaps and a Reporting Recommendation

  • Full-song evaluation lacks an accepted protocol, and long-range structure has no standard metric that distinguishes coherent form from locally plausible drift.Current batteries of learned judges are not standard and are validated against small rater pools.Source: §11.4 · p. 83
  • Reports should disclose the encoder, reference set, and sample count for distribution scores, and provide independent human comparisons when learned judges guide optimization.FAD estimates are biased at finite sample sizes, while unbiased MMD estimates remain variable; agreement with listeners depends on the metric, embedding, and evaluation population.Source: §11.4 · p. 84
  • A judge used as a training reward cannot independently establish improvement, because post-training against a judge can produce high scores without listener agreement.Reward hacking of this kind has been observed in live interaction models.Source: §11.4 · p. 84

12 Data, Provenance, and the Legal Record

The section separates evidence about training sources, memorization, attribution, and output detection, and places technical findings alongside the legal record. It emphasizes incomplete disclosure and cautions that each kind of evidence supports only limited inferences.

12 Data, Provenance, and the Legal Record

  • Training-data provenance, attribution, memorization tests, and output detectors answer distinct questions, and each leaves the others open.The section calls for technical measurements and the legal record to be read together while keeping their limits explicit.Source: §12 · pp. 85, 87
  • Documented-source systems provide reference points, but source descriptions and filtering procedures do not establish that a corpus is free of copyright or that every relevant right is cleared.A Creative Commons license does not guarantee clearance of all other rights.Source: §12 · p. 87

12.1 Litigation, Licensing, and Creators’ Claims

  • Label-company settlements and licensing announcements concern identified parties and proposed products, but do not establish authorization for every prior use or consent and payment for all affected creators.The musicians’ treatment and label licensing are separate questions; the union’s 2026 allegations have not been adjudicated.Source: §12.1 · p. 88
  • The Munich Regional Court found infringement concerning six compositions in Suno v3.5 and v4, but the judgment is not final and should not be generalized to every model or training use.The court also held that specified lyrics were reproduced in OpenAI models and prompted outputs; both judgments discussed are non-final.Source: §12.1 · p. 89

12.2 Memorization

  • Few system authors measure memorization, although independent studies examine its relation to training-set size, detection methods, sampling mitigations, and prompting attacks.YuE is noted as an exception among major releases reporting a memorization analysis alongside the model.Source: §12.2 · p. 89

12.3 Watermarking and Detection

  • Watermarks and detectors provide fallible evidence about output origin, but neither establishes training-data provenance or permission to use a recording.Codec resynthesis is a difficult watermarking case, while detector performance can degrade under speed and pitch manipulation or broadcast conditions.Source: §12.3 · pp. 89–91

12 Data, Provenance, and the Legal Record

  • 43 of 105 table rows carry only data code U, meaning undisclosed, in-house, or web-collected data; these codes describe disclosure, not legality or creator consent and compensation.Only 20 rows carry L or CC codes, and many academic and most industrial models do not disclose their corpora.Source: §12 · p. 86, §12.3 · p. 91

12.3 Watermarking and Detection

  • Input records, attribution methods, and output audits can complement one another, but their conclusions depend on the workflow and test and do not determine who should be credited or paid.Comparisons do not isolate licensing effects when training data, scale, and evaluation also differ.Source: §12.3 · p. 91

13 Discussion: Trends and Open Challenges

The reviewed collection shows shifts toward song generation, planner-renderer designs, faster sampling, preference alignment, and revised evaluation practice. It also identifies unresolved challenges in interpretability, live use, control, long-range evaluation, data, and cultural coverage, then offers audience-specific recommendations.

13.1 Trends Across Classes

  • Song-generation systems rose from none in 2023 to 25 in 2025–2026, while instrumental systems fell from six to five across those periods.Planner-renderer designs, faster sampling methods, preference alignment, open releases, evaluation practice, and legal developments also changed across the collection.Source: §13.1 · p. 92

13.2 Open Challenges

  • Fine-grained control can trade off against fidelity or require per-control training, while unifying adapters, conditioners, inference-time optimization, and plan-based approaches remains open.Plan-based control also lacks widely adopted evaluation of long-range form, motif development, and tension across minutes and genres.Source: §13.2 · p. 93
  • Evaluation lacks a widely adopted shared protocol; reusing a training-reward judge risks overfitting, judges inherit dataset biases, and controllability claims need counterfactual tests.Long-range musical structure is only partly scored, and a solution would require protocols validated against expert judgments.Source: §13.2 · p. 93
  • Training data under-represents many musical traditions, while detection degrades under manipulation, mixing, and broadcast conditions.The section also calls for reproducible open, licensed corpora at scale and release-time memorization reporting.Source: §13.2 · pp. 93–94
  • Interpretability remains limited, and live systems still need task-appropriate latency, stable reinforcement-post-trained behavior, musician-developed interfaces, and attention to embedded deployment and energy.Early interpretability work discovers and steers concepts, probes symbolic generators, and applies findings to editing.Source: §13.2 · pp. 92–94

13.3 Recommendations

  • System reports should document representation, factorization, conditioning, sampling, evaluation details, and whether a learned judge supplied a training reward.Evaluators should validate measures with listeners, document recruitment, use independent evaluators, and test control with counterfactual prompts.Source: §13.3 · p. 94
  • Model and data releases should document corpus sources, hours, licenses, memorization tests, and origin-verification conditions; readers should choose systems by task and control needs.The recommendations identify open backbones for adaptation, distinguish closed commercial song systems, and match adapters, control branches, inference-time optimization, or plans to different uses.Source: §13.3 · pp. 94–95

14 Conclusion and Future Work

The conclusion connects musical requirements to model representations, objectives, and sampling procedures, and proposes six questions for reading system reports. It identifies limits on comparisons and sets practical priorities for evaluation, control, releases, and interactive systems.

14 Conclusion and Future Work

  • The six questions organize system reports around representation, generation, control, planning, evaluation, and release documentation.The monograph links these design choices to the requirements of generation, editing, and performance.Source: §14 · p. 96
  • Representation constrains sequence length, reconstruction, and supported controls; runtime also depends on schedule, model size, hardware, and implementation.Streaming systems must meet both throughput and response-delay requirements.Source: §14 · p. 96
  • Control routes differ in computational cost and source-preservation behavior, while the choice of plan and renderer reliability remain open design questions.Planner-renderer systems help specify and revise structure, but single-stage systems remain alternatives.Source: §14 · p. 96
  • Evaluation measures provide different kinds of evidence, and a learned judge used during training cannot alone provide independent evidence of improvement.Results may depend on representation, reference material, protocol, or study population.Source: §14 · p. 96
  • The tables cannot isolate the effects of individual changes because reviewed systems also differ in data, scale, and evaluation.Practical priorities include shared full-song evaluation, control and editing comparisons, auditable release information, and interactive assessment with musicians.Source: §14 · pp. 96–97

Use of AI Tools

The authors report using AI assistants for literature search, bibliographic checks, attribute gathering, table assembly, and language editing.

Use of AI Tools

  • The authors used AI assistants for literature and bibliography work, gathering system attributes, assembling comparison tables, and editing language.They state that they checked every claim, table cell, and reference against cited sources and take responsibility for correctness.Source: Use of AI Tools · p. 98

A Search Strings and Counts

The appendix documents structured arXiv searches, their query groups, and the limits of their candidate counts. It distinguishes these searches from citation chasing and venue monitoring, which ran in parallel.

A Search Strings and Counts

  • The structured searches closed on 2026-09-11, used per-expression limits of 100 newest-first results, and yielded 1,115 unique candidates after de-duplication.The counts record candidates per group, those already found through citation chasing, and additions after abstract screening; two expressions reached the result limit.Source: A Search Strings and Counts · p. 99
  • The listed query groups cover music-generation topics including instrumental and song generation, control and editing, real-time systems, evaluation, provenance, and alignment.Other groups include surveys, codecs, instruments, cross-modal generation, and multi-agent composition.Source: A Search Strings and Counts · pp. 100–101
  • The appendix supplies additional search expressions for control, live generation, evaluation, provenance, alignment, instruments, and cross-modal or multi-agent topics.Table A.1 identifies topic groups and their start dates.Source: A Search Strings and Counts · p. 101
  • Figure A.1 counts works included from structured-search candidates, excluding parallel citation chasing and venue monitoring.Venue monitoring covered named conferences and journals.Source: Figure A.1 · p. 103

A.1 Screening and Sources

The appendix records how candidate papers were screened and how comparison-table attributes were sourced and checked. Its records preserve decisions and dates so later additions and artifact changes can be handled consistently.

A.1 Screening and Sources

  • Each search candidate was logged with identifying metadata and matching expressions, then screened by title and abstract into a reasoned decision: dedicated sentence, grouped citation, or exclusion.Candidates not added during collection were reread in October 2026; proposed additions were held for a later revision so the appendix counts describe one closed classification.Source: §A.1 · p. 103
  • Table A.2 lists coverage, search, and source-observation dates, while later systems can be added under the same record and sourcing rules.The appendix also identifies five proposed additions held for a later revision, including five that predate the eligibility cutoff.Source: §A.1 · pp. 103–104, Table A.2 · p. 104
  • Comparison tables use one record per system and typed attribute columns, with each value sourced preferentially from the paper, model card, repository, or product documentation.Artifact attributes and openness cells follow dated model-card or repository checks because releases and withdrawals can occur after papers; this check found two withdrawals.Source: §A.1 · p. 104
Loading 2610.38252v1…