PaperlayerView exact revision on arXiv

Source-linked AI summary

Tracing Inputs, Verifying Outputs: Validating Attribution in Music Generation

Taejun Kim, Wonil Kim, Jongmin Jung, Hyeongseok Wi, Sangeun Kum, Keunhyoung Luke Kim, Taehyoung Kim, Dongjoo Moon, Seungsoon Park, Taewan Kim, Virginie Berger, Juhan Nam, Jongpil Lee

arXiv:2610.09637v1cs.SDcs.LGeess.AS

TL;DR

Attributing AI-generated music requires evidence of both which audio sources were used and whether they shaped the output. This paper records audio inputs and tests their musical effects, while separately auditing for training reproductions. MixAudio reflects intended input characteristics, and musicDNA finds few outside-record reproductions while outperforming other tested detectors on human-judged flagged cases.

  • Problem

    Attribution requires separate evidence that a track was used as input and that its musical characteristics shaped the output; resemblance or input use alone cannot establish both.

    Source: §1 · p. 1

  • Method

    The paper records known-provenance audio inputs and tests their musical effects, then uses musicDNA to compare generated stems with training data for unsupplied reproductions.

    Source: §1 · p. 1

  • Results

    MixAudio outputs reflected intended input characteristics, outside-record reproductions were rare, and musicDNA achieved higher precision and recall than other tested methods on human-judged flagged cases.

    Source: §6 · p. 8

  • Takeaways & Limitations

    Input records and output analysis provide complementary evidence on which rights-holder reporting and compensation can draw.

    Source: §6 · p. 8

    The audit is specific to this model and catalog, examines only flagged stems, and covers melody-carrying instruments rather than bass or drums.

    Source: §5 · p. 8

Abstract

from arXiv · show

How can we verify whose music contributed to an AI-generated output? This paper demonstrates how input-based attribution can provide verifiable evidence of which audio sources were used in a generation and whether they shaped the output. To do so, we condition the generation solely on audio without any text input, then trace the inputs behind each output, and establish their musical effect. In prompt adherence tests and controlled input swaps, the stems generated by our generator, MixAudio, follow the prompt audio in timbre and the context audio in harmony. Yet these outputs may still reproduce training data not supplied as inputs. We therefore audit memorization with our musical version identification model, musicDNA, and find few reproductions outside the input records. On human-judged cases within the flagged pool, it achieves higher precision and recall than the other tested memorization detectors. The two evaluations suggest that input records and output analysis provide complementary evidence for attribution, on which rights-holder reporting and compensation can draw as the AI music economy takes shape. Audio examples are available at https://neutune.github.io/attr2027demo/

1. Introduction

The paper defines attribution as identifying the audio sources used and verifying whether their musical characteristics shaped the output. It combines input records with musical-effect tests and separately audits for reproductions of training music not supplied as input.

1. Introduction

  • The paper defines attribution as identifying the audio sources used as inputs and verifying whether their musical characteristics shaped the generated output.With clear scope, such evidence can ground recognition of a source’s contribution and inform rights-holder reporting and compensation.Source: §1 · p. 1
  • Attribution requires evidence both that a track was used as input and that its musical characteristics shaped the output; resemblance or confirmed input use alone cannot establish both.Rhythm, harmony, or playing style may also occur in other training tracks.Source: §1 · p. 1
  • Faithful conditioning does not rule out reproducing an unsupplied training track, so outputs must be compared with training data separately.Input records alone cannot account for the relationship between such a track and the output.Source: §1 · p. 1
  • MixAudio records known-provenance audio inputs, while musicDNA compares generated stems with training data for potential reproductions not supplied as inputs.The input-use record and the training-data comparison address distinct evidence needs.Source: §1 · p. 1
  • MixAudio’s stems reflected intended input characteristics; outside-record reproductions were rare, and musicDNA had higher precision and recall than other tested methods on human-judged flagged cases.The findings support using input records and output analysis as complementary evidence for attribution.Source: §1 · pp. 1–2

2. Background and Related Work

The background distinguishes input records, output resemblance, and training-data influence as different kinds of attribution evidence. It motivates testing audio-conditioning effects, auditing for memorization, and recording input use at the stem level.

2.1. Approaches to Attribution

2.2. Audio Conditioning and Faithfulness

  • MixAudio treats each stem’s audio prompt as guidance for timbre and playing style, while shared existing-stem context supplies harmonic and rhythmic context.Because similarity alone is insufficient, the evaluation compares a stem with its own and other prompts and swaps one conditioning input while holding the other fixed.Source: §2.2 · p. 2

2.3. Memorization Audits

  • Memorization audits use different detection methods, so a finding of no detected reproduction must be interpreted in light of what each method can detect.The paper compares methods on human-confirmed candidates to assess audit reliability.Source: §2.3 · p. 2

2.4. Stem-Level Input Records

  • Stem-level records connect each reference recording to its generated stem and link shared context to the track section it conditions.Examining only the full mix makes it difficult to identify which input was used for each stem or which characteristics appeared in the result.Source: §2.4 · p. 3

3. The Generator

MixAudio generates individual stems from registered audio, with an optional shared context, and records stem-level relationships between prompts and targets. Its conditioning design supports new-song and remix generation, while its rights-accounting choices are tied to evaluated model behavior.

3. The Generator

  • Users can generate a song from prompt audio alone or add, replace, or remove stems in an existing track using a shared context.The generator produces individual stems rather than a mixture, and the context is used for remix generation.Source: §3 · p. 3
  • For rights accounting, prompt use is associated with recording-side contribution and context use with composition-side contribution; this design rests on evaluated behavior, not copyright law.When context is mixed into the final master, the record also names the context’s recording rights holder.Source: §3 · p. 3
  • The generator predicts target-stem tokens from a sequence containing prompt–target pairs, optional prepended context, and key and beat inputs.Stem-level positional encoding links each prompt to its target; shared positional encoding aligns context and generated stems.Source: §3 · p. 3
  • Table 1 defines Matched and Random prompt comparisons and reports their difference, ∆=Matched−Random; Table 2 compares Base and Swap generations with the other conditioning input held fixed.The Table 2 values are averaged over four dataset sources; the supplied scale anchors are MFCC 0.66/0.92 and Chroma 0.02/0.20 for unrelated-pair/matched-reference comparisons.Source: Table 1 · p. 4, Table 2 · p. 4, §3 · p. 4

4. Evaluation

The evaluation tests whether recorded audio inputs shape generated stems and whether outputs reproduce training material absent from the input record. MixAudio follows prompt timbre and context harmony, while musicDNA performs best among tested detectors on human-confirmed cases in the flagged pool.

4. Evaluation

  • The audit tests whether stems follow their recorded inputs and whether they reproduce training stems whose rights holders are absent from the record.It uses conditioning experiments, a memorization audit, and listener judgments to evaluate detectors; the retrieval audit’s conclusion is limited to its scope.Source: §4 · p. 4, §4.3 · pp. 6–7

4.1. Setup

  • The setup uses in-domain training and validation audio and unseen external datasets; ACE-Step v1.5 in Lego mode is the comparison system.Both systems generate the same instances from the same prompts and contexts; instances are track sections, and some have no context.Source: §4.1 · pp. 4–5

4.2. Conditioning Faithfulness

  • MixAudio has a larger Matched-minus-Random prompt-adherence margin than ACE for every source and both style and timbre metrics.The margin discounts generic resemblance; ACE’s Random style scores are high because its outputs resemble one another.Source: §4.2 · p. 5
  • In conditioning swaps, MixAudio follows prompt timbre and context harmony independently, while ACE follows prompt timbre but shows no context-harmony response.MixAudio’s chroma correlation to the new context rises from 0.00 to 0.21 and to the old context falls from 0.21 to 0.02; pitch-shifting or time-scaling gives the same pattern.Source: §4.2 · pp. 5–6

4.3. Memorization Audit

  • The audit retrieves training-set matches for generated melody-carrying stems, excludes a training query’s own conditioning stems, and has listeners categorize every flagged stem.Thresholds are set at the 25th-highest validation score, flagging the top 5% of validation queries, and are then applied to training queries.Source: §4.3 · pp. 6–7
  • The audit found one unnamed material reproduction, a Type I training-split case, and none in validation; no Type II or minor D case appeared.Of 32 confirmed memorizations, 31 were duplicates, which the authors attribute to dataset curation rather than the model.Source: §4.3 · pp. 6–7

4.4. Evaluating Memorization Detectors

  • Detector evaluation scores six methods on 171 stems flagged by any retriever, including 32 confirmed memorizations and 139 non-memorizations.Recall is measured only against confirmed memorizations in this flagged pool, so cases missed by every retriever are not counted.Source: §4.4 · pp. 7–8
  • musicDNA flags 29 of 32 confirmed memorizations (recall 0.91 versus CLEWS 0.69) and has precision 0.49 versus CLEWS 0.37 on the judged pool.CLAP finds one confirmed case, while the MusicGen-style test flags four unrelated stems and the MusicLM-style test recovers one confirmed case while flagging 33 unrelated stems.Source: Figure 3 · p. 7, §4.4 · p. 8

5. Discussion and Limitations

MixAudio uses audio-only conditioning, and the paper does not test its quality against text-conditioned generation. The memorization audit is limited to this model and catalog, flagged melody-carrying stems, and a catalog without vocal targets or prompts.

No text conditioning

  • Audio-only conditioning cannot serve as a text-to-music system, and the paper does not test its quality against text-conditioned generation.Text can still enter indirectly through vector retrieval of a registered prompt or context recording; generation quality is higher than the closest audio-prompt baseline.Source: §5 · p. 8

Scope of the memorization audit

  • The memorization finding—that unrecorded training-stem reproductions are rare—applies to this model on this catalog, not to the architecture generally.The comparison’s reference labels were set by three author-listeners.Source: §5 · p. 8
  • The audit covers only melody-carrying stems and finds no composition-level copying, so its detector comparison does not test copying in other timbres.Vocal stems are neither targets nor prompts; the authors expect vocal targets and lyric prompts could produce composition-level cases.Source: §5 · p. 8
  • Audit counts are lower bounds because stems below every retriever’s threshold are not examined.A different catalog needs its own audit because memorization depends on the model and its training data.Source: §5 · p. 8

6. Conclusion

The paper frames attribution as identifying which audio sources were used and verifying whether their musical characteristics shaped the output. It evaluates input records alongside output analysis and finds these provide complementary evidence for rights-holder reporting and compensation.

6. Conclusion

  • Attribution requires identifying the music used and how its musical characteristics shaped the output, with reliable evidence central to rights-holder compensation.The paper examines what different approaches can establish and empirically validates input-based attribution.Source: §6 · p. 8
  • MixAudio outputs reflected intended input characteristics, while musicDNA found few unrecorded reproductions and had higher precision and recall than tested methods on human-judged flagged cases.The evaluation considers both whether recorded inputs shape outputs as intended and whether outputs reproduce training tracks absent from the records.Source: §6 · p. 8
  • Input records and output analysis provide complementary evidence that rights-holder reporting and compensation can draw on.The paper evaluates whether recorded inputs shape outputs and whether outputs reproduce training tracks absent from the records.Source: §6 · p. 8

Impact Statement

The paper aims to advance machine learning and says its attribution framework can help direct revenue to rights holders, while cautioning that records are imperfect and consequential decisions need human review.

Impact Statement

  • The proposed framework can help direct revenue from generated music to its rightful rights holders.The paper states that its goal is to advance machine learning.Source: Impact Statement · p. 8
  • Because the records are not perfect, the authors say they should not be trusted without human review for consequential decisions.The paper does not specifically highlight other societal consequences.Source: Impact Statement · p. 8
  • The authors state a goal of advancing machine learning but identify no societal consequences requiring specific emphasis.Source: Impact Statement · p. 8

A. Evaluation Setup Details

The evaluation uses track sections as instances, randomly assigns context and target stems, and gives each target a short excerpt from its own recording as a prompt. It specifies a runnable audio-prompt baseline and measures prompt adherence and conditioning swaps with audio similarity metrics.

A. Evaluation Setup Details

  • Instances are track sections, with short and long sections excluded; the training sample is stratified to match validation distributions, while external catalogs are used in full.The validation split is used in full, and sections are snapped to their first and last downbeat.Source: A. Evaluation Setup Details · p. 11
  • Each section’s stems are randomly divided into context and targets, and each target receives a 1–2 s prompt excerpt from its own recording.Sections with too few stems fall back to a smaller context size, making the realized contextless share exceed 0.15.Source: A. Evaluation Setup Details · p. 11
  • Each instance is generated once with a fixed seed; ACE-Step generates stems sequentially, and stems below the frame-energy retention threshold are dropped from both systems’ comparisons.The retained-stem rule requires at least 15% of 100 ms frames to exceed −60 dBFS.Source: A. Evaluation Setup Details · p. 11
  • ACE-Step v1.5 in Lego mode is the only considered runnable baseline that takes per-stem audio prompts and context, conditions on prompt audio, and generates individual stems.Table 5 lists candidate research systems with released weights and commercial services.Source: A. Evaluation Setup Details · p. 11, Table 5 · p. 11

B.1. Measures

  • Prompt adherence compares each generated stem with its matched prompt and a random other-track prompt, using centered MERT cosine for style and MFCC cosine for timbre.The random style-prompt column serves as a zero point after centering against 1,471 random training segments.Source: §B.1 · p. 12
  • Conditioning swaps use MFCC similarity for timbre and chroma correlation for harmony, comparing generated stems with matched versus unrelated prompt or context audio.Harmony is measured as Pearson correlation between 12-bin chromagrams averaged per bar on the model’s beat grid.Source: §B.1 · p. 12

B.2. Tempo and Pitch Transformations

Controlled transformations test whether generated stems preserve prompt characteristics while following changes to the context. MixAudio follows context tempo and pitch changes more closely than ACE.

B.2. Tempo and Pitch Transformations

  • 83% of units followed context time-scaling for MixAudio, versus 58% for ACE; chroma accuracy was 76% for MixAudio and 33% for ACE after context transposition.The corresponding ceiling was 90% for time-scaling; for transposition, the ceiling was 86% and the untransposed floor was 22%.Source: Table 6 · p. 12, §B.2 · p. 13
  • The test changes one conditioning input at a time while holding the other fixed, using time-scaling or pitch shifts to test the expected following behavior.Table 6 defines the oracle as the value perfect following would yield.Source: Table 6 · p. 12, §B.2 · p. 12
  • Tempo accuracy assigns each unit to the nearer base or transformation-following tempo candidate on a log scale; chroma accuracy tests whether tonal stems follow a context transposition.Chroma change is measured against the original context on the summed generated stems at beat level; timbre change compares regenerated stems with the original prompt.Source: §B.2 · pp. 12–13
  • Under prompt transformations, both systems kept tempo, harmony, and timbre nearly unchanged; tempo stayed at the base generation in 95% of MixAudio units and 91% of ACE units.Chroma and timbre changes stayed within 0.03.Source: §B.2 · p. 13

C.1. Audit Protocols in Prior Work

The audit compares retrieval-based memorization detectors, then uses listener judgments to assess flagged matches. The paper also reports how listener ratings and FAD are summarized for generation quality.

C.1. Audit Protocols in Prior Work

  • Token-based detectors generally require near-verbatim reuse, while general-purpose audio embeddings such as CLAP are dominated by timbre; Jukebox uses listening and YuE uses cover-song identification.The comparison concerns six memorization audits discussed in the paper.Source: §C.1 · p. 13, Table 7 · p. 13

C.2. Retrieval and Flagging

  • Retrievers compare generated and training stems in windows: musicDNA and CLEWS use minimum squared Euclidean distance, while CLAP and MERT use maximum centered cosine similarity.The search spans training stems of every instrument; training queries exclude their own track by track identifier.Source: §C.2 · p. 13

C.3. Listening Protocol

  • Three listeners judged each flagged stem-match; 171 flagged stems yielded 213 matches because different retrievers could return different training stems.The listeners were paper authors, and each match was heard with the full stem from which the prompt was cut.Source: §C.3 · p. 14
  • Listeners agreed on whether a match was a memorization in 191 of 213 cases (90%, κ = 0.82); majority vote settled categories, with joint listening for five cases where all three differed.They gave the same category on 178 matches (84%, Fleiss’ κ = 0.75).Source: §C.3 · p. 14
  • Borderline judgments distinguished shared note sequences, commonplace material, and duplicate recordings, with the commonplace test stricter because listeners judged a stem rather than a whole piece.A duplicate had to sound like the same recording as the prompt’s full stem in timbre and nearly every note.Source: §C.3 · p. 14

D. Generation Quality

  • Figure 4 reports mean opinion scores for all 48 cases with 95% confidence intervals; Table 8 reports FAD for generated stems and listener ratings for generated and same-case real stems.Ratings are 1–5 mean opinion scores with 95% confidence intervals; Quality, Prompt, and Context denote sound quality, prompt-audio match, and context fit, with higher better; FAD lower is better.Source: Table 8 · p. 14, Figure 4 · p. 15

D.1. Objective Evaluation

Objective generation quality is measured with FAD on individual stems against held-out real stems. MixAudio has lower FAD than ACE on each of three unseen conditioning sources.

D.1. Objective Evaluation

  • MixAudio has lower FAD than ACE on all three unseen conditioning sources, including the two external catalogs.The All row averages the three sources.Source: §D.1 · p. 15
  • FAD compares generated-stem embedding distributions with held-out real stems, using track-disjoint evaluation and reference sets from validation, MoisesDB, and MUSDB18.FAD is computed on individual stems rather than mixes.Source: §D.1 · p. 15

D.2. Subjective Evaluation

In listener ratings, MixAudio scores above ACE on sound quality, prompt match, and context fit. Its advantage over ACE is largest on context fit, where it also comes closest to the real stems.

D.2. Subjective Evaluation

  • The test covers 48 cases, with 16 each from validation, MoisesDB, and MUSDB18; both systems receive the same prompt and context inputs.Each case includes a real stem and one generated stem from each system; playback is limited to the first 20 seconds.Source: §D.2 · p. 15
  • Seventeen listeners rated candidates for sound quality alone, prompt match alone, and fit with the context when mixed.Each candidate could be played alone and with the context; ratings used a 1–5 scale.Source: §D.2 · p. 15
  • MixAudio is rated above ACE on all three axes, and their 95% confidence intervals do not overlap on any axis across all cases.The context-fit gap between MixAudio and the real stems is smallest, while its gap over ACE is largest.Source: §D.2 · p. 15
Loading 2610.09637v1…