PaperlayerView exact revision on arXiv

Source-linked AI summary

Ghost in the Encoder: Decodable Artist Identity Representations in Lyrics-to-Song Generation

Arhan Vohra, Choenden Kyirong, Laura Ibáñez-Martínez, Martín Rocamora

arXiv:2609.39552v1cs.SDcs.LG

TL;DR

Prior work has documented artist imitation and song replication in music generation, but their internal mechanisms remain poorly understood. This study probes artist identity from lyrics in ACE-Step 1.5 and traces the signal through generation. It finds that artist information persists into the diffusion backbone, while causal intervention remains unexplored.

  • Problem

    Mechanisms that may enable artist-level imitation and song-level replication in generative audio have not yet been dissected.

    Source: §2.1 · p. 2

  • Method

    The study uses linear probes on a 2,000-song, 100-artist case study of ACE-Step 1.5 to trace artist identity from lyrics through model activations.

    Source: §1 · p. 1, §3.3 · p. 3

  • Results

    Artist identity is decodable from lyrics without explicit artist information, persists through the DiT cross-attention interface into early audio-token hidden states, and is modulated by the CoT caption pathway.

    Source: §1 · p. 1, §5 · p. 6

  • Takeaways & Limitations

    Lyrics provide an artist-level conditioning channel that may be an interpretable form of leakage relevant to prompt-side replication exploits.

    Source: §5 · p. 6

    The probing study suggests possible explanations for replication patterns, but causal intervention via mechanisms remains to be explored.

    Source: §5 · p. 6

Abstract

from arXiv · show

Text-to-song generation models can be prompted to imitate specific artists or regurgitate entire songs from their training data. Although these phenomena have been documented behaviorally on small datasets, little is known about the internal representations that may give rise to them. Prior interpretability work on generative audio has focused on locating semantic concepts such as genre or time signature within model activations. In this work, we show that a trained model can be probed for linearly decodable representations of artist identity from song lyrics alone, without any additional identifiers. Through a controlled case study of ACE-Step 1.5 spanning 2,000 songs across 100 artists, we demonstrate that the artist associated with a given set of lyrics can be identified within the model's internal activations, and that this conditioning signal propagates from the lyric encoder to the diffusion backbone during inference. These findings indicate that lyrics constitute an artist-level conditioning channel not addressed by prompt-side replication safeguards. More broadly, our work highlights how latent-space analysis can be used to audit what generative music models have implicitly learned from their training data.

1. INTRODUCTION

Artist imitation, song memorization, and stylistic generalization all draw on the same training works, but the internal representations behind artist-level imitation remain poorly understood. This study probes artist identity from lyrics alone in ACE-Step 1.5 and traces the signal through generation.

1. INTRODUCTION

  • Music-generation research has documented song memorization and artist impersonation alongside concept generalization, but artist-level replication has largely been studied behaviorally on fewer than 50 songs.Prior accounts treat song generation as a black box, motivating a latent-space explanation.Source: §1 · p. 1
  • Using linear probing across the conditioning pipeline, the study examines artist-specific representations in ACE-Step 1.5 from lyrics alone, on a class-balanced dataset of 2,000 songs by 100 artists across five genres.Source: §1 · p. 1
  • The lyric encoder produces linearly decodable artist identity without explicit artist information, and the signal remains decodable through cross-attention, denoising, and generated audio.Artist identity is defined functionally as the artist label associated with a song’s lyrics, without assumptions about acoustic output.Source: §1 · pp. 1–2

2. RELATED WORK

Prior work documents memorization and artist imitation in generative systems, while mechanistic explanations for artist-level imitation and song replication remain undeveloped. This paper situates its study within lyrics-to-song architectures and earlier interpretability work on musical concepts.

2.1 Replication and Memorization in GenAI

  • Mechanisms that may enable artist-level imitation and song-level replication remain undissected despite broader work on attribution and anti-memorization interventions in generative audio.Earlier studies report music-model memorization and artist imitation, including lyric-triggered replication and descriptive artist prompts [1] [2].Source: §2.1 · p. 2

2.2 Lyrics-to-song Models

  • Lyrics-to-song systems route text through pretrained encoders into generative backbones, which may be autoregressive, diffusion-based, or hybrid.ACE-Step 1.5 combines a Qwen-based language model for structured Chain-of-Thought metadata with a Diffusion Transformer conditioned through caption, optional timbre, and lyric pathways.Source: §2.2 · p. 2

2.3 Interpretability of Music Generation Models

  • Prior music-model interpretability work probes or localizes musical concepts such as time signature, instrument presence, and genre, while this study examines text conditioning and artist identity.The study also asks whether this relationship can functionally explain imitation and replication reported in prior work.Source: §2.3 · p. 2

3. METHOD

The method uses a balanced lyrics dataset and activation probes to follow artist-identifying information through ACE-Step 1.5’s conditioning and generation pipeline. It also tests output-side artist decodability and song-level similarity to assess whether the two are related.

3.1 Dataset

  • The dataset contains top songs from 100 artists equally distributed across five genres, with artist-name mentions anonymized and post-2024 releases and song variants excluded.Songs are selected from Last.fm and Genius; because ACE-Step’s 1.8M-song training corpus is undocumented, training-set membership cannot be directly verified.Source: §3.1 · p. 2

3.2 Prompting and Activation Extraction

  • Hooks capture activations from lyric and caption streams, merged conditioning, cross-attention projections, and DiT audio-latent states across denoising steps.Inference compares no-CoT and with-CoT configurations; both use the same lyrics, while with-CoT derives a caption and infers BPM and key, with other inputs held fixed.Source: §3.2 · pp. 2–3, Table 1 · p. 3

3.3 Internal Probing

  • Internal probes average non-feature activation axes into feature vectors and train fold-specific linear multi-class logistic-regression classifiers using cross-entropy loss.This procedure measures how accurately artist identity can be decoded at each hook point.Source: §3.3 · p. 3

3.4 Output-side Probing

  • Output-side probing trains linear classifiers on LAION-CLAP and MuQ-MuLan song embeddings, while CLEWS-DVI compares generated audio with its corresponding original recording for song-level similarity.The comparison is used to examine whether artist decodability may be related to song-level replication.Source: §3.4 · p. 3

RQ1: Does the lyric encoder develop linearly decodable representations of artist identity?

The lyric encoder makes artist identity linearly decodable from lyrics without explicit artist information, and the signal remains after controlling for genre. Decodability increases across the encoder’s early layers before plateauing.

RQ1: Does the lyric encoder develop linearly decodable representations of artist identity?

  • 0.344 pooled-output accuracy identifies the source artist, about four times the TF-IDF bag-of-words baseline (0.085) and above the permutation-test null (p < 0.001).The probe was evaluated using held-out accuracy after 5-fold cross-validation.Source: RQ1: Does the lyric encoder develop… · p. 4
  • Artist decodability peaks at layer 5 (0.361), then stabilizes between 0.34–0.36 in layers 5–7, while Figure 1 shows the rise through the encoder layers.The Qwen3 input embeddings achieve 0.153 accuracy.Source: RQ1: Does the lyric encoder develop… · p. 4, Figure 1 · p. 4
  • Within-genre probes test whether artist identity decodability reflects individual artists rather than only genre vocabularies and songwriting norms.Genre itself is decodable from the lyric encoder output with accuracy 0.764.Source: RQ1: Does the lyric encoder develop… · p. 4
  • Mean artist decodability varies by genre, from 0.585 for hip-hop to 0.338 for country; TF-IDF is strongest in hip-hop at 0.403.In other genres, encoder probing is over twice as accurate as bag-of-words.Source: RQ1: Does the lyric encoder develop… · p. 4, Figure 2 · p. 4

RQ3: Does the DiT preserve artist-level decodability during generation?

Artist identity remains decodable as lyric conditioning passes through ACE-Step’s diffusion backbone, though decodability varies across layers and into generated audio. Genre-controlled probes also recover artist information from the generated audio.

RQ3: Does the DiT preserve artist-level decodability during generation?

  • Artist identity remains decodable at cross-attention K/V projections, averaging 0.292, a 15% decrease from 0.344 at the lyric encoder; artist-level decodability is highly correlated between the two probes (ρ ≈0.90, r ≈0.91).Figure 3 depicts decodable representations entering early DiT audio-token layers and becoming less decodable in deeper layers.Source: RQ3: Does the DiT preserve… · p. 4, Figure 3 · p. 5
  • Audio-token hidden states start at chance accuracy of 0.01 before denoising, rise sharply in early layers, and fall to 0.10–0.14 in layers 12–13 with gradual decay over time steps.The generator receives no style caption or metadata in the described no-CoT setting.Source: RQ3: Does the DiT preserve… · p. 4
  • MuQ-MuLan probes achieve mean artist accuracy of 0.109 on generated audio under no-CoT conditioning, compared with 0.449 on original recordings; CLAP-based accuracy is 0.043.The embedding probes use generated audio rather than the input lyrics.Source: RQ3: Does the DiT preserve… · p. 5
  • Genre-controlled MuQ-MuLan probes retain artist signal in all five genres, with accuracy from 0.128 for pop to 0.208 for rock against 0.05 chance.The audio-level genre ranking partially diverges from the encoder: rock artists are more decodable than hip-hop artists on average.Source: RQ3: Does the DiT preserve… · p. 5
  • The authors conclude that artist-specific conditioning persists from lyrics through internal representations to generated audio and is not only song-specific memorization.They note that causal evaluation would be needed to distinguish whether early DiT layers re-encode artist information, develop nonlinear representations, or discard it.Source: RQ3: Does the DiT preserve… · pp. 4–5

resentations in the generation pipeline?

CoT captioning changes artist decodability across the conditioning pipeline, while generated-audio probe accuracy and cover retrieval point in different directions. The authors treat this dissociation as a possible song-recognition trigger, not a settled mechanism.

CoT caption pathway

  • 0.008 no-CoT versus 0.069 with-CoT caption-embedding accuracy; with-CoT remains below the 0.085 TF-IDF lyric-classifier baseline.A TF-IDF classifier trained on LM-generated captions reaches 0.073, comparable to one trained on lyrics.Source: resentations in the generation pipeline? · p. 5
  • 0.344 lyric-encoder decodability falls to 0.085 at the merge point and from 0.292 to 0.089 in K/V projections.The authors caution that mean-pooling may dilute the lyric stream’s per-token signal rather than remove artist information.Source: resentations in the generation pipeline? · p. 5
  • With-CoT audio is harder to probe for artist identity, yet more likely to be retrieved as a cover of its original recording.Table 2 reports artist-probe accuracy on generated-audio embeddings; Table 3 reports CLEWS CoverID Retrieval@K.Source: resentations in the generation pipeline? · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6
  • Informal listening suggests with-CoT outputs retain more musical and structural resemblance to originals while differing in vocal timbre, gender, or instrumentation.The authors suggest auto-generated captions may contribute to these differences.Source: resentations in the generation pipeline? · p. 6
  • The authors suggest LM captions may act as song-recognition triggers that surface training-data reproductions, but leave this pathway for future study.This interpretation draws on the dissociation between artist-probe accuracy, cover retrieval, and exploratory listening observations.Source: resentations in the generation pipeline? · p. 6

5. CONCLUSION

ACE-Step 1.5 contains linearly decodable artist-identity representations that persist from the lyric encoder through cross-attention into early audio-token hidden states. The study frames latent-space analysis as a way to investigate learned artist representations while noting that causal intervention remains unexplored.

5. CONCLUSION

  • Artist-identity representations persist through the DiT cross-attention interface into early audio-token hidden states.The study reports that the signal can be retrieved without explicit artist information.Source: §5 · p. 6
  • The authors report that the signal is acquired during pretraining rather than inherited from the input encoder, using lexical and transformer-based baselines.They also find that the signal is modulated by ACE-Step’s CoT caption pathway.Source: §5 · p. 6
  • Whether causal intervention can alter these representations remains unexplored.The authors suggest extending probing to other models and architectures, alongside perceptual evaluation.Source: §5 · p. 6
  • The authors frame the identified conditioning channel as a possible interpretable form of leakage related to artist imitation and replication exploits.They connect this possibility to prior reported conditioning exploits that could bypass replication safeguards [1].Source: §5 · p. 6

Related papers

Loading 2609.39552v1…