PaperlayerView exact revision on arXiv

Source-linked AI summary

Steering Autoregressive Music Generation with Recursive Feature Machines

Daniel Zhao, Daniel Beaglehole, Taylor Berg-Kirkpatrick, Julian McAuley, Zachary Novack

arXiv:2510.19127v2cs.LGcs.AIcs.SDeess.AS

TL;DR

Fine-grained control of music-theoretic content across time remains challenging, especially without retraining or costly inference-time optimization. MusicRFM steers frozen MusicGen using RFM-derived activation directions and layer-, time-, and multi-direction mechanisms; moderate steering improves target alignment with limited drift and minimal prompt-adherence degradation.

  • Problem

    Fine-grained music-theoretic control across time remains challenging, while existing approaches may require intensive fine-tuning or costly per-step inference optimization.

    Source: §1 · p. 1

  • Method

    MusicRFM trains lightweight RFM probes to identify concept directions in MusicGen hidden states and injects them into activations at inference, with layer-, time-, and multi-direction controls.

    Source: §1 · pp. 1–2

  • Results

    Across synthetic and real-music settings, moderate steering improves target alignment with limited drift and minimal prompt-adherence degradation; note accuracy rises from 0.23 at η0=0.15 to 0.82 at η0=0.60.

    Source: §5.2 · p. 6, §8 · p. 10

  • Takeaways & Limitations

    MusicRFM enables fine-grained, interpretable control of notes, chords, and tempo without modifying the base model or using per-step optimization.

    Source: §8 · p. 10

    Longer generations can exhibit unwanted artifacts, and the authors identify reducing dependence on previously generated tokens as a future direction.

    Source: §7 · p. 10

Abstract

from arXiv · show

Controllable music generation remains a significant challenge, with existing methods often requiring model retraining or introducing audible artifacts. We introduce MusicRFM, a framework that adapts Recursive Feature Machines (RFMs) to enable fine-grained, interpretable control over frozen, pre-trained music models by directly steering their internal activations. RFMs analyze a model's internal gradients to produce interpretable "concept directions", or specific axes in the activation space that correspond to musical attributes like notes or chords. We first train lightweight RFM probes to discover these directions within MusicGen's hidden states; then, during inference, we inject them back into the model to guide the generation process in real-time without per-step optimization. We present advanced mechanisms for this control, including dynamic, time-varying schedules and methods for the simultaneous enforcement of multiple musical properties. Our method successfully navigates the trade-off between control and generation quality: we can increase the accuracy of generating a target musical note from 0.23 to 0.82, while text prompt adherence remains within approximately 0.02 of the unsteered baseline, demonstrating effective control with minimal impact on prompt fidelity. We release code to encourage further exploration on RFMs in the music domain.

1 INTRODUCTION

Fine-grained control of music-theoretic content across time remains challenging, and existing approaches may require intensive fine-tuning or costly inference-time optimization. MusicRFM steers a frozen MusicGen model using RFM-derived concept directions, with layer-, time-, and multi-direction control mechanisms.

1 INTRODUCTION

  • Fine-grained control of music-theoretic content such as pitch classes and chord qualities across time remains challenging, despite work on time-varying controls.Existing approaches may focus on broad controls or require intensive fine-tuning or costly per-step inference optimization.Source: §1 · p. 1
  • MusicRFM identifies interpretable directions in hidden states and steers generation along them without retraining the base model or changing its decoding procedure.The directions are intended to correspond to concepts such as pitches, chord qualities, or tempo.Source: §1 · p. 1
  • MusicRFM raises target-note classification accuracy from 0.23 to over 0.82 while CLAP text alignment remains within approximately 0.02 of the unsteered baseline.The authors describe this as a favorable trade-off between control and fidelity.Source: §1 · p. 2
  • Lightweight, layer-wise probes discover concept directions that are injected into MusicGen’s residual stream at inference for fine-grained control.The framework also uses layer-based selection or weighting, time-varying schedules, and simultaneous or staggered steering of multiple attributes.Source: §1 · p. 2

2 RELATED WORK

MusicRFM adapts RFM-based activation steering to autoregressive music generation, adding layer-, time-, and multi-direction control mechanisms. This extends prior work on controllable generation and activation-level steering toward fine-grained music-theoretic control.

2 RELATED WORK

  • MusicRFM combines activation-level steering with additional temporal and architectural controls for music generation.The work unifies threads from activation-level steering, decoding-time control, and controllable music generation.Source: §2 · p. 2, §2.3 · p. 3

2.1 CONTROLLABLE MUSIC AND AUDIO GENERATION

  • Prior autoregressive music controls often require compute-heavy fine-tuning, while existing approaches address multimodal or common musical controls such as piano rolls.Some parameter-efficient methods require tens to hundreds of GPU hours for each added control.Source: §2.1 · p. 2

2.2 ACTIVATION-LEVEL STEERING IN GENERATIVE MODELS

  • Music activation-steering work has addressed binary or broad concepts, leaving time-varying music-theoretic control an open question.The cited examples focus on binary controls or broad concepts such as instrument presence.Source: §2.2 · p. 2

2.3 RECURSIVE FEATURE MACHINES (RFMS)

  • MusicRFM extends RFM-derived steering to autoregressive music with top-K or exponential layer weighting, dynamic schedules, and simultaneous or staggered concept directions.The passage identifies these as three innovations over prior RFM steering in language models.Source: §2.3 · p. 3

3 METHODS

MusicRFM trains lightweight RFM probes on MusicGen activations and injects the resulting concept directions during inference. Its audio-domain design uses sequence-level features and layer-, time-, and multi-direction controls to improve steering robustness.

3 METHODS

  • The method trains lightweight RFM probes to extract concept-aligned directions and reinjects them into MusicGen activations at inference for controlled generation under text conditioning.The stated target concepts include specific notes, chord types, and slow or fast tempo.Source: §3 · p. 3

3.1 BACKGROUND ON RECURSIVE FEATURE MACHINES

  • RFMs iteratively train a predictor, compute an Average Gradient Outer Product, and update features to identify task-sensitive directions without backpropagation.The AGOP is positive semidefinite, and its orthonormal eigenvectors have eigenvalues that measure sensitivity.Source: §3.1 · p. 3

3.2 MUSICRFM: RFM STEERING FOR MUSIC GENERATION

  • The pipeline encodes audio with EnCodec, trains layer-wise probes to obtain AGOP eigendirections, and applies steering during inference.SYNTHEORY supplies fine-grained labels across seven music-theory categories, and mean-pooling over tokens improves probe performance by capturing temporal structure.Source: §3.2 · pp. 3–4

3.3 IMPROVING ROBUSTNESS IN AUDIO-DOMAIN STEERING

  • Audio-domain adaptations address the mismatch between text’s discrete, variable sampling and music’s continuous, fixed sampling; these modifications apply only at inference.The authors found some algorithmic choices from text steering ill-suited to text-to-music generation.Source: §3.3 · p. 4

3.3.1 LAYER PRUNING

  • Uniform steering across all 48 layers can degrade audio quality and text alignment, so layer pruning prioritizes informative layers and downweights noisy ones.Top-K selection ranks layers by validation probe AUC, while exponential weighting continuously scales steering by normalized probe scores.Source: §3.3.1 · p. 4

3.3.2 TIME-CONTROL SCHEDULES

  • Time-control schedules vary steering strength over generation, using deterministic rises, decays, or sinusoidal patterns and optional Bernoulli gating.The stochastic gate applies control with probability p, reducing over-steering and cumulative artifacts while retaining an expected bias toward the target.Source: §3.3.2 · p. 4

3.3.3 MULTI-DIRECTION AND STAGGERED CONTROL

  • Each concept direction can have its own coefficient and schedule, allowing multiple attributes to be enforced simultaneously or activated at different times.For example, tempo can be emphasized in the opening while harmonic structure ramps in later.Source: §3.3.3 · pp. 4–5

4 CLASSIFICATION RESULTS

MusicRFM’s multiclass probes perform better or comparably to the SYNTHEORY FFN probes across categories, with gains on several categories and from mean-pooling over last-token activations.

4 CLASSIFICATION RESULTS

  • RFM probes perform better or comparably to SYNTHEORY’s 2-layer FFN probes across categories, beating them in accuracy on scales, progressions, and intervals and in tempo R².Mean-pooled RFMs outperform last-token activations, particularly for temporally structured attributes such as tempo, progressions, and scales.Source: §4 · p. 5
  • RFM-derived eigenvectors provide orthogonal, eigenvalue-ranked axes of sensitivity that can be injected into activations for steering.The authors contrast these axes with FFNs, which they argue do not naturally yield such directions.Source: §4 · p. 5
  • Table 1 compares base SYNTHEORY FFN, linear probes, last-token RFMs, and MusicRFM, reporting tempo R² and accuracy for other categories.The caption notes that logistic-probe results are omitted where probes fail to converge.Source: Table 1 · p. 5

5 SINGLE-DIRECTION MUSICRFM STEERING RESULTS

The section evaluates single-direction steering through quantitative, external, listening, and real-music tests. Results show increased concept control with stronger steering, alongside trade-offs in distributional fidelity and prompt adherence.

5.2 BINARY-PROBE QUANTITATIVE STEERING RESULTS

  • Probe accuracy rises from 0.23 at η0=0.15 to 0.82 at η0=0.60, while CLAP alignment remains essentially flat across steering strengths.FD and MMD increase as steering strengthens; the authors identify moderate η0 values as a balance between concept control and distributional fidelity.Source: Table 2 · p. 6, §5.2 · p. 6

5.4 LISTENING TEST AND AUDIO SAMPLES

  • In a 12-participant listening test, both RFM settings improved perceived control over baseline, with optimized steering earning the highest ratings across attributes.Chord and interval control benefited most; tempo showed the largest relative gain over the no-steering baseline.Source: Table 3 · p. 6, §5.4 · p. 7

5.2 BINARY-PROBE QUANTITATIVE STEERING RESULTS

  • Prompt-only conditioning is near chance except for notes, while combined prompting and RFM typically yields the strongest results and control unavailable from prompting alone.For notes, prompting exceeds 95% accuracy at higher η0.Source: §5.2 · pp. 6–7

5.3 EXTERNAL EVALUATION METRICS FOR MUSICAL CONTROL

  • External evaluators find RFM improves notes and chord accuracy over prompt-only as η0 rises; tempo steering correlates with η0 (Spearman 0.283), increasing to 0.433 with prompting.Combining methods improves note accuracy but degrades chord performance, which the authors suggest may reflect low prompt-only chord accuracy.Source: §5.3 · p. 7, Table 4 · p. 7

5.5 EVALUATION ON MUSICBENCH (REAL MUSIC)

  • On MUSICBENCH, probes reach 75.3% notes and 67.5% keys accuracy but struggle on tempo (MSE 0.862); steering sensitivity varies by concept.Higher η0 raises FD/MMD and reduces CLAP, while moderate control preserves text adherence better than aggressive steering.Source: Table 5 · p. 8, §5.5 · p. 8

6 MULTI-DIRECTION AND TIME-BASED STEERING RESULTS

The section tests pairwise control of notes, chords, and intervals alongside time-varying note steering. Stronger multi-direction control improves note accuracy but worsens distributional metrics and prompt alignment; smooth schedules and crossfades produce the expected changes in probe scores.

6.1 MULTI-DIRECTION STEERING: PAIRWISE CROSS-CATEGORY CONTROL

  • Pairwise tests sample target classes from each of notes, chords, and intervals, then steer both concepts simultaneously at independently chosen coefficients.The experiments evaluate all four combinations of η0 = 0.3 and 0.6 for each pair, generating 1200 samples in total.Source: §6.1 · p. 8
  • Notes accuracy rises from 0.770 at η0 = 0.3 to 0.920 at η0 = 0.6 in pairwise steering.Table 6 reports averages over 200 generations per cell.Source: Table 6 · p. 8, §6.1 · p. 8
  • Higher steering strengths increase FD and MMD and significantly degrade CLAP alignment; multi-direction probe accuracy exceeds the single-direction setting.The authors hypothesize that lower prompt adherence compresses the generative manifold and reduces stylistic variance, making classes easier for probes to detect.Source: §6.1 · p. 8

6.2 TIME-DEPENDENT CONTROL: SMOOTH SCHEDULES

  • Probe softmax scores follow the prescribed exponential, linear, logistic, and sine schedules over generation time.The experiments analyze notes, selected for control quality and measurable probe accuracy.Source: §6.2 · p. 9
  • For crossfades, FD is 0.350, MMD is 1.922, and CLAP alignment is 0.250 across 500 randomly sampled note pairs.The setup decays the first note’s steering from η0 = 0.45 to 0 while raising the second from 0 to 0.45 over 0–1500 steps; probe scores show the first note falling and the second rising.Source: §6.2 · p. 9, Figure 1 · p. 9

7 LIMITATIONS & FUTURE WORK

The authors identify limits in temporal representation, the range of tested concepts and data, and model coverage, alongside a possible change to reduce autoregressive-context dependence.

7 LIMITATIONS & FUTURE WORK

  • Mean-pooled probe features discard temporal ordering, limiting control of sequential concepts such as scales, chord progressions, and time signatures relative to notes or chords.The authors have not yet quantified how much information higher-order eigenvector components retain.Source: §7 · p. 10
  • Autoregressive context can make the model over-dependent on previously generated tokens, reducing sensitivity to injected steering signals.Selective context masking is proposed as a possible way to address this.Source: §7 · p. 10
  • Experiments are limited to SYNTHEORY-based symbolic music-theoretic concepts; real-music training and perceptual attributes such as timbre remain future directions.The passage also identifies instrument identity and articulation style as possible extensions.Source: §7 · p. 10
  • Experiments target MUSICGEN-Large; applying RFMs to other audio models would require adapting probe extraction to their architectures.The authors cite multi-level codebooks and low-latency streaming as examples of model-specific adaptation needs.Source: §7 · p. 10

8 CONCLUSION

MusicRFM steers a frozen MUSICGEN-large model with RFM-derived directions, layer-aware weighting, and time-dependent schedules. The conclusion reports a trade-off between stronger concept control and generation quality, with notes most reliably controlled.

8 CONCLUSION

  • Moderate steering improves targeted-concept alignment with limited FD/MMD drift and minimal CLAP degradation, while aggressive steering strengthens control at the cost of artifacts.Notes are the most reliably controllable; multi-direction steering is feasible but amplifies drift, and simple schedules support intuitive manipulations such as crossfades.Source: §8 · p. 10

A OVERVIEW OF KERNEL RIDGE REGRESSION

  • Kernel ridge regression is the base learner used for iterative RFM feature learning via Average Gradient Outer Product matrices.The paper notes that its closed-form solution can be robustly computed and generally trains quickly on datasets under 70k samples.Source: A OVERVIEW OF KERNEL RIDGE REGRESSION · p. 13

B TUNING PROCEDURE FOR RFM PROBING

  • Probe tuning varies component counts by category, using fewer components for smaller or less complex categories and more for larger or more complex ones.For layer-wise probes, the tuning search maximizes AUC; for aggregation, it maximizes accuracy.Source: B TUNING PROCEDURE FOR RFM PROBING · p. 13

C STEERING ABLATIONS

  • Generation ablations examine layer contribution and per-timestep injection probability as steering controls affecting quality and concept alignment.The layer strategies include top-k selection and exponential score-weighting.Source: C STEERING ABLATIONS · p. 14

C.1 SETUP AND METRICS

The steering ablations vary layer weighting or selection and the probability of applying control at each timestep. Their captions frame the trade-off between selective steering, concept alignment, and artifacts.

C.1 SETUP AND METRICS

  • The ablations hold the schedule constant and vary layer weighting and injection probability.The section studies exponential score-weighting, linear score-weighting, and hard top-K selection for controlling how many and how strongly layers steer.Source: §C.1 · p. 14, §C.2 · p. 14

C.2 ABLATING LAYER PRUNING

  • Exponential weighting becomes more selective as κ decreases, concentrating weight on high-scoring layers; linear weighting maps the best layer to w0 and the worst to 0.The table caption describes exponential decay as interpolating between flat weighting as κ approaches 1 and highly concentrated weighting as κ approaches 0.Source: §C.2 · p. 14, Table 9 · p. 14
  • Hard top-K selection controls the effective number of steered layers, with the ablation sweeping K from 4 to 48.K=48 means all layers.Source: Table 10 · p. 14, §C.2 · p. 14

C.3 ABLATING INJECTION PROBABILITY P

  • Lower injection probability reduces artifacts but may weaken alignment, whereas higher probability strengthens control but risks over-steering.At each generation step, a Bernoulli gate determines whether control is applied.Source: Table 11 · p. 15, §C.3 · p. 15

D SINGLE DIRECTION METRIC GRAPHS

Figure 2 shows how control coefficient η0 relates to distributional metrics, prompt alignment, and probe accuracy. Stronger control increases FD and MMD, while CLAP alignment stays relatively stable for most categories.

D SINGLE DIRECTION METRIC GRAPHS

  • FD and MMD increase as the control coefficient η0 strengthens.The figure describes both as lower-is-better distributional metrics.Source: Figure 2 · p. 15
  • CLAP alignment to the text prompt remains relatively stable for most categories as control strength changes.CLAP is an up-is-better metric in the figure.Source: Figure 2 · p. 15
  • Probe accuracy trends upward with increasing η0, despite poor performance on generated data.Source: Figure 2 · p. 15

E CONTROL SCHEDULES USED FOR TIME CONTROL ABLATIONS ON NOTE

The schedule examples include exponential decay and logistic increase. The steering pseudocode applies weighted, scheduled directions during autoregressive generation, optionally gating their application at each step.

E CONTROL SCHEDULES USED FOR TIME CONTROL ABLATIONS ON NOTE

F RFM STEERING PSEUDOCODE

  • At each timestep, the algorithm scales a direction by η0, its layer weight, and the schedule value, then adds it to the layer activation when the gate fires.The gate is sampled from Bernoulli(p).Source: F RFM STEERING PSEUDOCODE · p. 16
  • The pseudocode repeats transformer-layer processing and steering before sampling the next sequence token, continuing until T timesteps.Source: F RFM STEERING PSEUDOCODE · p. 16
Loading 2510.19127v2…