PaperlayerView exact revision on arXiv

Source-linked AI summary

From Isolated Feature to Orbits: Discovering Music Concepts via Multi-SAE Alignment

Liwei Lin, Gus Xia

arXiv:2610.01864v1cs.SDcs.AI

TL;DR

Music foundation models encode musical information, but how that information is organized remains difficult to interpret, and feature-level analyses can miss structured relations. This paper aligns SAEs across pitch-shifted views to recover feature orbits, finding structures associated with chords, keys, and melodies across two models.

  • Problem

    How musical information is organized in music foundation-model representations remains difficult to interpret, while common probing and SAE approaches focus on individual concepts or features rather than their relations.

    Source: §1 · p. 1

  • Method

    The framework uses pitch transposition as an inductive bias, aligning multiple SAEs trained on shifted views to recover ordered orbits whose semantics can be grounded from a few anchors.

    Source: §1 · p. 2

  • Results

    Across two music foundation models, the method recovers chord-, key-, and melody-related orbits; chord rings support competitive recognition with minimal supervision, and MuQ-derived orbits consistently exceed MusicFM-derived orbit quality.

    Source: §1 · p. 2, §5.4 · p. 8, §5.4.1 · p. 9

  • Takeaways & Limitations

    The recovered orbit structures reveal relational organization in learned representations, including a previously unreported subdominant-centered orbit encoding relative key structure.

    Source: §7 · p. 10

    The method focuses on pitch-transposition and frame-level concepts; organizing pitch-invariant concepts or temporal aggregation remains unexplored, and the data and step size yield mostly Western music concepts.

    Source: §6 · p. 10

Abstract

from arXiv · show

How can we understand what a music foundation model has learned \textit{internally}? Most interpretability approaches, such as probing and Sparse Autoencoders (SAEs), focus on identifying individual features with minimal structural assumptions. We argue that many concepts are better understood as \textit{structured relations} rather than isolated features. This is especially prominent in music, where tonal structures are organized in the space of pitch and time. For example, concepts such as chords or keys are naturally expressed as structured sets (e.g., the 12 transpositions of a chord or the diatonic system within a key), rather than isolated features. In this study, \textbf{we shift from feature identification to structure-based analysis}, asking whether the learned inner representations of music foundation model emerge as organized structures over features. To this end, we introduce a framework that uses pitch transposition as an inductive bias to induce ordered orbits via multi-view SAE alignment. Concretely, we generate pitch-shifted input pairs and align their SAE representations to discover structured groups of pitch-related features. Experimental results show that this approach recovers orbit structures corresponding to chords, keys, and melodic patterns across two state-of-the-art music foundation models, while requiring only minimal grounding (e.g., a few anchor examples) to interpret entire concept families.

1 Introduction

Music-model representations are difficult to interpret, and feature-by-feature analyses miss the relational structure of many musical concepts. The paper uses pitch transposition and multi-SAE alignment to recover feature orbits, interpreting concept families from a few anchors and their shared invariants.

1 Introduction

  • The paper shifts SAE interpretability from isolated features to relational structures, where a few labeled anchors can help interpret related features and shared invariants reveal abstract concepts.For example, the invariant across a transposed chord orbit is major-chord quality.Source: §1 · pp. 1–2
  • Pitch transposition induces ordered orbits because pitch-specific concepts transform equivariantly while some components, such as chord quality, remain unchanged.A C major chord transposed upward by one semitone becomes C♯ major.Source: §1 · p. 2
  • Multiple SAEs trained on pitch-shifted views are aligned so same-index features represent adjacent transpositions, then a successor relation recovers within-SAE orbits.A few labeled anchors can propagate semantics around an orbit.Source: §1 · p. 2
  • Across two music foundation models, the framework recovers orbits associated with chords, keys, and melodic structure using minimal label grounding.The authors report that in some cases the approach supports one-shot transfer to downstream understanding tasks.Source: §1 · p. 2

2 Related Work

Prior work probes music representations or assigns meanings to individual SAE features, while SAE alignment methods match features across models or views. This paper extends these approaches by recovering relational orbits of pitch-related features.

2.1 Interpretability of Music Foundation Models

  • Probing and SAE studies identify musical attributes or individual interpretable features, but individual neurons may be polysemantic and feature-level analyses overlook relations among concepts.Probing tests predefined concepts with labels; SAE analyses often assign meanings post hoc.Source: §2.1 · p. 3

2.2 Evolution of SAE Architectures and Sparsity

  • Top-K SAEs preserve exactly K latents and are used here to address the reconstruction-sparsity trade-off associated with traditional L1-regularized SAEs.The cited work describes L1 regularization as susceptible to feature-magnitude shrinkage.Source: §2.2 · p. 3

2.3 SAE Alignment and Relational Interpretability

  • Unlike joint dictionaries or post-hoc feature matching, this work uses multi-view SAE alignment to recover entire orbits of pitch-related features.The approach centers analysis on the relational structures defining musical concepts.Source: §2.3 · p. 3

3 Methodology

The methodology treats pitch transposition as a transformation that organizes musical concepts into orbits, then trains aligned SAEs to recover corresponding feature structures. A successor map turns feature matches into rings, fixed points, or open sequences.

3.1 Transposition-Induced Orbits of Music Concepts

  • Pitch transposition generates ordered orbits for pitch-specific concepts, while pitch-invariant concepts form singleton fixed points.Within a transposed chord orbit, pitch changes but chord quality remains invariant, identifying the underlying abstract concept.Source: §3.1 · pp. 3–4

3.2 Multi-SAE Alignment with Shared Top-K Support

  • For each audio segment, the method creates adjacent one-semitone-shifted views and trains a Top-K SAE per view on temporally corresponding foundation-model representations.SAEs share a learned pre-encoder bias and latent dimensionality, while their encoder and decoder weights are learned independently.Source: §3.2 · p. 4
  • Joint shared Top-K support co-selects latent indices across views, coupling SAE dictionaries so same-index features are encouraged to represent adjacent transpositions.Activations are normalized to choose shared support, then the same mask is applied to unnormalized activations; this alignment is an inductive bias, not a guarantee.Source: §3.2 · pp. 4–5

3.3 Recovering Rings and Sequences

  • After training, decoder matches across adjacent SAEs define a successor map; repeated application yields rings, self-mapped fixed points, or open sequences, which anchors can interpret.With three SAEs, the middle SAE serves as reference; candidate matches are checked in both transposition directions and accepted only above a confidence threshold.Source: §3.3 · pp. 5–6

4 Experiments

Experiments apply the framework to MuQ and MusicFM representations and examine recovered orbits in music recordings. The reported patterns include chord rings, melody-like chains, and a subdominant-centered orbit related to key structure.

4.1 Datasets and Configuration

  • The study uses 326 Slakh2100 training songs and 164 validation songs, extracting frame-level representations from MuQ and MusicFM.The examples of recovered orbits are drawn from three multi-instrument vocal recordings in POP909.Source: §4.1 · p. 6, §4.2 · p. 7

4.2 Recovered Orbits

  • Both MuQ and MusicFM yield similar 12-feature rings and chains, including major- and minor-chord rings and a piano-roll-like melodic chain.The figure organizes feature activations by song time and feature index, with chord segments and orbit boundaries marked alongside piano-roll ground truth.Source: §4.2 · pp. 6–7, Figure 2 · p. 7
  • A 12-feature subdominant-centered ring in MuQ appears to encode scale degree relative to the key, behaving similarly across tested major keys but tending toward the relative major in minor keys.The authors describe this pattern as new evidence about how a foundation model encodes key in local frames.Source: §4.2 · p. 7

5 Downstream Task Evaluation

The evaluation tests whether discovered chord and key orbits support downstream music understanding and compares orbit-derived features across two foundation models. The results distinguish explicit salient structures from information that a supervised probe can recover or reconstruct.

5.1 Baselines; 5.2 Metrics; 5.3 Ablations

  • Evaluation includes supervised linear probes and external S-Key/LVCR baselines, key-accuracy and weighted chord-recall metrics, and a standard-SAE feature-matching ablation.Key metrics include segment- and song-level accuracy; chord metrics include root weighted recall and root-and-quality weighted recall.Source: §5.1 · pp. 7–8, §5.2 · p. 8, §5.3 · p. 8

5.4 Results

  • Across evaluations, MuQ-derived orbits were consistently higher quality than MusicFM’s; chord rings emerged in both, but the subdominant ring only in MuQ.All reported SAE results use sparsity K = 48.Source: §5.4 · p. 8

5.4.1 Chord Rings

  • Major- and minor-chord rings support chord recognition that is competitive with fully supervised methods despite minimal label supervision.MuQ-derived rings consistently outperform MusicFM-derived rings.Source: Table 1 · p. 8, §5.4.1 · p. 9

5.4.2 Subdominant Ring

  • The discovered subdominant rings capture musically meaningful key information, and MuQ-derived rings consistently outperform those from MusicFM.Table 2 reports key-detection accuracy while ignoring relative major/minor differences.Source: Table 2 · p. 8, §5.4.2 · p. 9

5.4.3 SAE Evaluates Features Saliency Beyond Linear Recoverability

  • Linear probes detect weak tonic and dominant signals, but matched standard-SAE features do not, suggesting those signals lack explicit, readily accessible concepts.Figure 3 contrasts generally high probe accuracy with concept-specific results from standard SAE.Source: §5.4.2 · p. 9, §5.4.3 · p. 9, Figure 3 · p. 9
  • Key information can be reconstructed by manually combining chord activations, while the unsupervised orbit-based SAE identifies the salient subdominant structure.This suggests key information can be distributed across other concepts rather than represented by a distinct tonic concept.Source: §5.4.3 · p. 10

5.4.4 Global Recoverability vs. Local Structure

  • Global-key probes use pooled frame representations, whereas this method identifies a local subdominant concept; probing also identifies the subdominant as the most salient local key cue.The cited prior probing studies are.Source: §5.4.4 · p. 10

6 Limitations

The study’s orbit analysis is bounded by its pitch-transposition focus and frame-level scope, and the observed rings and concepts may reflect its step size and data distribution.

6 Limitations

  • Pitch-transposition orbits do not organize pitch-invariant concepts such as rhythm without additional inductive biases.How other transformations may interpret other forms of music remains unstudied.Source: §6 · p. 10
  • The analysis is frame-level; how foundation models organize and aggregate frame-level concepts over time remains unexplored.Source: §6 · p. 10
  • The 12-sized rings may relate to the one-semitone step being 1/12 of an octave and, with the dataset distribution, yield mostly Western music concepts.Source: §6 · p. 10

7 Conclusion and Future Work

The conclusion presents pitch-transposition-induced SAE orbits as a way to interpret higher-level musical concepts, including a newly identified orbit encoding relative-key structure. The accompanying primer and dataset descriptions establish the musical concepts and evaluation settings used in the paper.

7 Conclusion and Future Work

  • The framework recovers transposition-induced SAE orbits that support interpretation of chords, keys, and melodic patterns, including a newly reported subdominant-centered orbit encoding relative-key structure.The orbit resembles the Circle of Fifths, and the framework uses minimal supervision.Source: §7 · p. 10
  • The authors propose that structured groups can reduce feature-level noise despite polysemanticity, and that coherent orbits may help assess conceptual organization.They also suggest applying known transformations as inductive biases beyond music and combining structure-aware analysis with feature-level methods.Source: §7 · p. 10

A.1 Pitch, Pitch Classes, and Equal Temperament

  • The primer defines pitch classes as pitches abstracted from octave height, represented modulo 12 by the twelve chromatic pitch classes.The twelve-tone equal-temperament octave is divided into 12 semitones.Source: §A.1 · p. 13

A.2 Chords and Pitch Transposition

  • A chord combines distinct pitch classes, and repeated one-semitone transposition cycles its pitches while preserving its internal interval structure.The primer gives a C major triad as an example and describes the resulting orbit as size 12.Source: §A.2 · p. 13

A.3 Harmonic Functions and the Circle of Fifths

  • The Circle of Fifths orders pitch classes by perfect fifths, placing tonic, subdominant, and dominant in adjacent positions.Clockwise movement steps up seven semitones toward the dominant; counter-clockwise movement steps up five toward the subdominant.Source: §A.3 · p. 13, Figure 4 · p. 14

A.4 Relative Keys and Key Signatures

  • Each major key shares its key signature and pitch-class set with a relative minor, paired with it on the Circle of Fifths.The relative minor tonic lies three semitones below the major tonic; Figure 4 places major keys outside and relative minors inside.Source: §A.4 · p. 13, Figure 4 · p. 14

B.1 Training and Validation Data

  • Training and validation use Slakh2100, synthesized from aligned MIDI performances; pseudo frame-level labels support balancing and probe training, but not SAE supervision.The key-balanced split uses up to 16 tracks per key class; synthesized audio lacks the full acoustic variability of real recordings, and downstream experiments evaluate transfer to real recordings. Table 4 summarizes dataset tracks retained after label-validity filtering.Source: §B.1 · p. 14, Table 4 · p. 15

B.2.1 Chord-Recognition Datasets

  • Chord recognition covers Slakh2100 test songs and real Chinese and Japanese popular-music recordings, while POP909 and RWC-POP-100 assess transfer from synthesized instrumental audio.Tracks are segmented into 30-second evaluation segments; the evaluation also includes genre-diverse and electronic-music collections for key detection.Source: §B.2.1 · p. 15, §B.1 · p. 15, §B.2 · p. 15

B.2.2 Key-Detection Datasets

The appendix describes key-detection datasets and filtering, then specifies the task-specific and probe baselines used for chord recognition and key detection.

B.2.2 Key-Detection Datasets

  • Key detection uses FMAKv2 (5,434 tracks), GTZAN-Key (830 tracks), and GiantSteps (604 excerpts), spanning genre-diverse recordings and electronic dance music.GiantSteps has a strongly minor-dominated mode distribution; Table 5 reports dataset major–minor distributions.Source: §B.2.2 · pp. 15–16, Table 6 · p. 16

B.3 Evaluation Preprocessing and Exclusion Criteria

  • Evaluation excludes undecodable, short, unlabeled, or key-changing tracks because key detection assumes one global key per song.GTZAN-Key modulation or unknown labels and GiantSteps excerpts with multiple key labels are removed.Source: §B.3 · p. 16

C Baselines

  • Baselines include supervised linear probes trained on the same data and frozen representations, plus task-specific systems evaluated with released checkpoints trained on external data.The appendix defines n, t, p, and s as song, frame, pitch-class, and sub-SAE indices.Source: C Baselines · p. 16

C.1 Chord Recognition

  • Chord recognition compares LVCR, a supervised large-vocabulary transcription system, with a 25-class linear probe covering 12 major chords, 12 minor chords, and no-chord.Unsupported chord qualities and unknown annotations are ignored when forming the probe's reduced MajMin targets.Source: §C.1 · pp. 16–17

C.2 Key Detection

  • Key detection includes released S-KEY and independent 12-output probes for tonic, subdominant, and dominant functions; the main comparison uses the subdominant probe.S-KEY extends STONE with transposition-invariant chroma pseudo-labels and is evaluated without fine-tuning.Source: §C.2 · p. 17

D Downstream-Task Inference

The section uses recovered SAE features and orbits directly for chord and key prediction, then evaluates them through metrics, post-hoc interpretation, and visual analyses. These analyses examine recovery thresholds, broader orbit graphs, and the geometry of subdominant features.

D Downstream-Task Inference

  • The inference pipeline evaluates recovered SAE orbits directly for downstream prediction, applying shared aggregation and decision rules to probe outputs where possible.It does not fit an additional classifier on orbit activations.Source: D Downstream-Task Inference · p. 17

D.1 Chord Recognition

  • For chord recognition, the method averages smoothed chord-feature activations within segments and jointly selects chord quality and root, using oracle or detected boundaries.Detected boundaries come from local peaks in a chord-boundary feature; silence features provide no-chord evidence.Source: §D.1 · pp. 17–18

D.2 Key-Signature Detection from Scale-Degree Features

  • Scale-degree key detection selects the most active pitch in a 12-class tonic, subdominant, or dominant ring, without distinguishing relative major from minor.Segment-level signature scores are averaged at song level before prediction.Source: §D.2 · p. 18

D.3 Key Detection from Chord Orbits

  • Chord-orbit key detection scores normalized major- and minor-chord activations against music-theory templates to select among 24 tonic-and-mode candidates.The chord rings are normalized separately so that one chord quality’s overall activation strength does not dominate the key score.Source: §D.3 · pp. 18–19

E.1 Chord Recognition; E.2 Key Detection

  • Chord recognition uses Root and MajMin WCSR, while key evaluation reports exact signature or 24-class accuracy and, where applicable, MIREX weighted scores.The chord metrics handle unsupported qualities, extensions, unknown annotations, and no-chord labels according to the stated vocabulary rules.Source: §E.1 · p. 19, §E.2 · pp. 19–20

F Batched Post-hoc Evaluation Protocol

  • A batched post-hoc protocol uses a few labeled anchors to label recovered structures and select auxiliary features, without changing training or orbit membership.A frozen chord probe also ranks SAE checkpoints after training; its supervision and gradients are not used to train the SAE.Source: F Batched Post-hoc Evaluation Protocol · p. 20

G.1 Sensitivity to the Recovery Threshold; G.2 Sparse Directed Orbit Graphs

  • Recovery consistency is examined across thresholds, and retaining more graph edges reveals near-cyclic structures that strict one-edge recovery can miss.No minor-chord orbit is recovered from MuQ layer 6 at any tested threshold; the broader graph analysis retains up to ten incoming and outgoing edges per node.Source: §G.1 · pp. 20, 22, §G.2 · p. 22

G.3 Geometric Structure of the Subdominant Ring

  • The subdominant-ring visualization places pitch-class-ordered features within the broader SAE dictionary, but t-SNE supports qualitative local-organization judgments, not quantitative global-distance claims.The plots project normalized decoder vectors after PCA; features are included if activated in at least 2,000 training and validation segments.Source: §G.3 · p. 23, Figure 9 · p. 23, Figure 10 · p. 23

G.4 Orbit Activation Patterns

Orbit activations show ordered, chord-aligned patterns in music excerpts, while also revealing imperfect local separation. The section then describes comparisons across SAE types and sparsity settings, but the supplied passages report their evaluation setup rather than comparative outcomes.

G.4 Orbit Activation Patterns

  • Synthetic excerpts compare key and mode across instrument families; ordered activation patterns recur across timbres, although activation strength and selectivity vary with timbre and voicing.The diagnostic set is used for post-hoc analysis rather than SAE training.Source: §G.4 · pp. 25–26
  • In POP909 excerpts, ordered orbit features change prominence with chord roots, and some orbit activations align with chord boundaries; a longer orbit tracks melody activity.The excerpts are real vocal recordings with complex arrangements, and the visualized rows group features by recovered orbit.Source: Figure 12 · p. 25, §G.4 · p. 26
  • C-minor activations overlap across neighboring chord segments, with dense instrumentation, vocals, non-chord tones, and relative-major ambiguity complicating local orbit–chord correspondence.The authors retain this example to show that recovered semantics are not perfectly disentangled at every frame.Source: §G.4 · p. 26

H.1 s-SAE and Layer-Wise Comparison

  • The layer-wise comparison evaluates linear probes, standard single Top-K SAEs, and orbit-aware SAEs on chord recognition and key detection, selecting each representation’s best result across K values.The supplied passages describe the comparison datasets and metrics, but do not state comparative outcomes.Source: §H.1 · pp. 26–27

H.2 Effect of Top-K Sparsity

  • The sparsity analysis varies K across 32, 48, and 64 for chord recognition, chord-derived key detection, and subdominant-ring key-signature detection.The supplied passages describe the plotted comparisons and reference probes, but do not report their outcome values.Source: §H.2 · p. 27, Figure 15 · p. 28

H.3 Training Dataset Ablation

Training aligned SAEs on real-world POP909 recordings still recovers the three core orbits with comparable downstream performance. The section also describes an initialization intended to improve correspondence across sub-SAEs without assigning fixed semantic identities.

H.3 Training Dataset Ablation

  • Training on a subset of vocal, complex-production POP909 recordings recovers major-chord, minor-chord, and subdominant orbits with comparable downstream performance.The reported configuration uses MuQ layer 2 and K = 32.Source: §H.3 · p. 28

I Robustness Analysis

  • The robustness analysis also examines sparsity-dependent key-detection results and downstream performance across random seeds, but the supplied passages give figure descriptions rather than outcomes.The figure captions specify K = 32, 48, and 64 for the key-detection analyses and K = 48 for the seed comparison.Source: Figure 16 · p. 29, Figure 17 · p. 29, Figure 18 · p. 30

I.1 Orbit-Aware Initialization

  • Orbit-aware initialization uses pitch-shifted observations to establish initial correspondence among sub-SAEs, but does not assign fixed semantic identities or determine whether orbits emerge.It is intended to reduce sensitivity to random initialization; feature IDs can vary across runs while relative transposition structure is less sensitive.Source: §I.1 · pp. 29–30

I.2 Robustness across Random Seeds

Across random seeds, the core orbit structures remain identifiable despite changes in feature IDs and local graph connections. Across GTZAN genres, the recovered structures remain informative, with performance and interpretability varying by genre.

I.2 Robustness across Random Seeds

  • Across four random seeds, major-chord, minor-chord, and subdominant structures remain identifiable despite changes in feature IDs and local graph connections.The authors describe the core orbit semantics as reproducible up to feature permutation.Source: §I.2 · p. 30, Figure 19 · p. 31

I.3 Genre Sensitivity

  • Across all nine GTZAN genres, recovered structures remain informative, though performance varies; blues performs lowest overall, while pop, country, and reggae generally perform strongly.Blues and hip-hop activations are harder to interpret than those of other genres, consistent with the key-detection performance.Source: §I.3 · p. 31, Figure 20 · p. 32, Figure 21 · p. 33

Related papers

Loading 2610.01864v1…