Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System

Yi Wang

Source-linked AI summary

Why GPT-Style Models Do Not Directly Transfer to Symbolic Music: Compression in the Wrong Coordinate System

Yi Wang

arXiv:2608.18025v1Aug 18, 2026cs.LGcs.AIcs.SD

TL;DR

Symbolic-music tokenization lacks a principled coordinate system that makes musical regularities predictively compressible while preserving context-dependent relations. The paper formulates two boundaries for constructing such representations, and experiments show that effective coordinates improve compressibility whereas compaction alone and fixed relational projections do not.

Problem
Symbolic-music tokenization lacks a principled account of which coordinate system makes recurring musical regularities predictively compressible, beyond merely grouping observations into reusable units.
Method
The paper formulates the Effectiveness–Losslessness Framework, defining Fact–Token and Token–State boundaries for effective, lossless coordinate construction.
Results
Musical-coordinate denesting and reversible canonicalization improve predictive compressibility, while sequence compaction and fixed relational projections do not.
Takeaways & Limitations
Effective tokenization should expose predictable regularities while preserving contextual freedom for higher-order musical organization to emerge.
Takeaways & Limitations
The claims are bounded to the tested representations, constrained model envelope, and evaluated symbolic-music setting, without establishing scaling behavior beyond it.

Abstract

from arXiv · show

GPT-style models achieve strong performance by representing language with finite vocabularies of reusable discrete tokens. This success has motivated symbolic music tokenizations to treat recurring musical structures, such as chords, motifs, and phrases, as reusable units analogous to linguistic tokens. However, tokenization derives its advantage not from reusable combinations alone, but from compression: effective compression requires coordinates in which recurring regularities form stable and predictable conditional distributions. The key problem is therefore not to find larger musical combinations, but to discover the coordinate system in which musical facts become predictively compressible. We formulate the Effectiveness--Losslessness Framework and define tokenization as the construction of a predictively effective and relationally lossless coordinate system. The Predictive Effectiveness Principle defines the Fact--Token Boundary: decoupling and denesting construct coordinate interfaces that expose predictive regularities. The Relational Losslessness Principle defines the Token--State Boundary: tokenization stops before context-dependent relations are fixed, leaving their computation to model states. Controlled symbolic-music experiments validate these boundaries. Effective coordinate construction improves predictive compressibility, while fixed relational projections constrain contextual modeling. Sequence compaction alone does not guarantee predictive compression, while preserving contextual freedom allows higher-order musical organization to emerge without explicit structural labels. These results reveal why GPT-style models do not transfer directly across modalities: architectures transfer, but tokenization interfaces do not. Tokenization must discover effective representations while preserving the relational freedom from which contextual structure can emerge.

1 INTRODUCTION

The introduction argues that GPT-style tokenization succeeds through predictive compression, not reusable combinations alone, and that symbolic music therefore requires a modality-specific coordinate system. It presents the Effectiveness–Losslessness Framework to define where tokenization should begin and end while preserving unresolved contextual structure.

Motivation: GPT-style models use finite reusable tokens as a compact interface between raw sequences and contextual modeling.The tokens can be recombined across contexts to represent an effectively unbounded range of linguistic structures.
Motivation: Symbolic-music tokenizations analogize chords, motifs, phrases, and sections to linguistic tokens, testing reusable units at different granularities.Existing approaches include event streams, compound tokens, regular time-step representations, and learned segment vocabularies.
Core problem: Grouping observations into larger symbols does not ensure predictive compression; compression requires coordinates where recurring regularities form stable, efficiently predictable conditional distributions.The relevant predictability is defined relative to a bounded model.
Framework: Tokenization must construct the coordinate system in which musical facts become predictively compressible, while determining where contextual structure should remain unresolved.The framework defines token boundaries by both representation placement and preservation of unresolved relations.
Framework: The Effectiveness–Losslessness Framework establishes where tokenization should begin and end, with the Predictive Effectiveness Principle defining the Fact–Token Boundary.The introduction identifies decoupling and denesting as coordinate-interface operations that expose stable objective regularities.

2 THE EFFECTIVENESS–LOSSLESSNESS FRAMEWORK

The Effectiveness–Losslessness Framework defines tokenization as constructing a coordinate system that makes observable facts predictively compressible while preserving contextual relational degrees of freedom. It separates coordinate construction from contextual compression, establishing boundaries for what tokens should expose and what model states should infer.

Framework: Tokenization must preserve declared observable distinctions, but effectiveness additionally requires coordinates that expose recurring regularities as stable, efficiently predictable conditional distributions.Shorter sequences or smaller vocabularies alone do not establish predictive effectiveness.
Framework: The compression pipeline separates coordinate construction, carrier coding, and contextual state formation, assigning representation-interface construction to tokenization and contextual abstraction to model states.State formation performs generally non-invertible predictive compression over the preserved representation.
Predictive Effectiveness Principle: The Predictive Effectiveness Principle defines the Fact–Token Boundary: decoupling removes artificial coupling, while denesting exposes reversible objective transformations for reusable conditional structure.These operations preserve genuine dependencies and align equivalent regularities instead of requiring the model to rediscover them.
Relational Losslessness Principle: The Relational Losslessness Principle defines the Token–State Boundary: tokenization must preserve context-dependent alternatives and may encode relations only when they are deterministic consequences of preserved observations.Fixing a relation before context is available removes alternative relational organizations from contextual computation.
Framework: A token is a predictively effective coordinate carrier that exposes stable objective regularities while preserving the contextual degrees of freedom required to determine relations.The tokenizer constructs the coordinate interface, whereas contextual states perform predictive compression over the preserved representation.

3 FROM THE TWO BOUNDARIES TO A COORDINATE-AWARE MUSIC MODEL

The model constructs musical tokens by exposing predictable event coordinates while preserving context-dependent musical relations for state-side computation. This yields coordinate-aware notes whose content and musical time are explicitly represented, while higher-order organization emerges contextually.

Predictive-Effectiveness Operations: Decoupling separates event content from musical position without assuming independence, preserving dependencies while avoiding a single joint event identity.Type, pitch, and duration describe content, whereas onset specifies position; simultaneity and overlap remain recoverable from these coordinates.
Predictive-Effectiveness Operations: Denesting derives musical time from onset and exposes periodic and relative temporal regularities through structured temporal coordinates and attention biases.The representation includes bar progress and periodic within-beat, beat-within-bar, bar, four-bar, and sixteen-bar phases.
Pitch Operations: Pitch-class/register decomposition and canonicalization preserve reversibility, whereas fixed chromatic or circle-of-fifths projections impose contextual relations before the model observes sufficient context.Canonicalization removes transposition as a predictive nuisance without supplying tonic, mode, scale-degree, key, or harmonic-function labels.
From the Two Boundaries: The Fact–Token Boundary preserves observable event facts in a predictively effective coordinate interface, while the Token–State Boundary leaves contextual interpretations unresolved.Context-dependent structures such as chords, motifs, phrases, voices, and tonal functions remain available to contextual states rather than being fixed by tokenization.
Coordinate-Aware Music Model: The coordinate-aware musical token is z_i = (q_i, τ_i), with q_i = (t_i, e_p_i, d_i) and τ_i = ϕ(o_i), separating event content from musical time.A causal Transformer processes these coordinates with coordinate-derived biases and predicts exact events in TYPE→TIME→PITCH→DURATION order.

4 RELATED WORK

Related work studies tokenization through predictive compression, serialization, abstraction, and music-specific coordinate choices. This paper distinguishes coding and contextual operations from coordinate construction and evaluates those choices within a boundary framework.

Tokenization and predictive compression: Tokenization quality depends on coding efficiency and predictive behavior, not token count alone, with lossless representations producing different predictive losses under finite context.Recent work also formalizes tokenization as a representation mapping independent of a particular vocabulary.
Serialization and combination: Symbolic-music systems study event serialization, temporal notation, compound representations, and sequence merging, whereas this work separates coding operations from coordinate construction and evaluation.The cited approaches optimize how representations are serialized or combined.
Abstraction and contextual computation: Hierarchical representations and learned segment codes add abstraction, while temporal layout and tokenization choices influence modeling behavior; this work separates deterministic coordinates from contextual abstraction.The separation identifies which structures are represented explicitly and which emerge in model states.
Music-specific coordinate priors: Music-specific models encode pitch, onset, and temporal structure through varied coordinates, including compared time and duration tokenizations, which this work tests under a fixed learner.The proposed boundary framework evaluates which coordinate operations improve predictive code length.

5 EXPERIMENTS AND DISCUSSION

Matched experiments show that predictive compression improves when representations expose temporal and canonical regularities, but worsens when serialization brevity or fixed relations replace coordinate construction. Contextual freedom also supports emergent musical organization, within the tested symbolic-music and model limits.

Musical-time coordinates: D lowers clean-test predictive code length by 42.73% relative to A as temporal coordinates progressively expose regularities hidden by serialization order.The A/D direction is independently replicated on ComMU under a matched within-corpus budget.
Pitch coordinate operations: Reversible canonicalization reduces predictive code length by 39.57% relative to F, while invertible pitch reparameterization alone shows no demonstrated gain.G removes the known shift before prediction and retains it only for exact reconstruction, without supplying tonal labels.
Serialization controls: K shortens J’s serialization by 71.07% yet requires 14.00% more predictive bits, whereas D reduces code length by 25.15% relative to J.Sequence compaction therefore does not substitute for coordinate construction when predictive code length is normalized by observable-event count.
Fixed relational projection: Fixed chromatic and fifths geometries increase predictive code length by imposing pitch relations before contextual computation, constraining relations that should remain available to the state.This contrasts with deterministic temporal coordinates, which expose relations without fixing them in advance.
Context utilization: Removing either context increases code length by about 0.27 bits/event, while shuffling identical content increases it by 0.669 bits/event.The intervention indicates dependence on ordered content–time binding.
Emergent organization and limitations: Generated continuations exhibit recurring patterns, variation, chordal texture, and longer-range recurrence without explicit structural tokens, providing qualitative existence evidence for emergent contextual organization.The claims remain bounded to the tested representations, constrained model envelope, and evaluated symbolic-music setting; predictive code length is model-relative rather than a global optimum or musical-quality measure.

6 CONCLUSION

The conclusion reframes tokenization as coordinate construction governed by complementary Fact–Token and Token–State boundaries. Symbolic-music results support effective coordinates and preserved contextual freedom, while contextual states can learn to compress remaining relational uncertainty.

6 CONCLUSION: Tokenization is framed as coordinate construction, with the Fact–Token Boundary selecting observable information and deterministic regularities and the Token–State Boundary preserving relations for contextual computation.Together, these boundaries define the framework’s account of what belongs in tokens versus model states.
6 CONCLUSION: Symbolic-music experiments show that coordinate denesting and reversible canonicalization improve predictive compressibility, whereas uninformative reparameterization and fixed relational projections do not.Sequence compaction alone is insufficient, while preserving contextual freedom enables higher-order organization to emerge.
6 CONCLUSION: The framework interprets contextual states as learned compressors of relational uncertainty that remains difficult to specify explicitly after effective coordinates represent observable facts and deterministic regularities.This interpretation is presented as broader than the specific application and not tied to a particular architecture.

AI USE STATEMENT

Generative AI tools supported research assistance across ideation, coding, experiments, literature discovery, presentation, and editing, while human authors retained final control and did not treat AI outputs as scientific evidence.

AI USE STATEMENT: Generative AI assisted with brainstorming, critical discussion, code drafting, experiment orchestration, literature discovery, figure and table preparation, and editorial revision, while authors retained final control over research decisions.The statement specifies that AI outputs were not treated as scientific evidence.

REPRODUCIBILITY STATEMENT

The paper addresses reproducibility through detailed protocol specifications, preregistered controls, provenance tracking, and implementation- and data-lineage-bound result hashes.

Reproducibility sections specify the lineage-aware split, prediction target, representation arms, shared model, optimization envelope, seeds, checkpoint rule, evaluations, tokenizer controls, and generation provenance.
Protocol, source-tree, split-manifest, and selected-checkpoint hashes bind each reported result to its frozen implementation and data lineage.
Test data were unavailable to training.

ETHICS AND DATA STATEMENT … E REPRESENTATION DETAILS

The paper documents non-commercial data use, frozen evaluation protocols, matched representation families, lineage-aware split integrity, and exact symbolic-music encodings. Its representations preserve recoverable musical coordinates while avoiding semantic partitions and withholding contextual relations from predictive inputs.

ETHICS AND DATA STATEMENT: Pop-K is used solely for non-commercial research under its creators’ Creative Commons Attribution–NonCommercial license, with source ownership neither claimed nor transferred.Source continuations are included only for scholarly comparison because some augmented segments may remain similar to source material.
A FROZEN PROTOCOL SUMMARY: A–E, F–G, and H–I each use frozen protocol, source-tree, and manifest hashes, with shared model envelopes within families.Trainable-parameter differences arise only from inactive representation paths that remain instantiated but frozen.
B FULL A–I ARM TABLE: F and G apply the same deterministic legal semitone shift per sample, epoch, and seed, retaining the shift for reconstruction but excluding it as a predictive feature.F predicts shifted absolute pitch, whereas G canonicalizes pitch tokenizer-side; the legacy implementation supplying reference information inside the model is excluded.
C FULL PAIRED FACTOR DECOMPOSITION: Table 3 reports held-out paired changes in bits per exact event, where negative values favor the left arm and rows reconstruct total change up to floating-point error.Absolute scores are compared only within the matched representation families defined in the main text.
D SPLIT AND TEST INTEGRITY: The Pop-K split groups samples by source lineage, exact content, or transposition-invariant duplicate evidence, while model selection uses validation data only.Test manifests are opened only after checkpoint selection; the H/I infrastructure resume reused verified caches and checkpoints without retraining, tuning, reselection, or reopening source files.
E REPRESENTATION DETAILS: All non-drum notes form one unsegregated event stream, with no supplied melody, bass, harmony, voice, chord, motif, phrase, or cardinality structure.Deterministic serialization separates observable events but does not imply performance order, melodic priority, voice membership, or semantic importance.
E REPRESENTATION DETAILS: Duration remains exactly recoverable per Note or REST event, and the decoder ordering is an exact-event chain-rule factorization rather than a semantic ranking.The musical-time interface combines directed bar progress with within-beat, beat-within-bar, bar, four-bar, and sixteen-bar phases to uniquely determine supported score onsets.
E REPRESENTATION DETAILS: D, E, H, and I derive deterministic attention biases from preserved musical-time coordinates, while pitch variants bijectively preserve supported absolute pitch without assigning musical identities.F and G retain identical shifted physical examples; G canonicalizes pitch and reconstructs it without tonic, mode, key, scale-degree, or tonal-function labels.

F SECONDARY CONTEXT EVIDENCE · G GENERATION PROVENANCE AND SOURCE CONTINUATIONS

A secondary probe shows that both left and right context substantially improve four-bar compatibility prediction, while showcase continuations are documented as provenance-controlled examples rather than prevalence estimates. The generation materials include exact-overlap scoring definitions, source comparisons, checkpoint identification, termination behavior, and distributed artifacts.

F SECONDARY CONTEXT EVIDENCE: Full-context accuracy was 75.93%, versus 48.87% for the target-only condition, on 6,112 windows from 99 held-out POP909 songs.Replacing left or right context reduced accuracy by 11.42 and 12.37 percentage points, respectively; replacing both reduced accuracy to 50.87%.
F SECONDARY CONTEXT EVIDENCE: The probe supports contextual dependence but is excluded from the primary tokenization result, numerical pooling, and representation ranking.It uses an earlier composition-level representation and is not pooled with the main fixed-target intervention.
F SECONDARY CONTEXT EVIDENCE: Replacing either context side degraded compatibility prediction, while replacing both approached the target-only control.This pattern is summarized in Figure 5’s secondary context-relation probe.
G GENERATION PROVENANCE AND SOURCE CONTINUATIONS: The two showcase cases were human-listening selections from one documented ten-case batch generated by the coordinate-aware note-centered causal checkpoint.They are not estimates of A–I prevalence.
G GENERATION PROVENANCE AND SOURCE CONTINUATIONS: Event Jaccard measures literal overlap of exact onset-relative-to-continuation-start, MIDI-pitch, and duration-in-ticks triples rather than musical quality.The definition spans the event representation and explicitly distinguishes overlap from quality.
G GENERATION PROVENANCE AND SOURCE CONTINUATIONS: Table 4 presents the showcase generation receipts for the documented examples.The supplied passage identifies the table as the source of the showcase receipts without providing their individual contents.
G GENERATION PROVENANCE AND SOURCE CONTINUATIONS: Case A’s model continuation is shown against a shared four-bar prefix, while source continuations provide exact visual comparisons for Cases A and B.Red boxes and blue labels in the Case A model figure are manual musical annotations, not model inputs.
G GENERATION PROVENANCE AND SOURCE CONTINUATIONS: The checkpoint SHA-256 is 946287c1997125c39edf0f1969880f8fc78dd854849459fc952c8dc5b0381f6b, and both examples terminate when the model emits its separate termination symbol.Annotated scores, source continuations, MIDI files, and WAV files are distributed under docs/demos/m4l-popk/.

H EVIDENCE BOUNDARIES · I TOKENIZER CONTROLS AND CONTEXT INTERVENTION · J INDEPENDENT-CORPUS A/D REPLICATION

The evidence supports bounded conclusions: matched experiments show directional predictive effects while leaving broader musical understanding, convergence, and raw-performance reconstruction unresolved. Tokenizer controls and an independent ComMU replication further test representation interfaces under matched conditions, with D favored directionally but without establishing universal superiority.

H EVIDENCE BOUNDARIES: The formal evidence is limited to one lineage-safe Pop-K task, a small fixed model envelope, three seeds, and five target-equivalent epochs.Held-out negative log-likelihood is model-relative predictive code length, not a global coding optimum, perceptual-quality measure, or human-preference score.
H EVIDENCE BOUNDARIES: H demonstrates no measurable predictive-code-length gain over D, because H−D was descriptive rather than an equivalence or non-inferiority test and one seed reversed direction.The near-zero mean difference should not be interpreted as proof of equivalence.
H EVIDENCE BOUNDARIES: The tested fixed continuous pitch topologies degrade performance, rejecting those replacements rather than all pitch geometry, learned contextual geometry, or identity-preserving auxiliary features.E and I also change effective active parameter counts.
H EVIDENCE BOUNDARIES: F/G assumes a known deterministic global shift and does not infer tonic, mode, key, scale degree, tonal function, or a reference frame.It is evaluated on a separately shifted observed distribution and is not formally ranked against D.
H EVIDENCE BOUNDARIES: Pairwise temporal bias establishes utility for deterministic temporal relations, while fixed-target intervention establishes sensitivity to ordered content–time binding without identifying human-readable musical variables.Neither result alone establishes phrase, section, cadence, motif, voice, or full-song structural understanding.
H EVIDENCE BOUNDARIES: Relationally lossless tokenization preserves context-dependent alternatives for state-side computation, but event-field reconstruction and sequence termination remain separate from byte-for-byte raw-MIDI or acoustic-performance recovery.The criterion does not imply recovery of controller data, instrumentation, timbre, articulation, or expressive micro-timing.
I TOKENIZER CONTROLS AND CONTEXT INTERVENTION: J compares an event-preserving REMI-like stream, train-only reversible BPE K, and D under matched data, split, seeds, model core, event-equivalent budget, and validation-only checkpoint selection.K does not add D’s multi-scale time coordinates or pairwise temporal relations; Table 5 normalizes predictive bits over exact Note/REST targets plus termination targets.
J INDEPENDENT-CORPUS A/D REPLICATION: The ComMU replication uses 9,299 songs with matched A/D training conditions, and all three paired seeds favor D on validation and sealed test.D primarily improves Type and Time on clean test while leaving Duration nearly unchanged; the result is directional replication, not convergence or a ComMU capability limit.