SheetSage2: Coherent Lead-Sheet Transcription with Synthetic Supervision

1. Introduction

• Lead-sheet transcription must coordinate musical dimensions on compatible grids, while complete, well-aligned audio–score training pairs remain scarce.Existing annotations often cover only selected attributes and may have imprecise timing.Source: §1 · p. 1
• Synthetic audio rendered from annotated MIDI supplies scalable supervision across music-understanding tasks and complements partially annotated real recordings.The Prober learns from partial annotations, and its structured decoders turn predictions into complete sequences.Source: §1 · pp. 1–2
• Task-specific structured decoders use tempo, meter, and pitch context to improve transcription consistency within and among tasks.Source: §1 · p. 2
• Autoregressive distillation transfers structured teacher outputs to a student that avoids task-specific dynamic programming while maintaining strong performance across evaluated tasks.Source: §1 · p. 2

2.1. SheetSage2-AR

• SheetSage2-AR encodes audio and preceding events to predict a unified event sequence, then converts it into ABC notation or an editable lead sheet.The event language combines absolute beat timestamps with relative positions on a sixteenth-note grid and typed musical tokens.Source: §2.1 · p. 2, Figure 1 · p. 3

2.2. An Overview of the Training Pipeline

• The three-stage pipeline renders labeled MIDI as aligned audio, trains Prober models on partial or noisy annotations, then distills their structured sequences into an autoregressive student.Source: §2.2 · p. 2

2.3. Synthetic Music & Synthetic Labels

• Symbolic analysis and pseudo-label bootstrapping expand MIDI supervision before rendering, including a melody pool grown from 17,134 seed files to 185,887 annotated songs (10.8×).The annotated MIDI collection and SLMS supply approximately 361,000 synthetic source records, rendered as 24-kHz audio for Prober training.Source: §2.3 · pp. 2–3

2.4. SheetSage2-Prober

• Two acoustic Prober models produce frame-wise task predictions, learning from partial and noisy labels while masking missing targets and accommodating annotation conventions.The non-melody model predicts binary features, while the melody model operates on beat-grid subdivisions conditioned on a vocal- or full-melody hint.Source: §2.4 · pp. 3–4

2.5. Structured Decoding

• A six-stage decoder fixes tempo and rhythm decisions in sequence, then constrains meter, chords, and melody to compatible beats and grid subdivisions.The shared grid promotes long-term tempo coherence and aligns dependent attributes.Source: §2.5 · pp. 4–5

2.6. Structured Decoding: Rhythm

• A pooled tempo anchor guides local pulse decoding across the recording, discouraging abrupt tempo switches while allowing local tempo and meter changes.Source: §2.6 · p. 5

2.7. Structured Decoding: Markovian Melody

• Markovian melody decoding scores each onset using both its pitch and interval from the preceding pitch, helping suppress local octave errors that distort melodic intervals.The preceding-pitch context changes the score assigned to a candidate onset, such as E4 following C4 versus C5.Source: §2.7 · pp. 5–6

3.1. Main Results

• SheetSage2-AR exceeds the strongest listed prior baseline on 12 of 15 benchmark–metric pairs and SheetSage2-Prober on 10 of 15.The results cover eight benchmark collections across rhythm, key, chords, structure, and melody.Source: §3.1 · p. 6, Table 2 · p. 7

3.2. Effectiveness of Synthetic Data

• Adding synthetic data improves beat, downbeat, chord, and structure averages, leaves melody nearly unchanged, and slightly reduces the key average.Synthetic-only training also reaches 57.76% vocal-melody F1 on RWC-Pop despite containing no singing.Source: §3.2 · p. 6, Table 3 · p. 7

3.3. Effect of Structured Decoding on Rhythmic Consistency

• Prober and AR mix tempo levels less often than Beat This! on both datasets, and the proposed decoder is more consistent than P-DBN.In Figure 3(a,b), P-DBN changes tempo level while Prober keeps a stable grid.Source: §3.3 · p. 6, Figure 3 · p. 7

3.4. Effect of Structured Decoding on Octave Consistency

• On 333 common passages, full decoding reduces the octave mixing ratio from 12.91% to 9.31%; AR reaches 11.71% without explicit melody DP at inference.The comparison tests the full melody decoder against a pitch-marginal variant that removes dependence on the previous pitch.Source: §3.4 · p. 6

Automatic Music Transcription

• Piano transcription systems recover note and pedal events but do not directly produce scores; unified approaches include SheetSage, TUTTI, MT3, and MuScriptor.Generalization across diverse recordings remains limited by the availability of aligned annotations (Rouard et al., 2026).Source: §4 · pp. 7–8

Synthetic Supervision and Label Mining

• Large MIDI collections support synthetic transcription supervision and label mining for tasks including structure, chords, and key.Prior examples include synthesized-audio transcription (Hu et al., 2026; Rouard et al., 2026) and content-based self-annotation.Source: §4 · p. 8

Structured Decoding

• MIR systems use DBNs and CRFs for structured decoding, while metrical-level and octave errors remain persistent challenges.SheetSage2 counters these errors with a shared tempo anchor and interval-dependent melody scores.Source: §4 · p. 8

Sequence-Level Distillation

• SheetSage2’s autoregressive student learns from structured teacher outputs without requiring the teacher’s decoders at inference.The training resembles sequence-level distillation (Kim & Rush, 2016).Source: §4 · p. 8

Rhythmic assumptions

• A sixteenth-note grid, one tempo anchor, and a limited meter inventory constrain representation of triplets, finer subdivisions, and large tempo changes.The fixed anchor is suited to songs with a single prevailing tempo.Source: §5 · p. 8

Harmonic assumptions

• The model assumes 12-tone equal temperament and a chord vocabulary that omits extended and altered chords such as ninths..Source: §5 · p. 8

Limited genre generalizability

• The lead-sheet format and training data target popular music with a steady beat, not classical or free-rhythm music.Classical music often requires several voices and expressive tempo, while free-rhythm music lacks a metrical grid.Source: §5 · p. 8

Conclusion

• SheetSage2-AR uses a MERT2-FS encoder to infer rhythm, harmony, melody, and form from audio.It transcribes these dimensions as a unified lead sheet.Source: §6 · p. 8
• The authors argue that expanding synthetic data with high-quality labels is promising while annotated recordings remain scarce.They motivate further advances in symbolic music analysis and generation to produce better, more diverse synthetic supervision.Source: §6 · p. 8

• AI tools helped implement and debug method and experiment code, including inference, evaluation, and analysis scripts.Source: AI Use Statement · p. 8
• AI tools also aided writing, figure creation, and reference searching and formatting.Source: AI Use Statement · p. 8
• The authors reviewed and tested AI-assisted code, edited AI-assisted materials, and take responsibility for the final content.Source: AI Use Statement · p. 9

• Prober combines public symbolic datasets, licensed Hooktheory annotations, a private lead-sheet dataset, and HarmonixSet training data.Source: Impact Statement · p. 9
• SheetSage2-AR training uses primarily CC0 and synthetic music, with most synthetic data supplied under license.Source: Impact Statement · p. 9
• Training and evaluation are dominated by Western tonal popular music, and the model likely serves those styles better than others.The authors warn that tools favoring these styles could further marginalize underrepresented traditions whose tuning, meter, or notation may not fit the lead-sheet format.Source: Impact Statement · p. 9

Reproducibility Statement

• The reproducibility statement directs readers to sections and appendices for event formats, training configurations, decoder recurrences, evaluation settings, and consistency diagnostics.Source: Reproducibility Statement · p. 9
• Model weights, inference code, preset settings, and benchmark scores are available, while some training data are private or licensed and benchmark audio remains under provider terms.Source: Reproducibility Statement · p. 9

A. Complete AR Event Format and Inference Details

• The event language places typed musical attributes in one chronological sequence, with absolute beat timestamps and relative positions on a sixteenth-note grid.Tokens encode meter, beat position, section, key, chord, melody pitch and source, and note duration; timestamps use 10-ms bins, and a notation builder outputs ABC or MIDI.Source: A. Complete AR Event Format and… · p. 13
• Greedy decoding applies a grammar mask, and recordings longer than 300 seconds are split into overlapping windows with prior events carried into the next prefix.Source: A. Complete AR Event Format and… · p. 13

A.1. Actual token sequences and score excerpts

• The appendix provides actual AR predictions and score excerpts, including examples of key and meter changes.The key example changes from G♯minor to C minor; another example shows a temporary 4/4-to-3/4 change.Source: §A.1 · p. 14, Table 6 · p. 14, Table 7 · p. 15

B.1. Training sets

• The multitask Prober trains on 393,980 songs totaling 22,016.7 audio hours, mostly rendered from MIDI, while SheetSage2-AR uses 441,094 pseudo-labeled recordings totaling 28,434.3 hours.The Prober corpus includes 32,755 real recordings; the AR corpus consists primarily of CC0 and synthetic music.Source: §B.1 · p. 17, Table 9 · p. 17

B.2. Pseudo-label bootstrapping

• Pseudo-label bootstrapping adapts symbolic models using seed labels, then applies them to Los Angeles MIDI; chord/key and melody labels use distinct extraction procedures.Chord and key seeds come from expert rules on revealing tracks, while melody seeds use tracks named “melody” and an onset-F1 threshold of 0.7.Source: §B.2 · p. 17

B.3. Models

• The Prober models adapt MERT2-FS encoders with LoRA, while the AR student learns from decoded Prober sequences in a two-stage training schedule.The melody Prober uses beat and source-hint embeddings; AR targets shift timestamps 30 ms earlier, and the AR model uses no pitch augmentation.Source: §B.3 · pp. 17–18

B.4. Prober output channels

• Prober non-melody channels are independent binary outputs, whereas melody outputs share a softmax across each grid position.The chord decoder uses root, chroma, and bass blocks, with tempo and section-boundary predictions also specified.Source: §B.4 · p. 18, Table 11 · p. 18, Figure 4 · p. 19

C. Exact Decoder Recurrences and Ablation Definitions

• Structured decoders maximize sequence scores; rhythm decoding evaluates candidate pulse intervals against tempo evidence, while the DBN-only ablation uses madmom’s joint beat/downbeat postprocessor.The pulse decoder selects a path with start and end constraints; the ablation specifies its activation processing and decoder settings.Source: C. Exact Decoder Recurrences and… · p. 20, §C.1 · p. 20, §C.4 · p. 21

C.2. Meter dynamic program

• The meter dynamic program assigns bar positions to recovered eighth-note pulses, favoring stable meter while allowing penalized local and global changes.Its default inventory contains five meters and 27 meter-position states; local changes accommodate incomplete bars or extra beats.Source: §C.2 · pp. 20–21

C.3. Melody dynamic program; C.5. Key, section, and chord decoding

• Melody decoding uses a sixteenth-note grid and a state tracking the last onset pitch and its age; key, section, and chord decoders operate on pooled measures or beats.Key changes incur a cost, section boundaries use boundary activations, and chord scores compare beat-averaged predictions with templates.Source: §C.3 · p. 21, §C.5 · p. 22

C.6. Spelling correction

• Key-dependent spelling rules resolve enharmonic ambiguity, selecting chord-root candidates by summed line-of-fifths costs around the key signature.The seven key-signature notes have zero cost, costs rise beyond either edge, and the selected spelling does not affect reported scores because evaluation uses sounding pitches or pitch classes.Source: §C.6 · pp. 23–24, Figure 5 · p. 23

D.1. Evaluation datasets

• Beat, downbeat, and melody use recording-level F1 with onset tolerances, key uses a weighted score, and chord recall is weighted by reference annotation span.Key evaluation gives fifth errors in either direction a score of 0.5; melody octave errors are ignored because reliable reference octaves are lacking.Source: §D.1 · p. 25, Table 15 · p. 25
• GTZAN, osu2017, GiantSteps, Chords1217, JAAH, and HarmonixSet supply rhythm, key, chord, and structure cohorts, with recording counts and evaluation tasks varying by collection.The supplied datasets include 999 GTZAN beat recordings, 142 osu2017 recordings, 604 GiantSteps recordings, all 1,217 Chords1217 recordings, 113 JAAH recordings, and 200 HarmonixSet test recordings.Source: §D.1 · p. 25
• RWC-Pop contributes all 100 recordings; Rock Corpus uses 200 vocal recordings, retaining six with no pitched vocal-reference notes at zero F1.On the 194 Rock Corpus recordings with nonempty references, SheetSage1, Prober, and AR score 50.71%, 68.02%, and 69.15%, respectively.Source: §D.1 · p. 25

D.2. Chord metrics beyond major/minor

• On JAAH, AR leads every extended chord metric; on osu2017 and Chords1217, Prober leads root, thirds, maj/min, triads, and MIREX, AR tetrads, and ChordFormer sevenths, with and without inversions.These metrics include inversion variants, which also require the correct bass.Source: Table 16 · p. 26, §D.2 · p. 26

D.3. External Vocal-Melody Transcription Baselines

• 13.59 points on RWC-Pop vocal and 13.14 points on Rock Corpus separate the closest external system, YourMT3+, from SheetSage1; all three external systems score lower on every melody benchmark.The external systems use their vocal output only; their results do not cover full melody because they do not support melody reduction for non-vocal instruments.Source: Table 17 · p. 26, §D.3 · p. 26

D.4. Task-training data ablation

• Synthetic-only predictions follow much of one RWC vocal phrase’s contour despite no paired vocal audio in training, but predict onsets and note lengths poorly.This is a selected phrase example, not a claim about all recordings.Source: §D.4 · p. 28, Figure 7 · p. 29

E.1. Global Tempo Consistency

• On osu2017 and Hooktheory, structured decoding mixes tempo levels less often than P-DBN, and AR largely preserves this advantage; Prober and AR mix less often than Beat This! across all three datasets.Assessable recordings require at least 80% of reference duration classified; Hooktheory excludes 166 recordings whose reference passages already differ in tempo by about a factor of two, and overlaps training data.Source: Figure 8 · p. 30, §E.1 · p. 30

E.2. Melody Octave Consistency

• 18.58–48.95% of units for SheetSage1 versus at most 16.03% for SheetSage2 variants mix octaves; Prober’s advantage is clearest on full melody, while AR varies by dataset.Octave offsets are assessed on onset- and pitch-class-matched notes within each unit; Hooktheory overlaps training data and is not a held-out test.Source: Figure 9 · p. 31, §E.2 · pp. 31–32

E.3. Transcription F1 of the Consistency Ablations

• P-DBN scores higher than Prober on GTZAN but lower on osu2017; melody F1 differs by less than 0.1 points between Prober and pitch-marginal decoding.Direct-pitch Prober raises Rock Corpus vocal F1 from 65.98% to 69.54%, but the authors favor consistency over a single-dataset F1 gain.Source: §E.3 · p. 32, Table 19 · p. 32

F. Lead-Sheet Excerpts from Full-Song Transcriptions

• SheetSage2-AR scores show two melody staves, chord symbols, key signatures, meter, tempo, and predicted section labels; SheetSage1 shows one melody staff and global key, meter, and tempo.The displayed scores are automatic transcriptions; pitches and sounding durations are unchanged by re-spelling at beat boundaries.Source: F. Lead-Sheet Excerpts from Full-Song… · p. 33
• Colored boxes identify selected disagreements in key, melody, chords, and section labels, with colors assigned to specific error categories.Minor errors are omitted; purple marks key disagreement, yellow melody errors, green chord errors, and red an incorrect section label.Source: F. Lead-Sheet Excerpts from Full-Song… · p. 33

F.1. RWC-MDB-P-2001 No. 1

• Both systems establish the correct beat grid in RWC-MDB-P-2001 Nos. 1 and 2.Source: §F.1 · p. 33, §F.2 · p. 33
• For No. 1, SheetSage2-AR follows the verse and chorus modulations but misses later key changes after the solo; SheetSage1 estimates only a global key.Source: §F.1 · p. 33
• For No. 1, SheetSage2-AR’s chords are generally acceptable, while SheetSage1 misses diminished quality in two measures; SheetSage2-AR’s solo melody simplifies subdivisions faster than 16th notes.Source: §F.1 · p. 33
• SheetSage2-AR labels No. 1’s post-solo C-minor passage as a bridge rather than the reference verse; SheetSage1 does not transcribe structure labels.Source: §F.1 · p. 33

F.2. RWC-MDB-P-2001 No. 2

• For No. 2, both systems choose E♭ major, a reasonable estimate given the chord progression; SheetSage2-AR and SheetSage1 make different chord errors, while its section boundaries are largely reasonable.Source: §F.2 · pp. 33–35