Source-linked AI summary
Randomized Scores and Diverse Timbres: Augmenting Automatic Music Transcription with Online-Generated Data
Haiwen Xia, Chao Zhang, Qiuqiang Kong
TL;DR
Precise note annotations are scarce, and it remains unclear whether synthetic AMT transfer depends more on realistic score structure or broad timbral coverage. This paper independently varies note-event sampling and timbral rendering, finding that moderate score corruption preserves useful transfer while broader timbral support improves it.
Problem
Precise note annotations are scarce, and the relative roles of realistic score structure and timbral coverage in synthetic-data transfer remain unclear.
Source: §1 · p. 1Method
An online sampler–renderer framework independently controls note-event distributions and timbral choices, generating audio–symbolic pairs for joint training with offline data.
Source: §2.1 · p. 1, §2.3 · p. 2Results
Takeaways & Limitations
The results support prioritizing note-level attribute coverage and acoustic realizations over reproducing realistic complete scores when designing synthetic AMT data.
Source: §5 · p. 4Online-generation gains become modest when diverse offline supervision is available, and the renderers have coarse labels, keyboard bias, and no common mastering post-processing.
Source: §4.6 · p. 4
Abstract
from arXiv · showhide
Automatic music transcription (AMT) is limited by the scarcity of audio recordings paired with precise symbolic annotations. Synthetic data can provide supervision at scale, but it remains unclear whether effective transfer depends on realistic score structure or broad timbral coverage. We study these factors separately through an online sampler--renderer pipeline. A unified corruption sampler ranges from unmodified MIDI clips through partial corruption to deeply randomized note-event distributions. The renderer converts these events to audio while independently controlling instrument and timbral coverage. A fixed transcription model is trained jointly on offline recordings and newly rendered examples. Controlled ablations reveal an asymmetry between the two factors: moderate corruption of the note-event distribution does not impair transfer and can improve it, whereas broader renderer-side timbral support consistently improves out-of-domain generalization under a fixed note-event distribution. Finally, online-rendered examples complement real and existing synthetic data in a strong combined-data regime. These results suggest that synthetic AMT data should prioritize coverage of note-level attributes and their timbral realizations over realistic joint score structure.
1. INTRODUCTION
Precise note annotations are scarce, motivating synthetic supervision, but prior approaches do not isolate the effects of score structure and timbral coverage. The paper introduces an online sampler–renderer framework to vary them independently and test their transfer effects.
1. INTRODUCTION
- Precise polyphonic note annotation is costly, leaving supervised datasets with limited coverage of musical and acoustic conditions.Synthetic audio paired with symbolic events can expand supervision without manually annotating every recording.Source: §1 · p. 1
- Rendering-based methods expand acoustic coverage while retaining coherent scores, whereas recombination methods preserve structure within source material and change polyphony and timbre together.These approaches leave unclear which score dependencies are necessary and how timbral coverage should be varied independently.Source: §1 · p. 1
- The online sampler–renderer formulation selects note events first, then renders them with chosen timbres, allowing either factor to be held fixed while the other varies.Each generated pair is used once during training alongside fixed offline recordings.Source: §1 · p. 1
- The paper reports that moderate note-event randomization preserves or improves transfer, broader timbral coverage improves out-of-domain generalization, and online data complement existing datasets.It also introduces a simplified multi-channel architecture that shares decoder weights across channels.Source: §1 · p. 1
2. ONLINE SAMPLER–RENDERER DATA GENERATION
The data-generation framework separates symbolic note-event sampling from timbral rendering, enabling controlled variation of score structure and acoustic realization. Online pairs are rendered once and used alongside offline data.
2.1. Problem Formulation
- Online training pairs are generated by independently sampling note events y and rendering them as audio x; fixing either control isolates score structure from timbral coverage.Each pair is rendered on the fly and used once, unlike fixed offline pairs that are revisited each epoch.Source: §2.1 · p. 1, Figure 1 · p. 2
2.2. Symbolic Sampler
- The symbolic sampler corrupts source MIDI notes using a distribution that retains each note’s program while sampling pitch and duration from corpus marginals and onset and velocity uniformly.Uniform duration draws are avoided because excessive long notes would make transcription ill-posed.Source: §2.2 · p. 2
- Source, Corrupted, and Note-factorized settings form one sampler family, ranging from unchanged clips to independent redraws of every note’s non-program attributes.Corruption rate ρ controls the fraction of notes altered, while interpolation strength s controls how far altered notes move from their originals.Source: §2.2 · p. 2
2.3. Audio Renderer and Timbral Coverage
- The renderer samples a compatible timbral choice for each symbolic target, so widening the supported choices increases timbral coverage without changing the symbolic distribution.Choices include compatible NSynth recordings for note-wise rendering or a Pianoteq preset for stem rendering.Source: §2.3 · p. 2
2.3.1. Note-renderer
- The note-wise renderer maps each symbolic note to a compatible NSynth recording, aligns its sounding start to the target onset, and applies a smooth decay when shortening it.Local per-note operations allow notes to be processed in parallel with low overhead.Source: §2.3.1 · p. 2
2.3.2. Stem renderer
- The stem renderer samples a compatible Pianoteq preset for a complete MIDI stem, varying coupled physical-model timbre parameters while keeping a coherent realization across notes.Controlled piano experiments compare 4%, 20%, and 100% of program-0 presets, then presets spanning programs 0–7; rendered pairs share a fixed offline-to-online batch ratio and equal loss weights.Source: §2.3.2 · p. 2
3. AUTOREGRESSIVE TRANSCRIPTION MODEL
The paper uses a compact encoder–decoder transcription model with instrument-class channels and shared decoder weights. All channels use the same autoregressive training objective, regardless of whether examples are online or offline.
3. AUTOREGRESSIVE TRANSCRIPTION MODEL
- A convolutional front end and temporal Transformer encoder produce a shared representation that is projected into instrument-class streams for autoregressive decoding.The model replaces YourMT3+’s specialized Perceiver TF encoder with a simpler stack and is held fixed within each data-generation comparison.Source: §3 · p. 2
3.1. Encoder and Channel Split
- The encoder projects shared frame features into C equal-width streams, one per channel, after convolutional and temporal Transformer processing.The convolutional blocks preserve temporal resolution by pooling only along frequency.Source: §3.1 · p. 2, Figure 2 · p. 3
3.2. Multi-Channel Decoder
- Channels are decoded in parallel with shared token embeddings, decoder-block parameters, and output head, while each channel attends to its own encoder stream and autoregresses over its own prefix.In piano-only experiments, only the piano channel is active.Source: §3.2 · p. 3
3.3. Training Objective
- Online and offline examples share the autoregressive token cross-entropy objective; their provenance affects batch composition, not the loss.For sources without an exact program label, probability mass is aggregated over the corresponding instrument family.Source: §3.3 · p. 3
4. EXPERIMENTS
Controlled experiments show that corrupted note sampling can retain transfer, while broader timbral coverage improves it. Online additions to strong offline data help selectively, with gains varying by domain.
4.1. Experimental Setup
- Experiments hold the architecture fixed while varying online-only sampling and timbral coverage or combining online examples with offline audio, across piano and multi-instrument tests.The evaluation reports Agn or Prog onset F1; MAPS is Agn-only and RWC averages five subsets.Source: §4.1 · p. 3
4.2. Randomized Scores Remain Informative
- Corrupted sampling stayed close to Source on multi-instrument sets and improved MAPS; s = 0.5 led RWC and Slakh, s = 1 led MAPS, while Source led MulTTiPop.MulTTiPop’s shared Lakh MIDI origin with Slakh training clips may explain the Source advantage there.Source: §4.2 · p. 3, Table 1 · p. 3
4.3. Timbral Coverage Improves Transfer
- Increasing Pianoteq preset coverage monotonically improved both datasets; adding programs 0–7 raised MAESTRO and MAPS by 1.80 and 2.43 points, respectively.The expanded setting scored 78.13 versus 77.14 for real-MAESTRO training on MAPS; real audio remained strongest on MAESTRO.Source: §4.3 · p. 3, Table 2 · p. 4
4.4. Combining Offline and Online Supervision
- Despite a 26-point Slakh gap, offline and online training were within one point on all three out-of-domain sets; joint led out-of-domain Prog/RWC, while two-stage led MAPS.The compared strategies were offline, online, joint, and two-stage training.Source: Table 3 · p. 4, §4.4 · p. 4
4.5. Strong Offline–Online Regime
- +NSynth improves MulTTiPop Agn by 0.76 points; +Mixed improves MAPS by 0.41 and RWC by 1.32/0.96 Agn/Prog points, but reduces both MulTTiPop scores.Online supervision is beneficial but domain dependent.Source: Table 4 · p. 4, §4.5 · p. 4
- +Mixed improved RWC Agn onset F1 by 1.07–1.80 points across all five subsets, while Bass improved by 5.3–6.1 points in four non-classical subsets.RWC Prog improved in four subsets, ranging from −0.74 to +2.48 points.Source: §4.5 · p. 4
- The offline-only model was competitive with YourMT3+ on MAPS and MulTTiPop despite replacing Perceiver TF with a standard Transformer encoder.It remained competitive with Muscriptor-Large on MAPS and RWC Prog, but trailed on MulTTiPop and RWC Agn.Source: §4.5 · p. 4
4.6. Analysis and Limitations
- Online-generation gains become modest with diverse offline supervision; coarse NSynth labels, Pianoteq’s keyboard bias, and omitted mastering constrain the approach, which is not a replacement for diverse paired recordings.Source: §4.6 · p. 4
5. CONCLUSION
The framework separates symbolic structure from timbral realization, and its simplified multi-channel architecture achieves competitive results without a specialized encoder. Experiments show that moderate relaxation of score structure retains useful supervision, broader timbral coverage improves transfer, and online-generated data complement offline training, though gains are modest and domain dependent.
5. CONCLUSION
- Broader timbral coverage consistently improves transfer, while online-generated data complement offline training with modest, domain-dependent gains.The results support prioritizing note-level attributes and their acoustic realizations over realistic complete scores.Source: §5 · p. 4
- Moderately relaxing score-level structure retains useful supervision.The conclusion recommends prioritizing note-level attributes over reproducing realistic complete scores.Source: §5 · p. 4
- A simplified multi-channel architecture attains competitive results without a specialized encoder.Source: §5 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The work is a computational study using existing datasets and synthetically generated audio, and no ethical approval was required.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The computational study used existing datasets and synthetically generated audio, requiring no ethical approval.Source: §7 · p. 5