Source-linked AI summary
TUTTI: Toward generalizable audio-to-score transcription via fully synthesized data
Jianhuai Hu, Yashan Wang, Shangda Wu, Zhancheng Guo, Shijie Liang, Wuna Meng, Chuanqi Yang, Xiaobing Li, Feng Yu, Maosong Sun
TL;DR
TUTTI addresses the scarcity of high-quality real-world audio-score pairs and the complexity of specialized acoustic front-ends with synthetic multi-instrumentation pre-training and a standard Transformer. It achieves state-of-the-art results across diverse tasks and transfers to unseen instruments, including saxophone.
Problem
High-quality real-world audio-score pairs are scarce and expensive to annotate, limiting generalizable A2S transcription across diverse musical compositions.
Method
TUTTI pre-trains a pure Transformer encoder-decoder on over 363,000 synthetic multi-instrumentation audio-score pairs and directly transcribes spectrograms into ABC notation.
Results
TUTTI achieves state-of-the-art performance across diverse tasks and transfers to unseen real-world instruments such as saxophone.
Takeaways & Limitations
Synthetic multi-instrumentation data can support a unified A2S model that generalizes across instrumentations, including instruments absent from pre-training.
Takeaways & Limitations
The authors note that CNN-based hybrid models might also gain from pre-training at the same scale, limiting the attribution of improvements specifically to the Transformer architecture.
Abstract
from arXiv · showhide
Generalizable Audio-to-Score (A2S) transcription is fundamentally constrained by the severe scarcity of high-quality, real-world paired data. Relying solely on existing human-annotated datasets often restricts the generalization of A2S models, limiting their efficacy primarily to single-instrumentation domains. To break this dependency on scarce real-world data, we introduce TUTTI (Transformer for Unified audio-To-score Transcription trained on Synthetic multi-Instrumentation Data), a pre-training paradigm driven by a purely synthetic, large-scale dataset. Rather than using human-composed scores, we leverage a symbolic music generation model to generate a massive, highly scalable multi-instrumentation corpus and create audio-score pairs with expressive acoustic characteristics. Capitalizing on the generated data, we employ a standard Transformer encoder-decoder architecture. We empirically demonstrate that pre-training a unified attention-based model on generated, multi-instrumentation data yields a consistently stronger foundational representation than single-instrumentation training. When fine-tuned with downstream real-world datasets, TUTTI outperforms previous approaches, establishing new overall state-of-the-art results across various A2S baselines. Notably, TUTTI shows remarkable cross-instrument transferability, effectively adapting to unseen instruments with highly competitive performance. The source code and the TuttiCorpus dataset will be made publicly available at https://github.com/a-musiclover/TUTTI.
1. INTRODUCTION
A2S transcription is limited by scarce aligned real-world data, fragmented single-instrument models, and complex task-specific architectures. TUTTI addresses these constraints with a unified Transformer trained on synthetic multi-instrumentation data.
- High-quality, perfectly aligned real-world audio-score pairs are rare and expensive to annotate, creating a central data bottleneck for A2S research.
- Existing A2S models mainly target single-instrument scenarios, leaving unified transcription across varied instrumentations unresolved.
- CNN-RNN or attention hybrid pipelines reduce overfitting in data-scarce settings but increase architectural complexity through specialized acoustic front-ends.
- TUTTI treats A2S as audio-to-text translation, directly decoding spectrogram features into ABC notation with a standard Transformer encoder-decoder.
- TuttiCorpus contains over 363,000 synthetic multi-instrumentation audio-score pairs for learning complex polyphonic structures through a generative pipeline.
- Multi-instrumentation pre-training outperforms single-domain training and transfers to unseen saxophone recordings absent from pre-training data.
2. RELATED WORK
Earlier A2S systems decomposed transcription into subtasks or used end-to-end hybrid architectures, while data scarcity continued to restrict generalization. Existing models therefore remained largely task-specific and instrument-limited.
- Early A2S methods separated multi-pitch estimation and rhythm quantization, but errors could accumulate across pipeline stages.
- End-to-end A2S commonly uses CNN front-ends for local acoustic features and RNN decoders for musical symbols, with CTC or Seq2Seq training.
- Hierarchical decoders and expressive-performance pre-training were introduced to model score structure and reduce reliance on scarce real-world recordings.
- Despite these advances, end-to-end models remained constrained by limited diverse data and the architectural complexity of CNN acoustic front-ends.
3. METHODOLOGY
TUTTI uses a Transformer encoder-decoder to translate continuous audio spectrograms into structured ABC notation. Its hierarchical decoder aligns global musical patches with acoustic segments before generating note-level characters.
- TUTTI adapts hierarchical bar-stream patching and reduced, interleaved ABC notation for cross-modal audio-to-score transcription.
- The Transformer encoder-decoder directly ingests audio spectrograms and autoregressively generates hierarchical symbolic patches.
- 3.2 Model Architecture: The acoustic adapter projects each 1025-dimensional spectrogram frame into a 512-dimensional embedding while preserving the temporal sequence.
- 3.2 Model Architecture: The acoustic encoder extracts local transient events and global structural dependencies from the continuous spectrogram embeddings.
- 3.2 Model Architecture: The patch-level decoder aligns bar-level pacing and polyphony with acoustic segments, while the character-level decoder generates pitches, durations, and structural symbols.
4. DATASET
TuttiCorpus is generated, normalized, expressively rendered, aligned, and segmented into a large multi-instrumentation corpus. Its long-tailed mixture of solo, small-group, and ensemble configurations provides substantial structural diversity.
- NotaGen generates 363,610 unique ABC scores conditioned on musical periods, composers, and instrumentations.
- 4.1.2 Expressive Performance Rendering (EPR): Expressive rendering combines strict quantization, piano-specific VirtuosoNet performances, and rule-based perturbations for other instruments.
- SFZ-based synthesis uses pitch ranges to select acoustic samples, mixes tracks with linear overlay, and normalizes audio to -23.0 LUFS.
- DualDTW aligns expressive MIDI performances with quantized MusicXML scores, enabling bar-level anchors for aligned audio and ABC segmentation.
- The corpus contains 363,610 unique and fully aligned audio-score pairs with broad instrumentation and compositional diversity.
- 4.2 Data Statistics: Solo configurations comprise 64.6% of the data, duets and trios nearly 29%, and about 24,000 instances contain quartets, quintets, or larger ensembles.
5. EXPERIMENT
Experiments evaluate TUTTI’s synthetic pre-training, multi-instrumentation learning, and performance against task-specific baselines across several A2S datasets. Results show that large-scale, diverse pre-training is essential for Transformer generalization and supports strong transfer to unseen instruments.
- 5.1 Settings: TUTTI is evaluated using a 95%/5% TuttiCorpus train-test split and comparisons with open-source models fine-tuned on identical datasets.The evaluation uses MV2H, averaging multi-pitch detection, voice separation, metrical alignment, harmonic detection, and note value accuracy.
- 5.2 Ablation Studies: Without large-scale synthetic pre-training, target-only training severely degrades and falls behind baseline models across all tasks.The result supports the necessity of synthetic pre-training for generalizing a data-hungry Transformer on limited real-world data.
- 5.2 Ablation Studies: Full-instrumentation pre-training outperforms matched single-instrumentation pre-training, indicating stronger representations for instrument-specific downstream tasks.The authors associate this advantage with learning robust acoustic features and harmonic relationships from diverse multi-timbral polyphony.
- 5.3 Comparative Evaluations: TUTTI outperforms all baselines across polyphonic and monophonic tasks, including a 7.4-point multi-pitch improvement on ASAP.The comparison uses MV2H scores across multiple datasets and instrumentation settings.
- 5.3 Comparative Evaluations: On Quartets, TUTTI raises the total score from 84.9 to 95.6, with voice and note value scores of 99.5 and 99.7.These results indicate reliable transcription of dense multi-track arrangements.
- 5.3 Comparative Evaluations: On unseen saxophone datasets, fully pre-trained TUTTI achieves 100.0 voice accuracy, unlike task-specific and non-pre-trained variants.Saxophones are absent from TuttiCorpus, making this comparison a test of transfer to an unseen instrument.
6. CONCLUSION
The paper presents TUTTI as a unified Transformer A2S framework trained on more than 363,000 synthetic multi-instrumentation audio-score pairs. It reports state-of-the-art results and transfer to unseen instruments, while identifying unconstrained full-piece transcription as future work.
- 6. CONCLUSION: TUTTI addresses data scarcity and architectural complexity with a unified Transformer encoder-decoder pre-trained on TuttiCorpus.TuttiCorpus contains over 363,000 synthetic multi-instrumentation audio-score pairs.
- 6. CONCLUSION: TUTTI outperforms established task-specific baselines across diverse polyphonic and monophonic tasks, establishing new overall state-of-the-art results.The conclusion also reports theoretical voice accuracy on unseen real-world instruments such as saxophone.
- 6. CONCLUSION: Extending TUTTI to unconstrained full-piece transcriptions remains a promising direction for future work.The current conclusion frames this as an avenue for extending the mechanism’s scope.