PaperlayerView exact revision on arXiv

Source-linked AI summary

MT3: Multi-Task Multitrack Music Transcription

Josh Gardner, Ian Simon, Ethan Manilow, Curtis Hawthorne, Jesse Engel

arXiv:2111.03017v4cs.SDcs.LGeess.AS

TL;DR

Multi-instrument AMT must recover notes, timing, pitch, and instrument identity despite challenging mixtures and limited labeled data. MT3 uses a unified sequence-to-sequence Transformer framework across diverse datasets, achieving state-of-the-art multitrack transcription and especially large gains for low-resource datasets while preserving strong high-resource performance. The paper also standardizes evaluation and identifies scope boundaries for its representation.

  • Problem

    AMT must transcribe multiple instruments from mixtures with fine pitch and timing, while many datasets are small and existing metrics do not jointly assess note and instrument accuracy.

  • Method

    MT3 uses a compact MIDI-like token vocabulary and sequence-to-sequence Transformer trained jointly on six multitrack datasets with differing instrument combinations.

  • Results

    MT3 achieves state-of-the-art transcription performance on each multitrack dataset and up to 260% relative gain for low-resource datasets while preserving strong high-resource performance.

  • Takeaways & Limitations

    The work provides a unified baseline for multi-task multitrack AMT alongside consistent note-and-instrument evaluation across datasets.

  • Takeaways & Limitations

    The system is limited to Western-tradition music whose pitches map to 12 discrete pitch classes, excluding many non-Western traditions and microtonal genres.

Abstract

from arXiv · show

Automatic Music Transcription (AMT), inferring musical notes from raw audio, is a challenging task at the core of music understanding. Unlike Automatic Speech Recognition (ASR), which typically focuses on the words of a single speaker, AMT often requires transcribing multiple instruments simultaneously, all while preserving fine-scale pitch and timing information. Further, many AMT datasets are "low-resource", as even expert musicians find music transcription difficult and time-consuming. Thus, prior work has focused on task-specific architectures, tailored to the individual instruments of each task. In this work, motivated by the promising results of sequence-to-sequence transfer learning for low-resource Natural Language Processing (NLP), we demonstrate that a general-purpose Transformer model can perform multi-task AMT, jointly transcribing arbitrary combinations of musical instruments across several transcription datasets. We show this unified training framework achieves high-quality transcription results across a range of datasets, dramatically improving performance for low-resource instruments (such as guitar), while preserving strong performance for abundant instruments (such as piano). Finally, by expanding the scope of AMT, we expose the need for more consistent evaluation metrics and better dataset alignment, and provide a strong baseline for this new direction of multi-task AMT.

1 INTRODUCTION

MT3 addresses multi-instrument transcription by combining a flexible sequence-to-sequence framework with diverse datasets and consistent evaluation. The resulting model achieves strong performance across datasets, with especially large gains for low-resource settings.

  • Motivation: MT3 targets arbitrary combinations of instruments across datasets, addressing the difficulty of separating notes, pitch, timing, and instrument identity from mixtures.The introduction identifies arbitrary multi-dataset transcription as an unresolved capability.
  • Motivation: Small AMT datasets create a low-resource setting: existing open-source collections contain one to a few hundred hours, versus 1k hours for LibriSpeech and 9k+ for CommonVoice.LibriSpeech alone exceeds the combined audio hours of all AMT datasets used in the paper.
  • Framework: A compact, flexible tokenization scheme converts between model tokens and multitrack MIDI, enabling sequence-to-sequence training across datasets with different instrument combinations.This permits simultaneous use of datasets previously trained in isolation because of instrumentation differences.
  • Evaluation and data: The authors assemble six diverse multitrack AMT datasets and define consistent splits and note-based metrics, including an instrument-sensitive metric for joint note and instrument accuracy.Together, the datasets form the largest known publicly available collection for multi-task AMT training.
  • Results: MT3 achieves state-of-the-art transcription performance on each multitrack dataset, outperforming prior dataset-specific models and professional-quality DSP software.It also maintains high instrument-labeling accuracy with many simultaneous instruments and is robust to instrument grouping.
  • Results: Up to 260% relative gain occurs for low-resource datasets under joint training, while strong performance is preserved on high-resource datasets.The result supports multi-task training as particularly beneficial for low-resource AMT.

2 RELATED WORK

Prior work largely specialized transcription models for particular instruments or fixed instrument sets. MT3 extends Transformer sequence-to-sequence modeling toward arbitrary polyphonic multitrack transcription using broadly off-the-shelf components.

  • Transformer foundations: Transformer models have supported unified text-to-text and multilingual sequence modeling, motivating their extension to multi-task music transcription.T5 unified language tasks, while mT5 used one Transformer across languages and leveraged high-resource data for lower-resource performance.
  • Audio applications: Transformers have also been applied to audio classification, captioning, compression, speech recognition, separation, enhancement, and generative audio modeling.These applications establish broader audio sequence-modeling precedent without directly solving multitrack AMT.
  • Prior AMT: Music transcription research historically emphasized solo piano, producing architectures and representations tailored to piano-specific transcription.Examples include dedicated onset and note outputs and amplitude-envelope representations.
  • Prior AMT: Existing multi-instrument systems addressed fixed instrument sets, source separation, queried instruments, synthesis, or low-resource transcription without predicting instrument identities.These constraints differ from MT3’s arbitrary-instrument, multitrack formulation.
  • MT3 positioning: MT3 extends an encoder-decoder Transformer from solo piano to polyphonic music with an arbitrary number of instruments while retaining spectrogram inputs, standard T5 configuration, and MIDI-like events.The design follows an off-the-shelf component philosophy.

3 TRANSCRIPTION MODEL

MT3 frames multi-instrument music transcription as sequence-to-sequence prediction, using a Transformer and MIDI-like vocabulary to map audio segments to symbolic events. The shared vocabulary supports diverse instruments and datasets, while segmentation and tie handling address long inputs and cross-segment notes.

  • Sequence-to-sequence formulation: MT3 treats multi-instrument transcription as sequence-to-sequence prediction from audio-frame sequences to symbolic note tokens.
  • Transformer architecture: A generic T5 encoder-decoder Transformer replaces dataset-specific architectures for varied transcription tasks.
  • Model inputs and outputs: Log Mel spectrograms are mapped to a MIDI-like vocabulary that adds instrument changes and ties while removing velocity.
  • Model inputs and outputs: The vocabulary represents instruments, pitches, note onsets or offsets, quantized times, drum events, ties, and sequence termination.
  • Segmented processing: Audio is split into non-overlapping segments because Transformer self-attention requires O(n^2) memory, and each segment is processed independently.
  • Segmented processing: Tie sections declare notes already active at a segment boundary, helping preserve notes that span independently processed segments.
  • Multi-task mixture: A shared vocabulary enables simultaneous training on mixtures of datasets with different instrumentations, while temperature sampling increases exposure to low-resource datasets.

4 EXPERIMENTS

The experiments evaluate MT3 across six diverse datasets, standardized transcription metrics, instrument-granularity settings, and leave-one-dataset-out generalization. MT3 outperforms prior systems across datasets and gains substantially from mixture training on low-resource datasets.

  • Datasets: The experiments cover six datasets spanning different sizes, recording processes, instrumentations, and genres, including three low-resource datasets with 1.3–34 hours of audio.GuitarSet, MusicNet, and URMP contain 3, 34, and 1.3 hours, respectively.
  • Evaluation: The evaluation uses Frame F1, Onset F1, Onset-Offset F1, and multi-instrument F1 to measure note and instrument-label accuracy.Multi-instrument F1 adds instrument agreement to the Onset-Offset F1 requirements.
  • Results: MT3 exceeds state of the art on all six datasets across Frame, Onset, and Onset+Offset F1 metrics.The comparisons include dataset-specific models designed, trained, and tuned for individual datasets.
  • Results: 54% and 263% Onset-Offset F1 gains occur on MusicNet and URMP, respectively, when using mixture training rather than single-dataset training.The mixture formulation particularly benefits the resource-limited multi-instrument datasets.
  • Instrument granularity: MT3 makes few instrument-label errors when onset and offset predictions are correct, even under the Full instrument grouping.Multi-instrument F1 scores are close to the corresponding onset-offset F1 scores.

5 CONCLUSION AND FUTURE WORK

The paper frames multi-instrument transcription as sequence-to-sequence modeling with a generic Transformer trained across datasets, advancing performance especially in low-resource settings. It also establishes consistent onset+offset+instrument evaluation and identifies unlabeled learning and augmentation as future directions.

  • Conclusion: A generic Transformer trained simultaneously across datasets advances state-of-the-art multi-instrument transcription, especially for low-resource data.The approach poses multi-instrument transcription as a sequence-to-sequence task.
  • Conclusion: The proposed evaluation uses note onset+offset+instrument F1 scores with a standard instrument taxonomy.
  • Future work: Future work includes self- or semi-supervised learning, data augmentation, and using high-quality transcriptions for symbolic music generation.These directions respond to the expense of labeled multi-instrument transcription data.

7 ETHICAL CONSIDERATIONS

The system is limited to Western-tradition music represented with discrete notes from 12 pitch classes, excluding music without a well-defined mapping to that representation.

  • Scope limitation: The system is trained on and applicable only to Western-tradition music with discrete 12-pitch-class representations.Non-Western traditions such as Indian ragas and Arabic maqams, and microtonal Western genres such as blues, fall outside this scope.

A MODEL AND TRAINING DETAILS

MT3 uses a standard T5-small Transformer implementation and different training durations for mixture, dataset-specific, and leave-one-dataset-out models.

  • Model: MT3 uses the T5-small Transformer architecture with T5.1.1 modifications implemented in t5x, FLAX, and JAX.
  • Training: Mixture models train for 1M steps, while dataset-specific and leave-one-dataset-out models train for 219 steps.The shorter schedules reflect faster convergence or computational constraints.

B DATASET DETAILS

The paper assembles several piano, guitar, classical, synthesized, and multi-instrument datasets, using standard or explicitly documented custom splits. Additional preprocessing, filtering, and augmentation address inconsistent dataset formats and expand training examples.

  • MAESTRO: MAESTRO provides 198.7 hours of piano performances with approximately 3 ms alignment between note labels and audio.Its standard split contains 962 training, 137 validation, and 177 test performances, with no instruments besides piano.
  • Slakh2100: Slakh2100 renders 2100 Lakh MIDI files with piano, bass, guitar, and drums using professional-quality virtual instruments.Training uses ten random subsets containing at least four instruments per file, expanding examples tenfold.
  • Cerberus4: Cerberus4 extracts exactly one piano, guitar, bass, and drums subset from each Slakh2100 MIDI file, producing 1327 tracks and 542.6 hours.It retains Slakh2100’s train, validation, and test partitioning.
  • GuitarSet: GuitarSet contains 360 guitar excerpts from six guitarists performing 30 lead sheets in comping and soloing versions across five styles, three progressions, and two tempi.Because no official split exists, the paper assigns progressions to training and validation, yielding 478 training and 238 validation tracks.
  • MusicNet and URMP: MusicNet contains 330 annotated classical recordings, while URMP combines isolated performer recordings into coordinated multi-instrument pieces spanning 14 instruments.MusicNet receives a custom train/validation/test split, and URMP’s validation split preserves varied duet, trio, quartet, and quintet instrumentation.
  • Annotation caveats: The paper flags potential label-quality issues in both MusicNet and URMP and discusses them in Appendix D.These concerns accompany the datasets’ annotation and splitting procedures.

C BASELINE DETAILS

The paper compares MT3 with an original Cerberus model, ReconVAT, and Melodyne. Melodyne provides a professional software baseline, but its proprietary algorithms and limited evaluation subsets constrain detailed comparison.

  • Cerberus: The Cerberus baseline uses the authors’ original four-instrument model trained on Slakh-redux with the published cropping, active-instrument filtering, and Slakh splits.The four instruments are guitar, piano, bass, and drums.
  • ReconVAT: ReconVAT evaluation uses the authors’ pretrained models and inference script, with MusicNet audio divided into 20-second segments because of resource limitations.Segments without active ground-truth notes are discarded because mir_eval metrics are undefined without reference notes.
  • Melodyne: Melodyne Studio 5.1.1.003 supplies a professional audio-software baseline with polyphonic sustain and polyphonic decay pitch-tracking algorithms.The paper applies it to raw WAV files from each dataset.
  • Evaluation scope: Melodyne evaluation uses random subsets of 30 test tracks for MAESTRO, Slakh10, Cerberus4, and GuitarSet, but the entire test sets for URMP and MusicNet.This uneven coverage reflects the manual effort required for large-scale transcription.
  • Evaluation scope: Because Melodyne is proprietary and lacks a programmatic interface, the paper cannot provide further details about its exact transcription algorithms.This limits reproducibility of the software baseline’s internal method.

D.1 EVALUATING ZERO-SHOT GENERALIZATION WITH LEAVE-ONE-DATASET-OUT

Leave-one-dataset-out experiments train MT3 without one dataset and test on the held-out data, assessing zero-shot generalization across differing audio, styles, and instrumentation. MT3 achieves nontrivial unseen-dataset transcription, but performance depends on instrument coverage in training data.

  • LODO procedure: LODO experiments remove one dataset from the mixed training set, then evaluate MT3 on datasets included in the full collection.Slakh2100 and Cerberus4 are jointly included or excluded because they share stems and synthesis software.
  • Generalization results: MT3 achieves nontrivial note prediction on most unseen datasets despite large differences in sonic qualities, compositional styles, and instrumentation.The reported LODO onset F1 scores range from 0.14 to 0.78.
  • Generalization results: MT3’s multi-instrument and onset-offset F1 scores outperform baseline models on the low-resource GuitarSet and URMP datasets.This result is reported within the LODO evaluation across the study’s datasets.
  • Generalization limits: The LODO results show sensitivity to omitting particular datasets, especially when those datasets provide unique instrument coverage.Without Slakh2100 and Cerberus4, MT3 cannot learn to identify bass and synthesizer because their combination uniquely contains those instruments.

D.2 ONSET-OFFSET THRESHOLD SENSITIVITY ANALYSIS

The paper examines onset-offset metric sensitivity to timing tolerance and discusses label errors in MusicNet and URMP. It also documents dataset-specific instrument mappings used for evaluation.

  • Label quality: Potential onset and offset labeling errors are observed particularly in MusicNet and URMP, where timing noise may affect model learning and generalization.The default correct-prediction threshold is only 50 ms, so small timing errors can substantially increase evaluation difficulty.
  • Threshold sensitivity: Figure 4 varies the onset-offset tolerance from 10 ms to 500 ms to assess MT3 performance under different timing thresholds.The default 50 ms threshold is marked by a dashed line.
  • Instrument mappings: MIDI program numbers are grouped into MIDI classes using floor(p/8), while Slakh datasets use synthesis-oriented instrument classes mapped to MT3 instrument tokens.The paper provides the corresponding mappings in Tables 7 and 8.
  • Figure assets: The appendix documents the instrument-icon sources used in Figure 1 under Creative Commons licensing.The listed icons cover instruments including bassoon, flute, violin, piano, viola, cello, bass, guitar, drums, oboe, clarinet, and French horn.
Loading 2111.03017v4…