PaperlayerView exact revision on arXiv

Source-linked AI summary

MuScriptor: An Open Model for Multi-Instrument Music Transcription

Simon Rouard, Michael Krause, Axel Roebel, Carl-Johann Simon-Gabriel, Alexandre Défossez

arXiv:2607.08168v2cs.SDcs.LG

TL;DR

Multi-instrument transcription across diverse genres remains challenging because existing models struggle with complex real music mixes and synthetic-to-real domain shift. MuScriptor combines synthetic pre-training, real-audio fine-tuning, reinforcement-learning post-training, and instrument conditioning, yielding an open-weight model that outperforms a state-of-the-art baseline by a wide margin.

  • Problem

    General-purpose transcription for multiple instruments and diverse genres remains challenging because models must handle varied timbres, overlapping frequencies, audio effects, and synthetic-to-real domain shift.

  • Method

    MuScriptor uses a decoder-only transformer with synthetic pre-training, real-audio fine-tuning, reinforcement-learning post-training, and optional instrument-presence conditioning.

  • Results

    MuScriptor outperforms a state-of-the-art baseline by a wide margin, with each training stage improving results over the previous one.

  • Takeaways & Limitations

    The released open-weight model provides configurable general-purpose multi-instrument transcription for real music across diverse genres.

  • Takeaways & Limitations

    The tokenization scheme cannot represent concurrent notes with the same instrument and pitch, requiring overlapping notes to be removed during evaluation.

Abstract

from arXiv · show

Existing methods for automatic music transcription are often limited to single-instrument recordings or fail on complex, real music mixes. Although previous work utilizes synthetic training data, the resulting models generalize poorly, leading to largely unusable transcription output in realistic, multi-instrument settings. In this work, we analyze the effectiveness of synthetic data for pre-training while combining it with fine-tuning on real music audio and post-training using reinforcement learning. We further introduce conditioning on instrument presence to customize transcriptions. Finally, we release MuScriptor, an open-weight multi-instrument music transcription model that works on real-world music recordings from across a diverse range of musical genres.

1. INTRODUCTION

MuScriptor addresses the challenge of general-purpose multi-instrument transcription by combining large-scale real annotated recordings with synthetic pre-training, reinforcement-learning post-training, and instrument-presence conditioning. The paper releases an open-weight model for diverse musical styles and analyzes synthetic-data training.

  • Problem: General-purpose multi-instrument transcription remains challenging because diverse music requires handling complex recordings across musical genres.Existing models are often too error-prone for downstream applications, despite potential uses in musicology, generative modeling, and music information retrieval.
  • Synthetic-data limitations: Synthetic data improves pre-training but is insufficient for general-purpose transcription because synthesized MIDI differs from professional music productions.Prior work combined approximately 1500 hours of synthetic data with only 250 hours of mostly single-instrument real recordings, with performance often degrading beyond synthetic test sets.
  • Contributions: The authors collect 170k real music recordings with aligned note annotations across genres and release the open-weight MuScriptor model.The training data contains 11k hours of recordings and is paired with a synthetic dataset of 1.45M MIDIs.
  • Training and analysis: MuScriptor uses a simple decoder-only transformer and compares real-data-only training with synthetic pre-training followed by real-data fine-tuning.The models are also post-trained with reinforcement learning on 300 high-quality transcribed pieces.
  • Instrument-conditioned transcription: Optional instrument-presence conditioning stabilizes predictions across recording segments and lets users customize transcription output.This conditioning is presented as a contribution alongside the open-weight model, synthetic-data analysis, and reinforcement-learning alignment.

2. RELATED WORK

Automatic music transcription evolved from specialized single-instrument models toward generalized sequence-to-sequence systems, but real-world performance remains limited by synthetic-data reliance. MuScriptor instead emphasizes synthetic–real data relationships and introduces reinforcement learning for multi-instrument transcription.

  • Architectural evolution: AMT progressed from single-instrument architectures using traditional methods toward recurrent and convolutional neural networks.Early systems focused especially on solo piano and used hidden Markov models, non-negative matrix factorization, or support vector machines.
  • Sequence-to-sequence transcription: Sequence-to-sequence models recast transcription as language modeling by autoregressively predicting discrete MIDI-like events.For multi-instrument transcription, MT3 uses an encoder-decoder Transformer to output tokens for pitch, timing, and instrument class.
  • Limitations of existing systems: MT3 established a strong baseline, but its synthetic-data mixture causes deteriorating results on real data.MT3 combines instrument-specific and multi-instrument synthesized training data.
  • Limitations of existing systems: YourMT3+ improves MT3 architecturally with hierarchical attention, a Mixture of Experts layer, and cross-dataset stem augmentation, yet still performs poorly on real-world datasets.Despite these refinements, MT3 remains a strong baseline that few approaches have managed to surpass.
  • MuScriptor’s distinction: MuScriptor differs from architectural refinements by focusing on synthetic and real training data, while introducing GRPO reinforcement learning for high-quality transcription alignment.Prior synthetic-data studies addressed single-instrument cases, and a general-purpose model was smaller-scale and did not release final weights.

3. METHOD

MuScriptor uses a decoder-only Transformer to autoregressively transcribe 5-second mel-spectrogram excerpts into MIDI-like tokens, optionally conditioned on target instruments. Its training combines synthetic and real music data with reinforcement-learning post-training, while evaluation uses instrument-agnostic note metrics.

  • Datasets: Training combines roughly 1.45 million synthetic MIDI files with 170 000 real recordings totaling over 11 000 hours, while DTest emphasizes pieces with multiple active instruments.Tracks with titles similar to DTest are removed from training sets to ensure a clean split.
  • Model architecture: MuScriptor uses decoder-only Transformers with 60M, 100M, 300M, and 1.3B parameters to transcribe 5-second, 16kHz mono audio excerpts.The model receives a mel-spectrogram and a list of target instruments.
  • Instrument conditioning: Instrument embeddings are concatenated with the projected mel-spectrogram as prefix conditioning, enabling customizable instrument-specific transcription.At inference, optional conditioning can transcribe only selected instruments and stabilize assignments across segment boundaries.
  • Evaluation: Evaluation uses Onset F1 and Offset F1, with onset matching requiring pitch agreement and onset error within ±50 ms, while Offset F1 also constrains note duration.Offset tolerance is max(50 ms, 0.2 × dref), where dref is the reference-note duration.
  • Reinforcement-learning post-training: The models are post-trained on DRL with REINFORCE and GRPO-style group-relative advantage normalization using G = 8 samples per segment and temperature τ = 0.75.Generated transcriptions are sampled autoregressively, rewards are standardized within each segment’s group, and the model is updated with the REINFORCE objective.

4. EXPERIMENTAL RESULTS

The experiments generally pre-train on DSynth, fine-tune on DReal, post-train on DRL, and evaluate on DTest. Unless noted otherwise, results use the 1.3B-parameter model with instrument conditioning.

  • Experimental setup: Experiments generally follow a DSynth pre-training, DReal fine-tuning, DRL post-training, and DTest evaluation pipeline.Unless stated otherwise, results use the 1.3B-parameter model with instrument conditioning.

4.1 Main Results

MuScriptor improves progressively through synthetic pre-training, real-audio fine-tuning, and reinforcement learning, while inference conditioning further affects transcription quality. Synthetic pre-training is especially valuable with limited real audio, and classifier-free guidance improves most non-RL configurations.

  • Training stages: Each training stage improves over the previous one, with DReal fine-tuning essential and improving all metrics by roughly 20 percentage points.The synthetic-only model is already competitive with the state-of-the-art AMT baseline on frame F1, while fine-tuning on DReal improves all metrics substantially.
  • Inference guidance: 34.5 versus 26.1 onset F1: αCFG = 2 improves the DSynth-only model over αCFG = 1, whereas CFG adds no improvement for the RL model.The RL model can therefore avoid the second forward pass at inference time.
  • Synthetic pre-training: 33.4 versus 9.9 offset F1: synthetic pre-training strongly benefits models using only 1% of DReal compared with training from scratch.Figure 4 compares scratch initialization and DSynth pre-training across different amounts of real music audio.
  • Cross-domain evaluation: MuScriptor shows substantial cross-domain gains on datasets excluded from training, especially for frame F1 and multi F1.Because no benchmark covers the full range of genres and instrumentation, the models are evaluated primarily on DTest and additionally on several existing datasets.
  • Instrument conditioning: Supplying the instruments present in a track as inference-time conditioning improves quantitative scores.The model supports both conditioned and unconditioned inference through conditioning dropout during training, and the supplied instruments come from ground-truth annotations.

4.2 Ablation Study

The ablations show that larger MuScriptor models perform better, while a 60M-parameter model still achieves strong frame F1 performance. Mel spectrograms outperform alternative input representations, and retaining overlapping same-instrument notes substantially reduces evaluation scores.

  • Model scale: Larger transformer models achieve better results, but the smallest 60M-parameter model still reaches a frame F1 score of 65.The authors expect the smaller MuScriptor model to remain applicable in lower-resource settings.
  • Audio representation: Magnitude CQT inputs with 100 Hz frame rate and three bins per octave perform slightly worse than mel-scaled spectrograms.The CQT configuration contains 252 bins in total.
  • Audio representation: Codec inputs at 50 Hz and MERT embeddings at 75 Hz yield even worse results than mel-scaled spectrograms.The codec and music-embedding alternatives use Encodec and MERT, respectively.
  • Note filtering: 60.4 to 51.8 onset F1, 49.0 to 41.9 offset F1, and 48.2 to 42.0 multi F1 when overlapping same-instrument notes are retained.These results come from the 1.3B model trained on all datasets with αCFG = 1 and motivate a different tokenization scheme.

5. CONCLUSION

MuScriptor is an open-weight, general-purpose model for multi-instrument transcription, trained on synthetic and real music audio with reinforcement-learning post-training. It outperforms a state-of-the-art baseline, supports optional instrument conditioning, and is presented as the first effective open-weight model across various genres.

  • Contributions: MuScriptor is an open-weight model for general-purpose, multi-instrument music transcription.The training used synthetic and real music audio datasets with aligned note annotations.
  • Training: Reinforcement learning was used for post-training refinement after training on synthetic and real music audio datasets.
  • Performance and conditioning: MuScriptor outperforms a state-of-the-art baseline by a wide margin and supports optional instrument conditioning at inference time.
  • Novelty and future work: MuScriptor is described as the first effective open-weight music transcription model across various genres, with future work targeting broader instrument vocabulary, concurrent same-class notes, and longer segments.
Loading 2607.08168v2…