Source-linked AI summary
D-Composer: Language Modeling for Simultaneous Drum Transcription and Sound Event Decomposition
Patrick O'Reilly, Saumya Pailwan, Annie Chu, Jason Smith, Bryan Pardo
TL;DR
Task-specific drum models do not fully recover manipulable event arrangements, and leading AbS approaches have limits in timbral generalization, paired training data, and within-class variation. D-Composer predicts event-level symbolic and acoustic tokens from drum audio; on transcription and one-shot extraction it performs comparably to or better than baselines while generalizing to broader drum timbres.
Problem
Task-specific models recover aspects of drum structure but do not fully decompose recordings into manipulable event arrangements, while leading AbS approaches struggle with broad timbres, paired-data demands, and within-class variation.
Source: §1 · pp. 1–2Method
D-Composer uses autoregressive language modeling to represent drum events with interleaved onset, class, and isolated-audio tokens, trained using synthetic data.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 4Results
D-Composer performs comparably to or better than a state-of-the-art AbS system and task-specific baselines on transcription and one-shot extraction while generalizing to broader drum timbres.
Source: §1 · p. 2Takeaways & Limitations
D-Composer produces a sequence-based decomposition in which sound events can be arranged as a manipulable piano-roll-like representation.
Source: §3.2 · p. 4The approach's motivating gap remains that leading AbS systems struggle with broad drum timbres, rely on high-quality paired audio-transcription data, and do not model within-class timbral variation.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Drum beats are often conceptualized and produced as arrangements of repeated single-drum sound events or one-shots. Drum Analysis-by-Synthesis (AbS) models combine this structural assumption with reconstruction-based training to perform MIR tasks such as drum transcription, drum source separation, and one-shot extraction, offering a flexible alternative to more traditional task-specific supervision. However, leading AbS approaches struggle to generalize to a broad range of drum timbres (e.g., electronic drum kits), rely on high-quality paired audio-transcription training data, and do not model intra-class timbral variation within a single arrangement (e.g., multiple kick sounds). To overcome these limitations, we propose D-Composer, an autoregressive language modeling framework that decomposes drum recordings into labeled sound events and their onset times. In drum transcription and one-shot extraction tasks, D-Composer performs comparably to or better than a state-of-the-art AbS system and task-specific baselines while generalizing to a broader range of drum timbres. Our results demonstrate that when equipped with abundant synthetic data and compact acoustic token representations, language modeling provides a simple recipe for unifying transcription and sound event decomposition.
1. INTRODUCTION
Drum recordings mix discrete sound events into audio that is difficult to access, while existing task-specific models do not fully recover a manipulable event arrangement. D-Composer represents each event with its onset, class, and isolated audio, and performs comparably to or better than the cited baselines on transcription and one-shot extraction while generalizing to broader drum timbres.
1. INTRODUCTION
- Task-specific models recover aspects such as event times but cannot fully decompose recordings into manipulable arrangements of discrete sound events, limiting creative applications.The paper identifies transcription, source separation, and one-shot extraction as related MIR tasks.Source: §1 · p. 1
- Leading AbS approaches struggle with broad drum timbres, require high-quality paired audio-transcription data, and do not model within-class timbral variation.Multiple kick sounds are given as an example of within-class variation.Source: §1 · p. 2
- D-Composer maps each sound event to tokens for its onset, drum class, and isolated audio, producing a structured event arrangement.Onsets are quantized to 10ms intervals; class is represented as a MIDI note, and isolated audio as acoustic tokens.Source: §1 · p. 2
- D-Composer performs comparably to or better than a state-of-the-art AbS system and task-specific baselines on transcription and one-shot extraction, while generalizing to broader drum timbres.Experiments cover real and synthetic recordings.Source: §1 · p. 2
- The paper introduces an ultra-low-bitrate codec that compresses 44.1kHz stereo drum events at 0.1kbps, an order of magnitude smaller than leading high-fidelity codecs.Source: §1 · p. 2
2. RELATED WORK
Prior drum-analysis work spans task-specific learning and Analysis-by-Synthesis, while recent audio-language approaches represent events with timestamped descriptions. D-Composer instead uses symbolic-acoustic tokens to represent isolated drum events and their arrangement.
2. RELATED WORK
- Earlier drum transcription systems were often tailored to acoustic kits, while later work explored larger augmented or synthetic datasets and conditional generative modeling to improve generalization.Related work also applies generative modeling to synthetic data for source separation and one-shot extraction.Source: §2 · p. 2
- IDM unifies transcription, separation, and one-shot extraction, but predicts dense transcripts, one shot per class, and uses a narrow range of acoustic drum timbres; D-Composer uses event-level language modeling and diverse synthetic data to address these trade-offs.The D-Composer representation also permits within-class timbral variation.Source: §2 · p. 3
- Timestamped audio captioning predicts event onsets, classes, and descriptions, whereas D-Composer replaces descriptions with compact acoustic tokens that recover isolated events.Source: §2 · p. 3
- Cerberus predicts piano-roll-like transcripts and separates full instrument classes with masks, while D-Composer predicts isolated drum events through event-based language modeling.Drum-transcription decomposition also has precedent in Wu & Lerch’s NMF approach.Source: §2 · p. 3
- Recomposer performs transcript-guided resynthesis of general sound mixtures, while D-Composer decomposes drum beats into isolated events and a transcript that arranges them for synthesis by summation.Source: §2 · p. 3
3. METHOD
D-Composer combines a compact codec, an interleaved event-token representation, synthetic training examples, and an autoregressive language model. Together these components map drum recordings to event-level symbolic and acoustic outputs.
3. METHOD
- D-Composer uses a low-bitrate codec, interleaved transcript and acoustic tokens, synthetic training data, and an autoregressive language model to decompose drum recordings into sound events.Compact acoustic tokens help keep sequence lengths reasonable when a recording contains many events.Source: §3 · p. 3
3.1 Drum One-Shot Codec (DOC)
- DOC encodes three-second one-shots as 32-token sequences, reducing sequence length by an order of magnitude compared with high-fidelity codecs and enabling training on consumer GPUs.Its flow-matching decoder reconstructs input latents from noise, conditioned on summary latents and upsampler activations.Source: §3.1 · pp. 3–4
3.2 Interleaved Token Representation
- Each event is represented by class, onset, and 32 acoustic tokens, ordered by onset and then class for tied onsets; this sparse format supports variable event counts.Onsets use 10ms intervals and acoustic tokens come from a vocabulary of 625.Source: §3.2 · p. 4
3.3 Training Data
- Training mixtures sequence one-shots according to MIDI scores, sampling up to four one-shots per present class to introduce within-class timbral variation.Augmentations are applied to one-shots, MIDI, and the resulting mixture; targets are derived from transformed audio and MIDI.Source: §3.3 · p. 4
3.4 D-Composer Language Model
- A transformer encoder maps input audio latents to vectors, and a causal decoder predicts the event classes, onsets, and isolated sound events as a token sequence.The model is trained end-to-end with teacher-forced cross-entropy loss.Source: §3.4 · pp. 4–5
4. EXPERIMENTS
The experiments assess transcription, one-shot extraction, reconstruction, and codec compression using several baselines and datasets. They use task-specific metrics and evaluate on recordings spanning synthetic and real-world sources.
4. EXPERIMENTS
- The evaluation measures transcription accuracy, one-shot isolation quality, round-trip reconstruction, and codec reconstruction and compression.The comparison systems include IDM, task-specific ADTOF and DOSE, and codec baselines DAC, CoDiCodec, and MelodyFlow VAE.Source: §4 · p. 5, §4.1 · p. 5
4.1 Models
- DOC is trained for 80k steps and D-Composer for 100k steps, with total training times of 14 and 46 hours respectively on two RTX 4090 GPUs.At inference, DOC uses five flow steps, while D-Composer uses nucleus sampling with structural token constraints.Source: §4.1 · p. 5
4.2 Metrics
- Transcription uses per-class F1 with a 50ms onset tolerance; extraction and reconstruction use multi-scale spectral loss, which the authors note only coarsely assesses audio similarity.For D-Composer extraction evaluation, a random predicted one-shot per class is selected because it predicts one per event.Source: §4.2 · p. 5
4.3 Evaluation Data
- Evaluation covers 4,500 recordings across three tasks and four datasets, including acoustic and electronic timbres, real-world recordings, and varied audio qualities.Transcription uses E-GMD, MDB-Drums, and synthetic data; reconstruction additionally uses Freesound Loop Dataset recordings.Source: §4.3 · pp. 5–6
5. DISCUSSION
D-Composer accurately locates and isolates drum events, with results varying by task and training domain. Its decompositions support arrangement editing, while codec quality and broader training coverage remain future-work areas.
5. DISCUSSION
- D-Composer matches state-of-the-art AbS transcription on acoustic and partially acoustic data, and exceeds IDM and ADTOF on broader-timbre synthetic data; training domain strongly affects performance.The results do not establish D-Composer as a superior transcription engine; both baselines also perform strongly at smaller model scales.Source: §5 · p. 6
- D-Composer achieves lower average MSL than IDM in every one-shot class and than DOSE in all but KD.The results indicate that it can isolate representative one-shot timbres from mixtures.Source: §5 · p. 6
- D-Composer achieves lower average reconstruction MSL than IDM on both electronic FSL10K and majority-acoustic MDB recordings.The round-trip results indicate recovery of source-recording timing and timbre.Source: §5 · p. 6
- DOC produces one-shot token sequences over 10 times shorter than state-of-the-art acoustic codecs, at the expense of degraded audio quality.The authors identify higher-quality low-bitrate codec development as future work; current fidelity suffices to compete with baselines in downstream tasks.Source: §5 · p. 6
- The decompositions map to piano-roll-like representations for arrangement edits, such as removing hi-hats or adjusting speed, and onset-level timing corrections.The authors suggest these representations may support further musical interactions.Source: §5 · p. 6
- Future work includes musical-decomposition interfaces, better low-bitrate codecs, expanded MIDI drum classes and attributes, and integration of annotation-free real-world mixtures into training.The proposed training integration may use reinforcement learning.Source: §5 · p. 6
7. ETHICS STATEMENT
The authors describe limits tied to training-data coverage, one-shot asset licenses, and computational cost. They initially refrain from releasing model weights while clarifying licensing ambiguity and frame the models for research and education, not commercial use.
7. ETHICS STATEMENT
- Training corpora are dominated by Western popular music, so the models may perform poorly on timbres and rhythms from other musical traditions.The authors suggest broadening training data to address this limitation.Source: §7 · p. 7
- One-shot assets came from commercial libraries whose licenses prohibit redistribution in repackaged form; the authors initially withhold model weights while resolving whether trained models fall under that restriction.The authors state that the models are intended for educational and research purposes, not commercial use.Source: §7 · p. 7
- The authors estimate total experimental energy use at about 396 kWh, or 175% of the estimated mean training cost for ISMIR papers from 2017–2023.They describe the estimate as approximate and omit inference because it used a single GPU for a small fraction of training time.Source: §7 · p. 7