Source-linked AI summary
DeepDrummer : Generating Drum Loops using Deep Learning and a Human in the Loop
Guillaume Alain, Maxime Chevalier-Boisvert, Frederic Osterrath, Remi Piche-Taillefer
TL;DR
DeepDrummer addresses interactive drum-loop generation from limited feedback about a user's musical preferences. It trains a per-user audio critic to guide loop generation and, in a 25-participant study, reports improved ratings after 80 interactions. The study was intentionally short, and the authors identify larger-scale data collection as future work.
Problem
The paper addresses how to train a deep-learning model to approximate a user's musical taste from limited feedback for an interactive drum-loop generation tool.
Source: §1 · p. 1Method
DeepDrummer combines a per-user neural critic trained on like/dislike feedback with a drum-loop generator to propose loops based on limited interactions.
Source: §1 · p. 1Results
Takeaways & Limitations
The authors report that DeepDrummer quickly achieves meaningful improvements by proposing loops and receiving like/dislike feedback.
Source: §7 · p. 7The experiment was intentionally limited to 80 ratings and 10 minutes because the authors sought to avoid fatigue and declining rating quality; they identify collecting more data as future work.
Source: §6.2 · p. 6
Abstract
from arXiv · showhide
DeepDrummer is a drum loop generation tool that uses active learning to learn the preferences (or current artistic intentions) of a human user from a small number of interactions. The principal goal of this tool is to enable an efficient exploration of new musical ideas. We train a deep neural network classifier on audio data and show how it can be used as the core component of a system that generates drum loops based on few prior beliefs as to how these loops should be structured. We aim to build a system that can converge to meaningful results even with a limited number of interactions with the user. This property enables our method to be used from a cold start situation (no pre-existing dataset), or starting from a collection of audio samples provided by the user. In a proof of concept study with 25 participants, we empirically demonstrate that DeepDrummer is able to converge towards the preference of our subjects after a small number of interactions.
1. INTRODUCTION
DeepDrummer uses limited user feedback to train a per-user critic and guide drum-loop generation toward loops the user is more likely to like. The work evaluates the approach with 25 participants and treats useful musical exploration—not a perfect model of changing preferences—as its goal.
1. INTRODUCTION
- Users rate loops as like or dislike, a per-user neural-network critic predicts ratings from audio, and a generator proposes 16-step patterns with four randomly selected drum sounds.The generator has no trainable parameters and few prior assumptions about musical structure; critic feedback filters its output for user rating.Source: §1 · p. 1
- A perfect model of musical preference is a non-goal because preferences are subjective and may change during measurement.The intended aim is a useful filter for quickly exploring musical ideas in an artistically useful way.Source: §1 · p. 1
- DeepDrummer combines deep learning and active learning to generate drum loops from limited human interactions.The stated application includes helping creators find inspiration.Source: §1 · p. 1
- 25 participants rated 80 drum loops each, and the experiment measured rating improvement after those interactions.The introduction reports meaningful gains within 80 interactions.Source: §1 · p. 1
2. RELATED WORK
DeepDrummer is positioned among music-generation approaches while differing from sequence modeling and adversarial training. Its user-feedback loop is related to interactive preference-learning systems, but it seeks families of appealing loops rather than one global optimum.
2. RELATED WORK
- Many music-generation methods frame generation as sequence modeling, where a probabilistic model is assumed to capture a manifold of desirable music [12] [18] [21] [22].Source: §2 · pp. 1–2
- DeepDrummer is not sequence modeling; its generator-critic framework places the computational heavy lifting in the critic and shares a music-generation connection with.It draws inspiration from GAN generator-discriminator concepts, but has no adversarial training or reference dataset to reproduce.Source: §2 · p. 2
- Like interactive preference-learning work and its musical application, DeepDrummer uses minimal user feedback, but aims to suggest many good loops in a family rather than optimize for one global maximum.Source: §2 · p. 2
- DeepDrummer's active-learning opportunities include confidently predicted-liked loops that prove erroneous, which can correct critic misconceptions with informative ratings.Source: §2 · p. 2
- The critic's audio-based like prediction resembles audio-similarity music recommendation systems and allows generalization across sounds in drum loops.Source: §2 · p. 2
3. MODEL AND FRAMEWORK
DeepDrummer generates symbolic drum patterns, renders them as audio, and uses a critic trained on user ratings to score them. It samples patterns according to the critic's predicted like probability, using Metropolis-Hastings to explore multiple modes rather than relying on greedy hill climbing.
3.1 Overview
- DeepDrummer generates random drum patterns, assigns one-shot samples to tracks, renders them as audio, and feeds loop audio as MFCC features to a neural critic predicting whether the current user will like or dislike it.Source: Figure 1 · p. 3
3.2 Network Architecture of the Critic
- The 3.2 Network Architecture of the Critic uses MFCC input because limited training data makes learning features from raw audio with 1D convolutions difficult.The selected network has four convolution layers, a 128-unit dense layer, a two-unit softmax output, and 291K learnable parameters; dropout and weight decay regularize it.Source: §3.2 · p. 3
3.3 Training the Critic
- The 3.3 Training the Critic initializes a user-specific critic and repeatedly trains it on that user's accumulated loop ratings to predict the probability of a like.The generator has no trainable parameters and does not use gradients from the critic.Source: §3.3 · p. 3
3.4 Sampling Drum Loops
- The 3.4 Sampling Drum Loops searches for loops with high critic scores, using Metropolis-Hastings because greedy hill climbing can reach low-quality or insufficiently varied local maxima.The sampling distribution is proportional to the critic score; Phase I samples from the predicted probability distribution, while Phase II continues iterations until a sample scores at least 0.95.Source: §3.4 · pp. 3–4
4. EXPERIMENTAL PROTOCOL
The protocol tests whether a critic can improve its match to user preferences over 80 ratings, while separating training from evaluation to account for possible shifts in user ratings. It uses a web interface after COVID-19 restrictions made a controlled in-person setup impossible.
4.1 Hypothesis and Protocol
- The study tests whether DeepDrummer can increase the proportion of liked loops over 80 binary ratings, with a deliberately short, minimal proof-of-concept experiment.The authors note that limited human attention motivated the 10-minute design; a production-ready tool would involve more components.Source: §4.1 · p. 4
4.2 Anticipating Preference Shifts
- Because ratings may shift as users spend time listening, the protocol separates model training from evaluation and randomizes presentation of earlier- and later-model loops.The authors distinguish this concern from experimental randomness and cite potential preference shifts.Source: §4.2 · pp. 4–5
4.3 Phases I & II of the Experiment
- Phase I updates the critic after each of 80 ratings; Phase II evaluates fresh loops from the initial and final critics in shuffled order, with no further training.Phase II ratings are for analysis only, and users are blind to which model generated each loop.Source: Figure 2 · p. 4, §4.3 · p. 5
4.4 Web-Based Experiment
- COVID-19 restrictions led the researchers to run the experiment through a web interface, with all 25 volunteers receiving the same instructions.The original plan was an in-person setup with a desktop and provided headphones.Source: §4.4 · p. 5
5. RESULTS
Across 25 participants, users liked a larger share of loops from the final critic than from the untrained critic. The per-user results also show that improvement was not uniform.
5. RESULTS
- Each user supplied 30 binary ratings for each critic, and θ represents the estimated probability that the user likes a loop.The paper interprets θ as the maximum-likelihood estimate of a Bernoulli-distribution parameter.Source: §5 · p. 5
- 37% of loops were liked without training, compared with approximately 54% after minimal training across 25 participants.The paper describes this as significant improvement after 10 minutes of like/dislike ratings.Source: §5 · p. 5
- 72% of users had ∆θ(i) > 0, and 36% had ∆θ(i) ≥0.2.Some users clustered around 0.0, indicating no improvement for those participants.Source: §5 · p. 5
6. DISCUSSION AND FUTURE WORK
The discussion reports preference gains for many users and examines whether the critic captures rhythmic structure. It outlines limits of the short study, possible melody extensions, and directions toward a production tool.
6.1 Frequencies and Rhythm
- The initial critic already generated loops users liked about 40% of the time, and the before/after distributions were bimodal.The two modes suggest participants were roughly split between those perceiving clear improvement and those perceiving little change without degradation; these measurements came from Phase II, designed to address shifting preferences.Source: Figure 4 · p. 6
- Across a limited number of interactions, a majority of users showed clear improvements; 36% of users had ∆θ(i) ≥0.20.Source: §6.1 · p. 6
- Best-rated loops converged toward similar rhythms, while the worst were very busy, supporting the view that the critic learns more than filtering unpleasant frequencies.The authors describe this as supporting evidence from an experiment producing the best and worst loops using critics trained across users.Source: §6.1 · p. 6
6.2 Larger Scale Experiment
- The experiment was intentionally limited to 80 ratings and about 10 minutes to avoid user fatigue and declining rating quality.The authors' informal experiments with hundreds of their own ratings suggested higher-quality loops and fewer false positives with more training, but they propose a larger Phase II study as future work.Source: §6.2 · p. 6
6.3 Melodies
- The approach may extend to melodies, but its broader applicability depends on generators that support mutation and hill climbing.The authors suggest melody generation may require autoencoders so mutations can occur in hidden representation space.Source: §6.3 · p. 6
6.4 Path Towards a Production Tool
- Proposed production directions include adding style and sample priors, seeding training data, clustering preferences, and removing unpleasant samples.Source: §6.4 · pp. 6–7
- Other proposed directions are letting users seed the system with their own music and pretraining an adaptable critic through meta-learning.Source: §6.4 · p. 7
7. CONCLUSION
The conclusion presents DeepDrummer as a human-in-the-loop music-generation system that uses a neural critic to explore candidate loops. An experiment with 25 participants measured rating improvement after 80 loops each; supplemental materials include a demo video and related resources.
7. CONCLUSION
- DeepDrummer uses a neural-network critic as a proxy for the user to explore a large music space, forwarding only the most relevant candidates.Source: §7 · p. 7
- In an experiment, 25 participants each rated 80 drum loops, and the authors report measured improvement in ratings after those interactions.Source: §7 · p. 7
A. SUPPLEMENTAL MATERIALS
- Supplemental materials include a 2.5 GB download and a short video demonstrating the experimental web interface and a desktop application trained with more samples.Source: A. SUPPLEMENTAL MATERIALS · p. 9, §A.1 · p. 9
A.3 Drum Samples
The drum-sample collection contains 340 audio samples, released under CC0.
A.3 Drum Samples
- The 340 samples are 44.1 kHz mono 16-bit PCM files, released under CC0; they include hits from named drum machines, synthesized hits, and intentionally unpleasant sounds.Source: §A.3 · p. 9
A.4 Collected Data
The collected dataset covers all 25 experiment users and includes generated loops, ratings, and critic models.
A.4 Collected Data
- The release includes 3,500 drum loops from both phases, users' ratings, and untrained and trained critic models; ratings are anonymized and stored as parseable JSON.Source: §A.4 · p. 9
A.5 Best and Worst Loops
The appendix reports what the models agree on in their highest- and lowest-ranked loops, then discusses design alternatives, generation, and hyperparameter tuning. The alternatives include trade-offs in eliciting feedback, updating the critic, and choosing its audio representation.
A.5 Best and Worst Loops
- The ensemble’s worst-ranked loops are busy and contain high-frequency noise, while its best-ranked loops have clear rhythms and more conventional drum sounds.The ranking averages scores from all trained models; the five best and worst clips are included in the supplemental material.Source: §A.5 · p. 9
B.1 A vs B Comparison
- In pairwise listening, users may forget clip A while hearing B, making the comparison format challenging for audio.The authors also report that binary like/dislike ratings can be confusing when users judge a clip relative to earlier outputs.Source: §B.1 · p. 9
B.2 Integer and Multidimensional Ratings
- Integer ratings could record indifference, and multidimensional ratings could capture reasons for dislike, but neither extension was tested.The authors considered binary classification sufficient for this proof of concept.Source: §B.2 · p. 9
B.3 Four-on-the-floor Constraint
- An optional four-on-the-floor constraint accelerated convergence to recognizable beats, but was omitted to test whether the model learned rhythmic elements without prior assumptions.Source: §B.3 · p. 9
B.4 Retraining Models
- Incremental interactive learning was more sample-efficient in practice than retraining from scratch after every rating.The authors suggest new feedback corrects the current model’s errors, whereas a newly trained model may not carry forward prior errors.Source: §B.4 · p. 10
B.5 Critic Model Design
- The MFCC-based critic produced better results more quickly in qualitative evaluations than raw-waveform or FFT-based alternatives.The authors attribute this choice to the small dataset, which makes learning useful features from raw audio difficult.Source: §B.5 · p. 10
C. GENERATION BY METROPOLIS-HASTINGS SAMPLING
- Metropolis–Hastings repeatedly scores perturbed loops and can accept worse candidates temporarily to escape local maxima.The chain targets liked loops under the generator’s uniform prior; temperature adjusts acceptance from optimization toward equal desirability.Source: C. GENERATION BY METROPOLIS-HASTINGS… · p. 10
D. HYPERPARAMETER OPTIMIZATION
- Human-in-the-loop tuning makes large hyperparameter sweeps impractical, so the authors used a neural-network proxy and later validated its choices themselves.The interactive model’s training data depends on the current model; proxy tastes are less rich than human tastes, and static-data tests do not guarantee performance in another setting.Source: D. HYPERPARAMETER OPTIMIZATION · pp. 10–11
E. WEB SPLASH PAGE
Figure 6 shows the landing page for the online experiments. The passage identifies it as DeepDrummer’s splash page.
E. WEB SPLASH PAGE
- Figure 6 shows DeepDrummer’s splash page for the online experiments.Source: Figure 6 · p. 11, E. WEB SPLASH PAGE · p. 11