Source-linked AI summary
LakhNES: Improving multi-instrumental music generation with cross-domain pre-training
Chris Donahue, Huanru Henry Mao, Yiting Ethan Li, Garrison W. Cottrell, Julian McAuley
TL;DR
Multi-instrumental music generation must model dependencies among instruments in a setting with less training data than piano generation. LakhNES pre-trains on mapped Lakh MIDI music and fine-tunes on NES-MDB; the paper reports improved quantitative performance and strong qualitative results.
Problem
Multi-instrumental generation adds intricate inter-instrument dependencies to polyphonic music, while less data is available than for piano generation.
Source: §1 · p. 1Method
LakhNES heuristically maps heterogeneous Lakh MIDI ensembles into the NES four-voice ensemble, pre-trains on that dataset, and fine-tunes on NES-MDB.
Source: §1 · p. 1Results
Lakh MIDI transfer improves quantitative performance by 10%, and user studies indicate strong qualitative results for LakhNES.
Source: §1 · p. 1Takeaways & Limitations
LakhNES can generate chiptunes from scratch, continue human-composed material, and produce melodies corresponding to human-specified rhythms.
Source: §1 · p. 1LakhNES has difficulty maintaining rhythmic consistency, which can reveal computer-generated chiptunes in a Turing test.
Source: §7.1 · p. 6
Abstract
from arXiv · showhide
We are interested in the task of generating multi-instrumental music scores. The Transformer architecture has recently shown great promise for the task of piano score generation; here we adapt it to the multi-instrumental setting. Transformers are complex, high-dimensional language models which are capable of capturing long-term structure in sequence data, but require large amounts of data to fit. Their success on piano score generation is partially explained by the large volumes of symbolic data readily available for that domain. We leverage the recently-introduced NES-MDB dataset of four-instrument scores from an early video game sound synthesis chip (the NES), which we find to be well-suited to training with the Transformer architecture. To further improve the performance of our model, we propose a pre-training technique to leverage the information in a large collection of heterogeneous music, namely the Lakh MIDI dataset. Despite differences between the two corpora, we find that this transfer learning procedure improves both quantitative and qualitative performance for our primary task.
1. INTRODUCTION
The paper adapts symbolic music language models to multi-instrumental generation, where inter-instrument dependencies and less available training data pose challenges. It uses NES-MDB and transfers knowledge from Lakh MIDI to improve generation performance.
1. INTRODUCTION
- Multi-instrumental generation adds intricate inter-instrument dependencies to the polyphony of piano music, while having less training data available.The work extends symbolic piano-generation results to this setting.Source: §1 · p. 1
- NES-MDB offers 46 hours of chiptunes for a fixed four-instrument ensemble, though it is smaller than the 172-hour MAESTRO piano dataset used for Music Transformer [3].Its structural homogeneity makes it appealing for multi-instrumental generation research.Source: §1 · p. 1
- Lakh MIDI pre-training followed by NES-MDB fine-tuning improves quantitative performance by 10%.The procedure heuristically maps Lakh MIDI's heterogeneous ensembles into the NES four-voice ensemble.Source: §1 · p. 1
- LakhNES produces strong qualitative results in user studies and can generate chiptunes from scratch, continue human-composed material, or create melodies matching human-specified rhythms.LakhNES denotes the model pre-trained on Lakh MIDI and fine-tuned on NES-MDB.Source: §1 · p. 1
2. RELATED WORK
Earlier machine-learning music generation often focused on monophonic melodies or polyphonic piano scores, while other multi-instrumental approaches impose constraints on generation. This work sits among systems for multi-instrumental generation from scratch and contrasts with approaches requiring complex inference or producing only fixed-length outputs.
2. RELATED WORK
- Earlier music-generation approaches manually encoded rules or rearranged human-composed fragments, while more recent work favors machine learning that extracts patterns from music corpora.See for an extensive overview.Source: §2 · p. 2
- Earlier polyphonic methods use piano-roll or event-based representations, but none of the described methods generate multi-instrumental music.Prior work also modeled simple monophonic melodies.Source: §2 · p. 2
- Multi-instrumental systems for harmonizing with human-composed material require complex inference to generate without human input.Recent from-scratch methods [21] [22] [23] are limited to fixed lengths, unlike this paper's arbitrarily long sequences.Source: §2 · p. 2
3. DATASETS AND TASK
The task uses NES-MDB, a symbolic corpus of music for a fixed four-voice NES ensemble. The paper represents scores as event sequences in which events mark musical changes, rather than as a time-dense piano roll.
3. DATASETS AND TASK
- NES-MDB contains approximately 46 hours of symbolic music composed for the NES sound chip's fixed ensemble.The fixed-ensemble corpus is presented as an unusually large resource for multi-instrumental generation.Source: §3 · p. 2
3.1 NES ensemble preliminaries
- NES-MDB's ensemble has four monophonic voices: two pulse generators, a triangle voice, and a noise voice; the authors exclude a fifth waveform-playback voice and ignore expressive attributes to focus on composition.Typically, the triangle voice plays bass, the pulse voices alternate melody and harmony, and noise provides percussion.Source: §3.1 · p. 2
3.2 Event-based task
- The event-based representation replaces redundant piano-roll timesteps with a time-ordered sequence of musically salient events, including time shifts and instrument-specific note on/off events.Its 631-event inventory preserves the score apart from minor timing quantization; contemporaneous events are ordered P1, P2, TR, NO.Source: §3.2 · pp. 2–3, Figure 1 · p. 2, Table 1 · p. 3
4. METHODOLOGY
The paper models music as a sequence of events and uses Transformer-XL, then transfers knowledge from mapped Lakh MIDI scores to NES-MDB. Its method addresses the mismatch between heterogeneous source ensembles and the NES’s fixed ensemble.
4. METHODOLOGY
- The model factorizes an N-event sequence into left-to-right conditional probabilities, enabling music generation by sampling each event given previous outputs.The optimization objective is to maximize the likelihood of real event sequences, using a Transformer architecture.Source: §4 · p. 3
4.1 Transformer architecture
- Transformer-XL extends the decoder Transformer with recurrence, allowing it to use state from previous segments and learn dependencies beyond the current training segment.Its position-aware mechanism supports generalization to different amounts of recurrent memory; the authors focus on this architecture rather than Music Transformer for simplicity and because its recurrence is sufficient for long-term dependencies.Source: §4.1 · p. 3
4.2 Pre-training
- The transfer-learning protocol maps heterogeneous Lakh MIDI into NES-compatible scores, then pre-trains on that data and fine-tunes on NES-MDB.The authors identify careful consideration of musical invariants during cross-domain mapping as a primary methodological contribution.Source: §4.2 · pp. 3–4
4.2.1 Mapping Lakh MIDI to the NES ensemble
- Melodic instruments are range-filtered and randomly assigned to NES melodic voices; percussion voices are randomly assigned to one of 16 noise types.Multiple assignments are output when an example has a variable number of melodic instruments; mapping 175k MIDI files produces 775k NES-compatible examples.Source: §4.2.1 · p. 4
5. EXPERIMENTS
The experiments train Transformer-XL on NES-MDB event sequences, apply standard and multi-instrumental data augmentation, and test Lakh MIDI pre-training against n-gram and LSTM baselines.
5. EXPERIMENTS
- Transformer-XL is trained on 512-event excerpts averaging about 9 seconds, with recurrent attention providing access to twice that history.The 12-layer, 8-head model uses a 2e−5 learning rate and stops when validation performance ceases to improve; training reaches early stopping in less than a day on four Titan X GPUs.Source: §5 · p. 4
5.1 Data augmentation and pre-training
- Augmentation transposes melodic voices, adjusts speed, randomly removes instruments, and sometimes shuffles melodic score-to-instrument assignments; Lakh pre-training runs four epochs before NES-MDB fine-tuning.Lakh pre-training lasted a week and remained underfit, with validation performance still improving; the same hyperparameters were used for pre-training and fine-tuning.Source: §5.1 · p. 4
5.2 Baselines
- Baselines include unigram and 5-gram models, with backoff for unseen 5-grams, and an LSTM configured to have approximately as many parameters as Transformer-XL.The models are evaluated on the NES-MDB event-based representation; Table 2 reports parameter counts, epochs before early stopping, and test perplexity.Source: §5.2 · pp. 4–5, Table 2 · p. 5
6. QUANTITATIVE ANALYSIS
Transformer-XL strongly outperforms the n-gram and LSTM baselines on NES-MDB, and LakhNES improves further over augmentation-only training. More Lakh pre-training improves fine-tuned performance, with diminishing returns.
6. QUANTITATIVE ANALYSIS
- Test PPL was 3.5 for Transformer-XL versus 37.2 for n-gram and 14.1 for LSTM; LakhNES performed 10% better than augmentation-only training.Data augmentation improved LSTM performance by 10% and Transformer-XL performance by 22%.Source: §6 · p. 5
- Increasing Lakh MIDI pre-training from 1 to 2 to 4 epochs improved fine-tuned test PPL, but with diminishing returns.Lower test perplexity indicates a better fit to unseen data.Source: §6 · p. 5
7. USER STUDY
The authors use two Mechanical Turk studies to test whether generated chiptunes sound human-like and which methods listeners prefer. LakhNES leads the generative methods in preference, while its advantage in the Turing test over an un-pretrained Transformer-XL is not statistically significant.
7. USER STUDY
- Because perplexity may not track human judgments, two Mechanical Turk studies compared four models, including LakhNES, with human-composed music.The studies examined both human-likeness and listener preference.Source: §7 · p. 5
7.1 Turing test
- Judges identified the human clip in five-second human/generated pairs; workers needed perfect control accuracy, and each method received about 180 evaluations.Control pairs used randomly generated clips.Source: §7.1 · p. 5
- LakhNES was mistaken for human more often than the 5-gram model (p < .0001) and LSTM (p = .07), but its advantage over un-pretrained Transformer-XL was not significant (p = .32).Lower accuracy indicates that judges more often confused generated clips with human-composed clips.Source: Figure 4 · p. 6, §7.1 · p. 6
7.2 Preference test
- The preference test used ten-second clips so longer-term structure could influence choices, and excluded workers who ever preferred random music to real data.Pairs compared clips from different methods.Source: §7.2 · p. 6
- 26% of comparisons preferred LakhNES over real data, versus 10% for the LSTM; overall, LakhNES beat other generative methods but trailed real data.Figure 5 reports the proportion of pairwise comparisons in which each method was preferred.Source: Figure 5 · p. 6, §7.2 · p. 6
8. PAIRING LAKHNES WITH HUMANS
LakhNES can support human composers by continuing human-composed material or generating scores around fixed rhythms. For sound examples, the authors report better qualitative results with sampling temperature 0.95 and top-k sampling with k = 32.
8. PAIRING LAKHNES WITH HUMANS
- LakhNES can continue human-composed material or generate the rest of a score from fixed rhythmic material, supporting composers’ work beyond generating chiptunes from scratch.For sound examples outside the user study, the authors found that temperature 0.95 and top-k sampling with k = 32 improved results qualitatively.Source: §8 · p. 6
9. CONCLUSION
The paper presents LakhNES as a method for multi-instrumental music generation, using an event-based representation and language models. It reports that cross-domain pre-training improves quantitative and qualitative results, with uses in both chiptune generation and collaboration with human composers.
9. CONCLUSION
- LakhNES combines an event-based representation and powerful language models for multi-instrumental generation, with cross-domain pre-training improving results quantitatively and qualitatively.The system generates chiptunes from scratch and can collaborate with human composers.Source: §9 · p. 6