The NES Music Database: A multi-instrumental dataset with expressive performance attributes

• Most music-generation datasets model compositional semantics but omit expressive performance details, producing rigid-sounding demonstrations.Source: §1 · p. 1
• NES music provides a cohesive multi-instrumental source because the NES APU supports four simultaneous monophonic voices and preserves exact performance characteristics.Source: §1 · p. 1
• NES-MDB formalizes separated composition and expressive-performance mapping as complementary tasks, with baselines and reproducible evaluation.Source: §1 · p. 1
• The accompanying library converts between NES machine code and machine-learning representations, enabling realistic playback of generated music.Source: §1 · p. 1

2.1 Blended composition

• Blended scores represent whether any voice plays each note, simplifying modeling but losing voicing and requiring reassignment for multi-instrumental rendering.Source: §2.1 · p. 2

2.2 Separated composition

• Separated scores encode a monophonic sequence for each voice and explicitly model temporal and contrapuntal dependencies between voices.Source: §2.2 · p. 2
• NES hardware’s strict voice limit makes separated-score modeling appropriate for NES-MDB, unlike datasets with uninhibited polyphony.Source: §2.2 · p. 2

2.3 Expressive performance

• Expressive performance maps a composition to dynamics and timbre, modeling P(e | c) after composition to produce an expressive score.Source: §2.3 · p. 2

2.4 Task summary

• The dataset defines blended composition, separated composition, and expressive performance as three tasks, with the latter two combinable for novel NES music generation.Source: §2.4 · p. 3

3. DATASET DESCRIPTION

• The dataset excludes the NES sampler because its melodic or percussive use is compositionally ambiguous, retaining four defined instrument voices.Source: §3 · p. 3
• Music is extracted by logging APU-register writes from emulated games, deduplicated, and split into train, validation, and test sets without composer overlap.Source: §3 · p. 3

3.1 Extracting expressive scores

• Expressive scores are emulated at 44.1 kHz, downsampled to 24 Hz, and stripped of dynamics, timbre, and voicing to derive separated and blended scores.Source: §3.1 · p. 3
• The expressive representation constrains melodic pitches, provides 16 noise values, and uses instrument-specific velocity controls, including binary velocity for triangle.Source: §3.1 · p. 3
• Pulse duty cycles and noise modes encode timbre, yielding state spaces of 4621, 89, and 481 for pulse, triangle, and noise generators.Source: §3.1 · p. 3

• The evaluation reports negative log-likelihood and accuracy on unseen data for separated composition and expressive performance.Source: §4 · p. 3
• The paper presents these results as statistical baselines, cautioning that they do not necessarily measure a model’s ability to generate compelling musical examples.Source: §4 · p. 3

4.1 Separated composition experiments

• 83% global accuracy for the bigram baseline falls to 0% at POIs, showing why evaluation emphasizes timesteps where voices change.The unigram baseline similarly drops from 37% globally to 4% at POIs.Source: §4.1 · p. 4, §5 · p. 4
• Artificially emphasizing POIs during training produces unrealistically sporadic music, so the authors recommend unbiased emphasis for NES-MDB.Source: §4.1 · p. 4

4.2 Expressive performance experiments

• The expressive-performance task predicts velocity and timbre from separated scores, with autoregressive models additionally conditioning on prior expressive outputs.LSTM Note+Auto processes the score and autoregressive expressive characteristics before projecting to six categorical outputs; MultiReg baselines provide simpler comparisons.Source: §4.2 · pp. 4–5

4.1 Separated composition experiments

• DeepBach achieves the best separated-composition statistical performance at POIs, but its future context makes sampling more expensive than alternatives.LSTM Quartet permits efficient ancestral sampling, whereas DeepBach uses bidirectional processing to access future notes; LSTM Soloists outperform RNN Soloists.Source: §4.1 · p. 4, Table 3 · p. 5

4.2 Expressive performance experiments

• LSTM Note+Auto substantially improves global expressive-performance performance over LSTM Note, while their POI performance is comparable.The reported pattern suggests score information helps identify when to change, whereas prior velocity and timbre help select the next value.Source: §4.2 · p. 5
• MultiReg Note outperforms MultiReg Note+Auto at POIs because the autoregressive version overfits more quickly despite receiving more information.Source: §4.2 · p. 5

4.3 Blended composition experiments

• Blended-score modeling is included as a standard benchmark, although the representation discards voicing and is not immediately useful for generating NES music.NES-MDB uses 88 possible notes with at most three simultaneous voices in this format.Source: §4.3 · pp. 5–6
• Models assign higher likelihood to NES-MDB than four comparison datasets, and its larger size delays overfitting in sequential models.The authors note that their implementations may differ slightly from prior implementations while achieving comparable results on the original datasets.Source: Table 5 · p. 5, §4.3 · p. 6

5. RELATED WORK

• Bach chorales provide separated four-voice compositions but only 389 chorales and no expressive information, while Piano-midi.de provides expressive timing and dynamics only for solo piano.Nottingham and MuseData are standardized blended-score collections covering folk, orchestral, and piano music.Source: §5 · p. 6
• Lakh MIDI is much larger than NES-MDB but has unconstrained polyphony, inconsistent expressive characteristics, and broad genre, instrument, and time-period variation.Other expressive symbolic corpora include the Magaloff Corpus and a small video-game-music transcription dataset.Source: §5 · p. 6

5.1 Statistical composition

• Statistical composition research progressed from expert systems to RNNs and other models for monophonic melodies, blended polyphony, symbolic priors, and harmonization.Prior work includes HMMs, bidirectional RNNs, Gibbs-sampling approaches, restricted Boltzmann machines, and NADEs [1, 2, 5, 13, 16, 17, 21, 25, 26, 30, 31, 34, 35].Source: §5.1 · p. 6

5.2 Statistical performance

• Expressive-performance research models dynamics, timing, articulation, accompaniment, and velocity using rule-based, graphical, spectral, and recurrent approaches.Most cited statistical systems focus on piano performances rather than multi-instrumental music.Source: §5.2 · p. 6

• NES-MDB provides a large corpus for studying multi-instrumental polyphonic composition and expressive performance generation.It enables examination of the full pipeline from composition through performance.Source: §6 · p. 6
• The dataset parses NES machine code into familiar formats such as MIDI, removing the need to work directly with low-level game-system details.Source: §6 · p. 6
• An open-source converter lets researchers audition generated compositions as waveforms rendered by the NES.Source: §6 · p. 6
• Reproducible baselines and evaluation methodology are provided to encourage further research emphasizing expressive performance in music generation.Source: §6 · p. 6