PaperlayerView exact revision on arXiv

Source-linked AI summary

Modelling Symbolic Music: Beyond the Piano Roll

Christian Walder

arXiv:1606.01368v1cs.SD

TL;DR

Polyphonic music presents a multivariate modelling problem because notes can sound together and valid combinations are highly varied. The paper reduces note prediction to a univariate sequence for an LSTM, with rhythmic structure given, and reports improved results on four benchmark datasets after transposition augmentation.

  • Problem

    Polyphonic music is multivariate because notes can occur together, and plausible chords can contain note combinations that are implausible.

    Source: §1 · pp. 1–2

  • Method

    The paper represents music so an LSTM predicts notes as a univariate sequence, with onset and offset times fixed and training examples transposed across keys.

    Source: §3.2 · p. 3, §3.3 · p. 4, §3.4 · p. 5

  • Results

    The basic LSTM outperformed previous methods on all four datasets, and the augmented model outperformed the basic model and prior approaches on all four.

    Source: §4.2 · p. 7

  • Takeaways & Limitations

    The approach handles arbitrary fixed rhythmic structures rather than requiring uniform time discretization, and motivates a new benchmark dataset.

    Source: §5 · p. 9

    The model assumes note onset and offset times are fixed; modelling rhythmic structure is beyond this work’s scope.

    Source: §3.3 · p. 4

Abstract

from arXiv · show

In this paper, we consider the problem of probabilistically modelling symbolic music data. We introduce a representation which reduces polyphonic music to a univariate categorical sequence. In this way, we are able to apply state of the art natural language processing techniques, namely the long short-term memory sequence model. The representation we employ permits arbitrary rhythmic structure, which we assume to be given. We show that our model is effective on four out of four piano roll based benchmark datasets. We further improve our model by augmenting our training data set with transpositions of the original pieces through all musical keys, thereby convincingly advancing the state of the art on these benchmark problems. We also fit models to music which is unconstrained in its rhythmic structure, discuss the properties of this model, and provide musical samples which are more sophisticated than previously possible with this class of recurrent neural network sequence models. We also provide our newly preprocessed data set of non piano-roll music data.

1 Introduction

The paper frames algorithmic composition as a challenging sequential-modelling problem, drawing motivation from links between music and language perception and recent NLP advances. Polyphony makes the task multivariate, with many possible but not equally plausible note combinations.

1 Introduction

  • A growing body of evidence links music and natural-language perception, and people are sensitive to subtleties and irregularities in both.The paper uses this connection to motivate investigating algorithmic composition.Source: §1 · p. 1
  • Deep learning has become dominant in several NLP subdomains, while data-driven natural-language generation is an active research topic.The paper points to these developments as potential analogues for music composition.Source: §1 · p. 1
  • Musical notes can occur simultaneously, making music multivariate, while valid note combinations are highly multimodal.Plausible chords can contain note combinations that are themselves implausible.Source: §1 · pp. 1–2

2 Related Work

Prior work connects NLP sequence models to music modelling, where recurrent neural networks have been applied to composition. Existing approaches include note-and-rhythm models, harmonic-state models, and uniform-time representations, each with different scope and constraints.

2.1 Natural Language Processing

  • LSTMs model sequences using recurrent outputs and contextual state, and are designed to capture long-range dependencies.The passage describes language-model outputs predicting the next word and word vectors transformed through an embedding matrix.Source: §2.1 · p. 2

2.2 Algorithmic Composition

  • Polyphonic music has been modelled as a sequence of sounding-note sets, with recent RNNs reported to model musical sequences more effectively in practice.Classic sequence-modelling work includes hidden Markov models.Source: §2.2 · p. 2
  • CONCERT models a melody with its rhythm and an accompanying chord sequence, using a five-dimensional pitch representation and explicitly modelled note durations.The pitch representation is intended to capture perceptual distance.Source: §2.2 · p. 2
  • Many music models use uniform time discretization, which limits realistic applicability; other NLP ideas used for music include hierarchical grammars.Source: §2.2 · p. 3
  • Allan and Williams used an HMM with a fixed harmonic sequence to learn chord transitions and note-level emissions.Source: §2.2 · p. 3
  • Earlier LSTM composition work investigated blues, but used a small dataset and model parameter space by current GPU-hardware standards.LSTM suitability for composition was noted in the original work and investigated by Eck and Schmidhuber.Source: §2.2 · p. 3

3 Our Approach

The approach applies an LSTM language model to music by turning note prediction into a univariate sequence task. Its representation orders notes over time while taking rhythmic structure as given, and training data is augmented with transpositions across keys.

3.1 LSTM Language Model Architecture

  • The approach starts from a multi-layer LSTM language model with computational refinements including dropout regularization.The authors aim to apply state-of-the-art language-model insights by reducing music modelling to a suitable form.Source: §3.1 · p. 3

3.2 Reduction to Univariate Prediction

  • The reduction orders notes in a chord as a tuple and factors its probability into successive univariate predictions, using an end symbol when tuple length is unknown.This note-wise approach is suited to sparse chords and requires fewer within-chord prediction steps than the described binary-indicator alternative.Source: §3.2 · p. 4

3.3 Data Representation

  • Notes are predicted sequentially in temporal order, then ascending pitch for simultaneous onsets; onset and offset times are assumed fixed rather than modelled.The figure representation presents input columns and predicts the MIDI number of the next note to turn on, with a lower bound for pitch ordering.Source: §3.3 · p. 4
  • The representation supports arbitrary timing rather than uniform time discretization and distinguishes repeated same-pitch notes by explicitly representing onsets and offsets.Its input also encodes rhythmic features such as event duration, time since the previous column, and current time in the piece.Source: §3.3 · p. 5

3.4 Data Augmentation

  • Data augmentation encodes domain-specific prior knowledge through transformed training examples, while directly encoding such knowledge in a model is described as more challenging.Source: §3.4 · p. 5
  • Transposing each training piece through all 12 major or minor keys gives a simple, widely applicable augmentation that substantially improves predictive power across the considered datasets.The procedure uses shifts of −6 through 5 semitones and avoids the need for key identification, including for pieces that modulate.Source: §3.4 · p. 5

4 Experiments

Experiments compare the model with prior piano-roll methods, then test it on MIDI data with unrestricted rhythmic structure. Transposition augmentation improves piano-roll results, while the new task demonstrates rhythmic flexibility and reveals octave structure in the learned embedding.

4.2 Benchmark Tests

  • Augmented LSTM convincingly outperforms the basic LSTM and prior approaches on all four piano-roll datasets, except DBN-LSTM; pooling with augmentation is best overall.The pooled augmented model has a small deterioration on the J.S. Bach test set, but cross-dataset training and testing perform worse than the pooled model.Source: Table 1 · p. 6, §4.2 · p. 7

4.1 Implementation Details and Experimental Setup

  • The implementation uses two LSTM layers, an 800-dimensional embedding, dropout keep fraction 0.45, batches of 50, and recurrence unrolled for 150 steps.Training typically took 10–20 epochs; sampling a complete piece took about a second.Source: §4.1 · p. 6

4.2 Benchmark Tests

  • The piano-roll comparison uses the same preprocessed data and train/test/validation splits as [10], omitting the forward-information row to match its modelling problem.The data are sets of notes on an eighth-note grid, making likelihoods directly comparable.Source: §4.2 · p. 7

4.3 A New Benchmark Problem

  • For the new task, MIDI events are represented by onset, offset, note, and part; the model predicts notes conditional on rhythm and ignores part numbers.The authors identify modelling individual-part coherence as future work.Source: §4.3 · p. 8
  • Timing is quantized to the nearest 96th of a quarter note unless that would collapse a note; offset quantization reduces columns and supports longer-range dependencies.Onset quantization also makes ordering of near-simultaneous notes more consistent.Source: §4.3 · p. 8
  • For 88 possible notes, the new task’s uniform-random baseline is approximately −4.47 per note, and its Table 3 results are qualitatively similar after scale adjustment.The new results are mean log-likelihood per note; the earlier setup reports per time step.Source: Table 3 · p. 7, §4.3 · p. 8
  • Prior piano-roll algorithms are impractical for this MIDI task because its minimum temporal resolution is 480 ticks per quarter note, versus two ticks in the earlier setup.The resolution follows from the greatest common divisor of note durations in the data.Source: §4.2 · p. 7, §4.3 · p. 8
  • Samples use the test pieces’ supplied rhythmic structures, and the learned embedding shows octave-related notes as functionally similar despite orthogonal raw inputs.The authors did not identify similarly clear higher-order relationships, while noting that the embedding is only the first layer of a multilayer model.Source: §4.3 · pp. 8–9

5 Summary and Outlook

The paper reduces polyphonic music modelling to a temporal sequence of univariate categorical variables, enabling use of neural language models. Transposition augmentation improves performance, and pooling adds a moderate improvement; the approach also supports fixed rhythmic structures beyond uniform piano-roll grids.

5 Summary and Outlook

  • The approach reduces polyphonic music modelling to a temporal sequence of univariate categorical variables, allowing use of neural language models from NLP.The paper reports a relatively large performance improvement from augmenting training data with transpositions through all musical keys, and a moderate additional improvement from pooling sources.Source: §5 · p. 9
  • The model handles any fixed rhythmic structure rather than requiring uniform time discretization, motivating a new non-piano-roll benchmark dataset.The paper identifies modelling rhythm, user constraints, and longer-range dependencies or motif discovery as future directions.Source: §5 · p. 9

Supplementary Material Modelling Symbolic Music: Beyond the Piano

The supplementary-material passage identifies an arXiv version and its date.

Supplementary Material Modelling Symbolic Music: Beyond the Piano

A.1 Introduction

The appendix introduces preprocessed symbolic-music data derived from the MIDI files used in. Its event-based representation records note starts and ends, addressing a limitation of piano-roll data.

A.1 Introduction

  • The preprocessed datasets come from the MIDI files used in, whose piano-roll representation uses a regular eighth-note grid of on/off indicators.The paper points readers to the data archive and summarizes the four datasets in Table 1.Source: §A.1 · pp. 13–14
  • The five-tuple representation describes each note event using piece, track, MIDI note number, start time, and end time.Times are recorded in ticks; 2400 ticks equal one beat, or one quarter note.Source: §A.1 · pp. 13–14
  • Unlike piano-roll data on an eighth-note grid, the event representation records when notes begin and end.This distinguishes two successive eighth notes of the same pitch from a single quarter note.Source: §A.1 · p. 13

A.2 Preprocessing

The preprocessing pipeline cleans MIDI note events and standardizes their timing. It also filters events and tracks that are unsuitable for the intended pitched-note modelling.

A.2 Preprocessing

  • Sustain-pedal signals are combined with key presses to recover individual note on/off events.Source: §A.2 · p. 14
  • Overlapping or duplicate notes on the same channel are removed sequentially by start time, ignoring events that overlap an already added note.The authors note that this cleanup is ill posed because MIDI software packages handle such cases differently.Source: §A.2 · p. 14
  • Channels with fewer than two note events and percussion tracks are removed.The former were information tracks; percussion is excluded because its MIDI numbers are not comparable to pitched instruments.Source: §A.2 · p. 14
  • Timing is resampled to 2400 ticks per quarter note, the lowest common multiple of the original MIDI file resolutions.This is the final listed processing step.Source: §A.2 · p. 14

A.3 Exploratory analysis

The exploratory appendix presents dataset summaries and plots of note counts, part counts, and peak polyphony. The datasets differ in their recorded part counts and in the sources of high polyphony.

A.3 Exploratory analysis

  • The exploratory plots include note distributions and the number of notes per piece, and Table 1 summarizes the datasets used.Source: §A.3 · p. 14, Table 1 · p. 15
  • Part counts vary by dataset: JSB is fixed at one, NOT has one or two, PMD reaches six, and MUS reaches 27.The JSB count reflects missing MIDI track information; many JSB pieces are in fact four-part harmonies. PMD’s maximum occurs in a three-part Bach fugue with left and right hands tracked separately.Source: §A.3 · pp. 14–15
  • Peak polyphony in PMD is mainly attributed to pedal-sustained musical runs, while in MUS it is mainly attributed to large orchestral works.Polyphonicity is the number of simultaneously sounding notes.Source: §A.3 · p. 15
Loading 1606.01368v1…