PaperlayerView exact revision on arXiv

Source-linked AI summary

Music Transformer

Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Ian Simon, Curtis Hawthorne, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, Douglas Eck

arXiv:1809.04281v3cs.LGcs.SDeess.ASstat.ML

TL;DR

Modeling music requires long-range repetition and relative timing, but existing relative attention uses quadratic memory on long sequences. The paper introduces a linear-memory relative-attention Transformer that generates structured compositions and achieves state-of-the-art results on Piano-e-Competition.

  • Problem

    Existing relative attention requires O(L^2D) memory, making it impractical for long musical sequences where relative timing matters.

  • Method

    The paper trains Transformer language models on serialized symbolic-music tokens with a memory-efficient relative-attention mechanism that reduces memory to O(LD).

  • Results

    The model generates minute-long compositions with long-term structure, coherent motif continuations, conditioned accompaniments, and state-of-the-art results on Piano-e-Competition.

  • Takeaways & Limitations

    Relative attention supports coherent long-range symbolic-music generation and enables applications such as expanding compositions from an initial motif.

  • Takeaways & Limitations

    The content-dependent relative-information formulation remains O(L^2D) and is not directly scalable beyond J.S. Bach Chorales.

Abstract

from arXiv · show

Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.

1 INTRODUCTION

Music generation requires modeling recurring musical elements and their relative timing over long sequences. This work adapts relative self-attention for music and reduces its memory cost, enabling Transformers to model minute-long expressive piano performances.

  • The model targets musical coherence by referencing distant motifs, phrases, and sections to repeat, vary, and develop them over time.
  • Relative attention captures timing and optionally pitch relations, extending Shaw et al. (2018) to improve music modeling and generation.
  • An event-based representation supports minute-long music at roughly 2K sequence length while preserving timing resolution below 10 milliseconds.
  • Transformers generate 60s (~2000 tokens) of expressive piano music with internal consistency and achieve state-of-the-art perplexity on these performances.
  • The proposed algorithm reduces relative attention’s memory complexity from O(L^2D) to O(LD), making long musical sequences practical.

2 RELATED WORK

Prior music-generation work has primarily used sequential models or alternative pianoroll-based architectures, while self-attention connects to self-similarity methods and offers a powerful but quadratically scaling mechanism for long sequences. These approaches motivate addressing efficient attention for musical modeling.

  • Sequence models: Sequence models, including HMMs, RNNs, LSTMs, and bidirectional LSTMs, have been canonical for music modeling, often requiring scores or performances to be serialized into a single sequence.Examples include Eck & Schmidhuber (2002), Liang (2016), Oore et al. (2018), and Hadjeres et al. (2017).
  • Pianoroll representations: Pianoroll-like representations can instead be decomposed into multi-hot pitch-vector sequences modeled with RBMs or NADEs.The passage also notes that pianorolls are image-like and can be modeled by CNNs.
  • Self-similarity and attention: Lattner et al. (2018) impose repetition structure through self-similarity templates, whereas self-attention generalizes self-similarity by using separate projections for queries and keys.Self-similarity uses the same projection for both components, while self-attention maps inputs through different projections.
  • Attention scaling: Self-attention has been applied to image generation, speech, and summarization, but its time and space complexity grows quadratically with sequence length.This scaling challenge is central to applying attention to long sequences.

3 MODEL

The model treats symbolic music as discrete-token language modeling and uses an autoregressive Transformer with relative attention to represent timing. A skewing procedure reduces relative-attention intermediate memory from O(L^2D) to O(LD), extending the approach to long and locally attended sequences.

  • Music representation: Music is serialized as a sequence of discrete tokens, with genre-specific representations for polyphony and time discretization.JSB Chorales use a raster-scan serialization of a four-voice pitch matrix discretized to sixteenth notes.
  • Transformer architecture: The autoregressive Transformer decoder uses self-attention, positional information, feedforward layers, and a causal mask that prevents attending to future positions.Each layer consists of self-attention followed by a feedforward sub-layer, while the upper-triangular mask enforces autoregressive prediction.
  • Relative attention: Relative attention supplies distance-aware timing information but the Shaw et al. (2018) formulation requires O(L^2D) intermediate space, limiting long-sequence application.It constructs an intermediate tensor containing embeddings for relative distances between all query and key positions.
  • Memory-efficient relative attention: O(LD) intermediate memory replaces O(L^2D) through a skewing algorithm that computes QEr^T and remaps absolute-by-relative indices into the required absolute-by-absolute logits.The procedure pads, reshapes, and slices the resulting matrix without instantiating the quadratic intermediate tensor R.
  • Local relative attention: For local attention, the same skewing idea supports non-overlapping blocks in which each block attends to itself and the preceding block.The local formulation uses relative embeddings shaped for the two-block context and produces the desired N-by-N relative-logit matrix.

4 EXPERIMENTS

Experiments show that relative-attention Transformers improve musical modeling across chorales and piano performances, achieving state-of-the-art Piano-e-Competition performance and producing more coherent motif-based continuations. Qualitative and listening evaluations further support improved structure and perceived quality, while attribute-specific relative attention has limited scalability beyond chorales.

  • Relational attention: Separate relative embeddings for timing and pitch capture additional relational information, but this approach is not directly scalable beyond J.S. Bach Chorales.Timing embeddings represent sixteenth-note distance, while pitch embeddings represent pairwise pitch intervals.
  • J.S. Bach Chorales: On J.S. Bach Chorales, adding relative attention and richer timing or relational information improves note-wise validation NLL.The evaluation uses score-based four-part chorales discretized on a 16th-note grid; concatenating rather than adding positional sinusoids further improves both baseline and relative Transformers.
  • Piano-e-Competition: Transformer with relative attention achieves state-of-the-art validation performance on Piano-e-Competition, outperforming the LSTM-based models considered.The comparison uses event-based representation with sequence length L = 2048; Table 3 reports validation NLL.
  • Motif continuation: Relative-attention Transformer continuations elaborate a primed motif into repeated and varied phrases with clear contour, whereas baseline Transformer and LSTM outputs reuse it less coherently.The Chopin’s Étude Op. 10, No. 5 samples are twice as long as the training sequences.
  • Listening study: In listening tests, relative attention significantly improves perceived sample quality over the baseline Transformer both in aggregate and head-to-head comparisons.The study compared the baseline Transformer, relative-attention Transformer, PerformanceRNN, and validation-set excerpts using the same priming sequence.

5 CONCLUSION · A DOMAIN-SPECIFIC REPRESENTATIONS

The paper concludes that relative attention makes Transformers well-suited to symbolic-music generation, producing compelling long-term structure and enabling motif expansion. It also argues that domain-specific serialization choices and memory-efficient relative attention support broader sequence-modeling applications.

  • 5 CONCLUSION: Relative attention makes the Transformer well-suited for generative modeling of symbolic music, with samples exhibiting compelling long-term structure.The authors view this result as encouraging for further research in music generation.
  • 5 CONCLUSION: The model can expand a given primer, suggesting potential applications as a creative tool.
  • 5 CONCLUSION: The significant improvement from relative attention exposes a shortcoming of the original Transformer that may also limit performance in other domains.The authors suggest that modeling periodicity across time scales and relations between scalar features such as pitch could improve time-series models.
  • 5 CONCLUSION: The memory-efficient relative-attention implementation enables processing much longer sequences, including long texts and potentially audio waveforms.
  • A DOMAIN-SPECIFIC REPRESENTATIONS: Adapting sequence models for music requires deciding how to serialize polyphonic texture.
  • A DOMAIN-SPECIFIC REPRESENTATIONS: The choice between score and performance data determines which representations naturally encode the required information while keeping sequence lengths reasonable.

A.1 SERIALIZED INSTRUMENT/TIME GRID (J.S.BACH CHORALES)

The J.S. Bach Chorales dataset contains four-part score-based choral music represented on a serialized grid with sixteenth-note time resolution. The grid organizes the soprano, alto, tenor, and bass voices by MIDI pitch across discretized time.

  • A.1 SERIALIZED INSTRUMENT/TIME GRID (J.S.BACH CHORALES): The J.S. Bach Chorales dataset consists of four-part score-based choral music encoded at sixteenth-note time resolution.This resolution supports a serialized grid-like representation of the music.
  • A.1 SERIALIZED INSTRUMENT/TIME GRID (J.S.BACH CHORALES): The grid places the four voices—soprano, alto, tenor, and bass—on MIDI-pitch rows and discretized-time columns.Figure 6 illustrates the conversion from pianoroll to grid representation.
  • A.1 SERIALIZED INSTRUMENT/TIME GRID (J.S.BACH CHORALES): Figure 6 encodes the opening measure of BWV 428 with quarter-note soprano and alto parts and eighth-note tenor and bass parts.The example specifies pitches G4 and D4 for soprano and alto, B3 and A3 for tenor, and A2 and G2 for bass.

A.2 MIDI-LIKE EVENT-BASED (PIANO-E-COMPETITION)

Piano-e-Competition contains polyphonic piano performances with millisecond-level expressive timing and dynamics, making grid representations impractically long. The performances are therefore serialized as one-hot event sequences with sustain-aware preprocessing and a defined MIDI event vocabulary.

  • Millisecond-level timing and dynamics make grid representations too long, so polyphonic performances are serialized as one-hot event sequences, following Oore et al. (2018).
  • MIDI preprocessing extends note durations during sustain-pedal periods until the next same-pitch note or the sustain period’s end, whichever occurs first.The pedal is down at control values >= 64 and up after values < 64.
  • The vocabulary contains 128 NOTE_ON, 128 NOTE_OFF, 100 TIME_SHIFT, and 32 SET_VELOCITY event types.TIME_SHIFT events represent 10ms-to-1s shifts, while SET_VELOCITY events quantize 128 MIDI velocities into 32 bins.
  • The encoding represents polyphonic timing, sustain, note duration, and dynamics through serialized performance events, illustrated by an arpeggiated C-major chord and later F note.The example includes pedal release at 2 seconds and velocities of 80 for the C chord and 100 for the F.

B SUPPLEMENT OF LISTENING TEST · B.1 STUDY PROCEDURE

The listening test compared model-generated and validation-set continuations after a shared priming sequence, using paired excerpts with controlled timing and 512-event continuations.

  • B.1 STUDY PROCEDURE: Participants heard paired excerpts sharing a priming sequence, separated by 2.5 seconds of silence before the repeated sequence and continuation.Continuations came from one of the models or the validation set.
  • B.1 STUDY PROCEDURE: The study evaluated all possible continuation pairs across data and model samples, excluding pairs drawn from the same model.Each continuation was 512 events long.

B.2 ANALYSIS

Participants rated the relative Transformer as more musical than the baseline Transformer, while its advantage over LSTM was not statistically significant; real music remained better than LSTM and baseline Transformer but did not differ significantly from the relative Transformer.

  • B.2 ANALYSIS: The relative Transformer was rated more musical than the baseline Transformer both within matched pairs and across aggregate comparisons (p < 0.01/6).The overall model-rating difference was statistically significant: χ2(2) = 63.84, p = 8.86e-14 < 0.01.
  • B.2 ANALYSIS: Within matched pairs, no consistent statistically significant difference was observed between baseline Transformer versus LSTM or LSTM versus relative Transformer.Pairwise comparisons used the Wilcoxon signed-rank test with a significance threshold of 0.01/6 = 0.0016.
  • B.2 ANALYSIS: Across aggregates, LSTM was perceived as more musical than the baseline Transformer, while the relative Transformer narrowly missed outperforming LSTM (p = 0.018).The relative Transformer often exhibited more structure, but the 10–15-second listening samples may have reduced this difference.
  • B.2 ANALYSIS: Real music was rated more musical than LSTM and the baseline Transformer, but did not differ significantly from the relative Transformer in aggregate comparisons.The authors attribute this result partly to the short samples, while noting that real music is still qualitatively better.

C VISUALIZING SOFTMAX ATTENTION · D PREVIOUS FIGURES FOR THE “SKEWING” PROCEDURE

The visualizations expose how relative-attention Transformers distribute softmax attention across prior musical events and recurring structures. Additional figures illustrate the skewing procedures for converting relative-indexed attention matrices into absolute-indexed forms.

  • C VISUALIZING SOFTMAX ATTENTION: Attention visualizations show which previous memories receive higher softmax probability, revealing how the model may build recurring structures and how far it attends backward.The visualized pianorolls come from a Transformer with relative attention; each figure includes a query and attended previous notes.
  • C VISUALIZING SOFTMAX ATTENTION: Figure 8 shows a query at a later peak attending to all previous high notes on the recurring triangular contour, back to the piece’s beginning.The figure visualizes long-range attention to repeated high-note peaks.
  • C VISUALIZING SOFTMAX ATTENTION: Figure 9 shows a left-hand query attending to immediate past neighbors and mainly earlier left-hand chords in the lower pianoroll.Most attention lines are distributed in the lower half of the pianoroll.
  • D PREVIOUS FIGURES FOR THE “SKEWING” PROCEDURE: Figure 10 depicts global attention skewing from an absolute-by-relative (i_q, r) matrix to absolute-by-absolute (i_q, j_k), including masks and removed entries.Relative-distance-zero positions are marked, and purple outlines identify entries removed in step 3.
  • D PREVIOUS FIGURES FOR THE “SKEWING” PROCEDURE: Together, the skewing figures distinguish global and local relative-attention transformations while showing how relative distances map into absolute attention positions.Figure 10 includes self-attention masks and introduced entries, whereas Figure 11 emphasizes the 2N −1 local relative-index range.
  • D PREVIOUS FIGURES FOR THE “SKEWING” PROCEDURE: Figure 11 depicts local attention skewing for an (i_q, r) matrix with 2N −1 relative indices into an (i_q, j_k)-indexed matrix.The figure labels box shapes and relative distances during the procedure.
Loading 1809.04281v3…