PaperlayerView exact revision on arXiv

Source-linked AI summary

Encoding Musical Style with Transformer Autoencoders

Kristy Choi, Curtis Hawthorne, Ian Simon, Monica Dinculescu, Jesse Engel

arXiv:1912.05537v2cs.SDcs.LGeess.ASstat.ML

TL;DR

The paper tackles the challenge of controlling global structure in symbolic music generation without exhaustive performance labels. It introduces a Transformer autoencoder that aggregates performance encodings into a global style representation and combines it with melody embeddings. Across MAESTRO and a 10,000+ hour YouTube piano dataset, the method improves likelihood and style similarity relative to baselines.

  • Problem

    The work addresses how to learn and use global conditioning for controllable symbolic music generation when comprehensive performance annotations are difficult to obtain.

    Source: §1 · p. 1

  • Method

    The Transformer autoencoder aggregates performance encodings across time into an unsupervised global style representation that can be combined with melody embeddings.

    Source: §1 · p. 1

  • Results

    48% higher similarity for YouTube and 45% higher similarity for MAESTRO were achieved relative to an unconditional baseline.

    Source: §5.2 · p. 5

  • Takeaways & Limitations

    The model enables separate control of performance style and melody, including adaptation to a particular style from a single input performance.

    Source: §1 · pp. 1–2

Abstract

from arXiv · show

We consider the problem of learning high-level controls over the global structure of generated sequences, particularly in the context of symbolic music generation with complex language models. In this work, we present the Transformer autoencoder, which aggregates encodings of the input data across time to obtain a global representation of style from a given performance. We show it is possible to combine this global representation with other temporally distributed embeddings, enabling improved control over the separate aspects of performance style and melody. Empirically, we demonstrate the effectiveness of our method on various music generation tasks on the MAESTRO dataset and a YouTube dataset with 10,000+ hours of piano performances, where we achieve improvements in terms of log-likelihood and mean listening scores as compared to baselines.

1. Introduction

The paper addresses controllable symbolic music generation by learning global performance-style representations without requiring exhaustive annotations. Its Transformer autoencoder combines these representations with melody conditioning to control style and melody separately.

1. Introduction

  • Transformers lack a clear mechanism for learning and incorporating global conditioning at inference, while comprehensive performance annotations are difficult and expensive to obtain.These challenges make desired musical features ambiguous without descriptive tags.Source: §1 · p. 1
  • The Transformer autoencoder aggregates encoder outputs across time to learn an unsupervised holistic representation of performance style.The representation can be combined with structural conditioning such as melody embeddings.Source: §1 · p. 1, Figure 1 · p. 2
  • The model supports both style-conditioned generation from a performance and melody harmonization in the style of another performance.These tasks separately manipulate performance style and melody without explicit labels.Source: §1 · pp. 1–2
  • The method was evaluated on MAESTRO and a YouTube corpus containing more than 10,000 hours of piano performances.It generates performances similar in style to inputs and accompaniments that follow a given style, including from one input performance.Source: §1 · pp. 1–2

2. Preliminaries

The work represents expressive piano performances as discrete token sequences and builds on Music Transformer’s relative-attention architecture for long-term musical coherence.

2.1. Data Representation for Music Generation

  • The datasets include over 1,100 MAESTRO performances and approximately 400K YouTube performances spanning more than 10,000 hours.Both datasets are represented as sequences of discrete tokens for language modeling.Source: §2.1 · p. 2
  • The performance vocabulary encodes note onsets, note offsets, time shifts, and quantized velocity values to capture expressive dynamics and timing.It contains 128 note-on events, 128 note-off events, 100 time-shift events, and 32 velocity bins.Source: §2.1 · p. 2

2.2. Music Transformer

  • Music Transformer replaces absolute positional encodings with relative attention to track event-order regularities and periodicity more effectively.Its memory-efficient implementation enables generation of sequences spanning over a minute.Source: §2.2 · p. 2

3. Generation with Transformer Autoencoder

The model uses Transformer encoders and a decoder to compress performance information into a global style representation, combine it with melody when needed, and train autoregressively for conditional generation.

3.1. Model Architecture

  • The architecture uses six-layer Transformer encoders for performance and melody inputs, while the decoder adds attention over encoder outputs during generation.The decoder conditions each next token on encoder outputs and previously generated tokens.Source: §3.1 · pp. 2–3

3.2. Conditioning Tasks and Mechanisms

  • Mean-aggregating performance embeddings across time creates a bottlenecked global style representation that the decoder attends to when predicting outputs.The bottleneck discourages memorization and is suited to generating samples similar in style to the input.Source: §3.2 · p. 3
  • For melody-and-performance conditioning, separate encoders represent the melody and performance before their embeddings are combined for decoding.The melody is quantized to a 100ms grid and encoded with a vocabulary distinct from the performance representation.Source: §3.2 · p. 3
  • The model compares sum, concatenate, and tile strategies for combining melody and mean-aggregated performance embeddings.The best combination method varies across datasets.Source: §3.2 · p. 3

3.3. Model Training

  • Training perturbs conditioning performances by pitch shifts up to 6 semitones and timing stretches of at most ±5% to improve generalization.At test time, inputs remain unperturbed.Source: §3.3 · p. 3
  • The model is trained end-to-end by maximum likelihood, reconstructing the input performance or predicting a new performance from combined melody and performance embeddings.The objective maximizes autoregressive log pθ(x) over sequence tokens.Source: §3.3 · p. 3

4. Performance Similarity Evaluation

The evaluation represents performance style through eight pitch- and rhythm-related features, then compares performances using windowed feature distributions and their overlap. It also reports language-model likelihood results across datasets and conditioning variants.

4. Performance Similarity Evaluation

  • The evaluation captures performance style with eight features spanning pitch and rhythm, including note density, pitch range, pitch statistics, velocity, and duration.These features are intended to characterize rhythmic, melodic-register, expressive-volume, articulation, dynamics, and phrasing properties.Source: §4 · pp. 3–4
  • Conditional models outperformed unconditional counterparts; concatenate outperformed other baselines on YouTube, while sum worked best in one melody-conditioned setting.Table 2 reports note-wise test NLL on MAESTRO and YouTube with melody conditioning.Source: §4 · p. 4, Table 2 · p. 4

4.1. Overlapping Area (OA) Metric

  • 2s sliding-window histograms preserve changing dynamics and stylistic motifs better than a single performance-wide scalar.The authors use relative feature measurements within windows rather than averages such as overall note density.Source: §4.1 · p. 4
  • Gaussian-smoothed feature histograms yield per-feature means and variances, whose Overlapping Area measures similarity between two performances.OA identifies feature-wise similarities, while OAavg summarizes similarity across all features.Source: §4.1 · p. 4
  • The OA metric was less sensitive to melody-specific features than KL and symmetrized KL divergences, and identified relevant characteristics in generated performances.This supports using OA to quantify performance-style similarity rather than relying on divergences that respond more strongly to melody.Source: §4.1 · p. 4
  • For melody-and-performance conditioning, similarity averages OA to the generated sample from both the source performance and conditioning performance.This accounts for contributions from the extracted melody and the separate conditioning performance.Source: §4.1 · p. 4

5. Experimental Results

Across MAESTRO and YouTube experiments, the Transformer autoencoder improves likelihood, conditioning similarity, latent interpolation, and perceived control over generated performances. Separate performance and melody conditioning supports style transfer while preserving melody-related variation.

5.1. Log-Likelihood Evaluation

  • The encoder-bottleneck Transformer autoencoder outperformed other baselines, while conditional variants outperformed their unconditional counterparts.Source: §5.1 · p. 5
  • Concatenation achieved the lowest NLL on YouTube, whereas summation outperformed other embedding combinations on MAESTRO.Source: §5.1 · p. 5

5.2. Similarity Evaluation

  • Performance-conditioned samples had 48% higher OA similarity than the unconditional baseline on YouTube and 45% higher similarity on MAESTRO.The evaluation generated one sample for each of 500 test-set conditioning signals.Source: §5.2 · p. 5
  • The Melody & Performance autoencoder performed best overall across almost all evaluated features against melody-only, performance-only, and unconditional baselines.The comparison used 717 conditioning performances and 717 distinct performances for melody extraction.Source: §5.2 · p. 6

5.3.1. PERFORMANCE AUTOENCODER

  • For the YouTube dataset, normalized relative distance from performance A to interpolated samples slowly increased as α rose from 0 to 1, with a similar trend for MAESTRO.The relative distance was averaged across all features and elements in the set.Source: §5.3.1 · p. 6, Figure 3 · p. 6

5.3.2. MELODY & PERFORMANCE AUTOENCODER

  • With melody held fixed, performance interpolation increased distance from performance A while keeping distance to the conditioning melody fairly constant, suggesting separate control of style and melody.The interpolation effect was slightly weaker because generated samples also depended on the conditioning melody.Source: Figure 2 · p. 6, §5.3.2 · p. 7

5.4.1. PERFORMANCE CONDITIONING

  • Human evaluations showed a robust effect of performance conditioning, with the conditioned sample preferred 58 times in 164 Ground Truth-versus-Conditioned comparisons.The listening studies collected 492 performance-conditioning ratings and 714 melody-and-performance ratings overall.Source: §5.4.2 · p. 7, §5.4.1 · p. 7

5.4.2. MELODY AND PERFORMANCE CONDITIONING

  • Listening tests found that combined melody-and-performance conditioning outperformed performance-only conditioning and remained competitive with melody-only conditioning.The difference between melody-only and combined conditioning was not statistically significant, with p = 0.0894.Source: §5.4.2 · pp. 7–8

6. Related Work

The paper situates its Transformer autoencoder among sequential autoencoders and controllable music-generation methods. Its distinctive choice is to learn an unsupervised global style representation directly from performance data rather than relying on labeled attributes.

  • The similarity evaluation extends symbolic-music measures to polyphonic piano performances rather than monophonic melodies.Source: §6 · p. 8
  • The Transformer autoencoder bridges sequence-to-sequence modeling, attention-based Transformers, and sequential autoencoders for musical sequences.Source: §6 · p. 8
  • Its deterministic architecture uses self-attention with relative positioning in both encoder and decoder to capture local and global features.Source: §6 · p. 8
  • Unlike methods conditioned on predefined attributes or labeled tokens, it learns a global style representation from conditioning data in an unsupervised way.Source: §6 · p. 8

7. Conclusion

The paper concludes that the Transformer autoencoder supports conditional music generation with long-term structure and enables adaptation to a single input performance. Experiments on MAESTRO and YouTube show style-similar outputs relative to baselines, while future work targets distribution shifts.

  • The model generates style-similar samples from varied conditioning signals across MAESTRO and YouTube, both quantitatively and qualitatively, relative to baselines.Source: §7 · p. 8
  • A single input performance can be used to adapt the outputs of the generative model.Source: §7 · p. 8
  • Future work includes variational or few-shot training for inputs drawn from distributions that differ slightly from the training set.Source: §7 · p. 8

Supplementary Material

The supplementary material specifies how melodies are represented and extracted from polyphonic performances. It uses a compact token vocabulary, temporal quantization, and Viterbi inference based on pitch transitions.

A. Additional Details on Melody Representation

B. NLL Evaluation for ”Noisy” Model

The supplementary material identifies the noisy-model NLL evaluation and records architecture settings for the datasets. The listed MAESTRO configuration follows the setup of Huang et al. (2019b).

B. NLL Evaluation for ”Noisy” Model

C. Model Architecture and Hyperparameter Configurations

  • The supplementary architecture section states that the implementation mostly uses Tensor2Tensor’s default Transformer with eight self-attention heads.Source: C. Model Architecture and… · p. 11

C.1. MAESTRO

  • The MAESTRO configuration follows the hyperparameter setup of Huang et al. (2019b).Source: §C.1 · p. 11

C.2. YOUTUBE DATASET

The YouTube setup uses eight hidden layers with slightly increased dropout, while interpolation experiments show consistent relative-distance behavior across Transformer autoencoders.

C.2. YOUTUBE DATASET

  • Eight hidden layers and slightly increased dropout are used for the YouTube dataset.Source: §C.2 · p. 11

D. Additional Relative Distance Interpolations

E. Internal Dataset Performance Interpolations

The YouTube experiments illustrate latent-space interpolation for performance and melody-conditioned performance autoencoders, with the conditioning melody held fixed in one interpolation.

E. Internal Dataset Performance Interpolations

  • Figures 6 and 7 demonstrate latent-space interpolation effects on YouTube piano-roll performances for both Transformer autoencoder variants.Source: E. Internal Dataset Performance… · p. 11
  • Table 5 reports note-wise test NLL for melody-conditioned MAESTRO and YouTube piano performances using event-based sequences of length L = 2048.Source: Table 5 · p. 12
  • The interpolation coefficient α controls movement between the latent encodings of starting and final performances.Source: Figure 6 · p. 12
  • In one interpolation, the original conditioning melody remains fixed throughout the transition between performances.Source: Figure 7 · p. 13
Loading 1912.05537v2…