PaperlayerView exact revision on arXiv

Source-linked AI summary

Theme Transformer: Symbolic Music Generation with Theme-Conditioned Transformer

Yi-Jen Shih, Shih-Lun Wu, Frank Zalkow, Meinard Müller, Yi-Hsuan Yang

arXiv:2111.04093v2cs.SDcs.MMeess.AS

TL;DR

Prompt-based conditioning does not ensure that a musical idea will recur or develop in a generated continuation. The paper retrieves recurring themes and conditions an encoder–decoder Transformer on them; evaluations report repetition with perceptible variations and improved generated-music quality within the tested scope.

  • Problem

    Prompt-based conditioning does not guarantee that the conditioning sequence will recur or develop in a continuation, while clear recurrence and development remain difficult in conditional open-ended generation.

    Source: §I · p. 1, §II · p. 2

  • Method

    The approach retrieves recurring fragments using contrastive melody representations and clustering, then uses a jointly trained encoder and decoder with gated parallel attention to condition generation on a theme.

    Source: §I · p. 2

  • Results

    The Theme Transformer outperforms the prompt-based baseline and basic seq2seq model across listening-test metrics, with MOS similar to original data on theme-related metrics.

    Source: B. Listening Test: Result · p. 9

  • Takeaways & Limitations

    Objective and subjective evaluations report that Theme Transformer repeats thematic material with perceptible variations and enhances generated-music quality compared with the decoder-only Music Transformer and simple Seq2seq Transformer.

    Source: §X · p. 11

    The reported evaluations concern polyphonic piano music generated and assessed using the POP909 dataset.

    Source: §I · p. 2

Abstract

from arXiv · show

Attention-based Transformer models have been increasingly employed for automatic music generation. To condition the generation process of such a model with a user-specified sequence, a popular approach is to take that conditioning sequence as a priming sequence and ask a Transformer decoder to generate a continuation. However, this prompt-based conditioning cannot guarantee that the conditioning sequence would develop or even simply repeat itself in the generated continuation. In this paper, we propose an alternative conditioning approach, called theme-based conditioning, that explicitly trains the Transformer to treat the conditioning sequence as a thematic material that has to manifest itself multiple times in its generation result. This is achieved with two main technical contributions. First, we propose a deep learning-based approach that uses contrastive representation learning and clustering to automatically retrieve thematic materials from music pieces in the training data. Second, we propose a novel gated parallel attention module to be used in a sequence-to-sequence (seq2seq) encoder/decoder architecture to more effectively account for a given conditioning thematic material in the generation process of the Transformer decoder. We report on objective and subjective evaluations of variants of the proposed Theme Transformer and the conventional prompt-based baseline, showing that our best model can generate, to some extent, polyphonic pop piano music with repetition and plausible variations of a given condition.

I. INTRODUCTION

Prompt conditioning does not ensure that a musical idea recurs or develops, so the paper introduces theme-based conditioning with a dedicated memory for the theme. It combines automatic theme retrieval with an encoder–decoder Transformer and evaluates the approach on polyphonic piano music.

I. INTRODUCTION

  • Prompt-based generation does not guarantee that a condition or earlier material will recur meaningfully, and its influence may diminish as a piece grows.Figure 1 illustrates the drift from a piece’s opening fragment.Source: §I · p. 1, Figure 1 · p. 1
  • Theme-based conditioning selects a fragment that recurs in its training piece and treats it as thematic material to be developed through repetition and variation.The approach gives the condition a separate memory alongside the decoder’s memory of previously generated music.Source: §I · pp. 1–2
  • The paper retrieves themes by clustering fragments in a contrastively learned embedding space and choosing the earliest fragment in the largest cluster.Musically informed augmentation and density-based clustering support the retrieval process.Source: §I · p. 2
  • A jointly trained Transformer encoder supplies the condition memory through cross-attention, while gated parallel attention and theme-related design changes address longer generations.The proposed modifications also include theme-related tokens and theme-aligned positional encoding.Source: §I · p. 2
  • Objective and listening-test evaluations report theme controllability and variation similar to original POP909 test pieces.The paper also provides example generated music and source code.Source: §I · p. 2

II. RELATED WORK

Prior music-generation work has advanced with deep learning and Transformers, but generating clear phrases with recurrence and development remains difficult. The paper contrasts prompt-based conditioning with a separate encoder memory for theme-based generation.

II. RELATED WORK

  • Transformer models can generate coherent minute-long polyphonic piano music with local repetitions and variations, building on earlier hierarchical RNN approaches.The cited hierarchical model uses a higher-level RNN for bar-level dependencies and a lower-level RNN for notes within bars.Source: §II · p. 2
  • Generating clear musical phrases and sections with recurrence and development remains difficult, and prior structural-generation efforts were not designed for conditional, open-ended development of a given idea.Recent deep models often use an opening fragment as a prompt without assigning it a specific musical role.Source: §II · pp. 2–3

III. PROMPT-BASED VS. THEME-BASED GENERATION

  • Prompt-based conditioning treats the condition as the target sequence’s prefix and generates a continuation from the next timestep.The condition influences later elements through the decoder’s generative modeling.Source: §III · p. 3
  • Theme-based conditioning starts generation at timestep 1 and uses both decoder self-attention and cross-attention to an encoder representing the condition.Training pairs require a condition that repeats, possibly with variations, in the target sequence.Source: §III · p. 3

IV. FINDING THEMATIC CONDITIONING MATERIALS

The paper addresses the difficulty of obtaining musically meaningful theme labels by retrieving recurring fragments automatically. It partitions music into fragments, learns melody embeddings with contrastive learning, and clusters fragments to identify candidate themes.

IV. FINDING THEMATIC CONDITIONING MATERIALS

  • Manual theme labeling is tedious and requires musical expertise, while the available public theme dataset is limited to Western classical music and does not mark every theme occurrence.Automatic detection of motifs or themes also remains challenging.Source: §IV · p. 3

A. Sketch of our approach

  • The retrieval approach clusters fragments in a learned embedding space and uses the earliest fragment in the largest cluster as the piece’s thematic condition.The largest cluster represents the most recurrent group of similar fragments; contrastive learning supplies the fragment representation and distance measure.Source: A. Sketch of our approach · p. 3

B. Segmentation

  • Because tested segmentation algorithms produced noise and over-segmentation, the method instead slices pieces into non-overlapping two-bar fragments at bar lines, disregarding bars without melody.The authors assume a two-bar fragment carries enough information to influence generation; their segmentation can omit the opening notes of phrases that begin with an anacrusis.Source: B. Segmentation · pp. 3–4, C. Melody embedding learning by… · p. 4

C. Melody embedding learning by contrastive learning

  • The melody embedding represents melody notes as a fixed-length vector, with a BERT-like bidirectional self-attention network and a mean of its final-layer hidden states.The embedding is trained by contrastive loss to bring positive-pair fragments closer than negative-pair fragments.Source: C. Melody embedding learning by… · p. 4
  • Contrastive training treats augmented versions of a fragment as positive pairs and fragments from different pieces as negative pairs, making the representation invariant to the chosen variations.Augmentations include scale-based pitch shifts, last-note duration changes, and note splitting or combination; the implementation uses six bidirectional self-attention layers.Source: C. Melody embedding learning by… · p. 4

D. Clustering

DBSCAN clusters a piece’s melody fragments in a learned embedding space, forming a variable number of clusters according to fragment similarity. The method discards fragments that do not belong to any cluster, while the conditioning excerpt includes both melody and accompaniment.

D. Clustering

  • DBSCAN forms a variable number of clusters per piece, requiring each included fragment to lie within ε of its cluster centroid in melody-embedding space.The minimum cluster size is 2, and ε is set to 0.13 based on toy sequences with perceptually tolerable alternations.Source: D. Clustering · pp. 4–5
  • Fragments that belong to no DBSCAN cluster are discarded.The resulting cluster-count and theme-cluster-size distributions are shown in Figure 3.Source: D. Clustering · p. 5
  • Only a fragment’s melody is used for clustering, but the conditioning excerpt contains both melody and accompaniment.Source: D. Clustering · p. 5

V. THEME-CONDITIONED TRANSFORMER

The seq2seq model encodes the conditioning theme separately from the generated segment, allowing the decoder to attend to both its own history and the condition. Theme-region markers and segment embeddings distinguish thematic from non-thematic tokens.

V. THEME-CONDITIONED TRANSFORMER

  • A seq2seq Transformer encodes the condition and generates a target segment using decoder self-attention and cross-attention to the encoder.The encoder uses bidirectional self-attention layers, while each decoder layer cascades unidirectional self-attention and cross-attention.Source: A. Seq2seq Model · p. 5
  • THEME-START and THEME-END tokens mark two-bar fragments assigned to a theme cluster, dividing the sequence into theme and non-theme regions.Source: A. Seq2seq Model · p. 5
  • Original test pieces have average theme inconsistency and uncontrollability values below ε.Source: A. Seq2seq Model · p. 5
  • The condition can influence tokens in non-theme regions without explicitly appearing in them.Source: A. Seq2seq Model · p. 5
  • A binary theme-region mask and an all-ones encoder mask pass through shared segment embeddings into the encoder and decoder inputs.The decoder mask marks theme-region tokens with 1 and other tokens with 0.Source: A. Seq2seq Model · p. 5

B. Proposed Theme Transformer

Long music sequences and the distance between a condition and later generated material can make it difficult for the decoder to use the encoder. The proposed Theme Transformer uses parallel attention with XOR gating and separate positional encodings to address these issues.

B. Proposed Theme Transformer

  • For target sequences that may exceed 4,000 tokens, vanilla positional encoding can weaken the condition’s influence at later timesteps, and the decoder may ignore the encoder.The source and target relationship may also be subtler than in the cited NLP tasks.Source: B. Proposed Theme Transformer · p. 5

1) Gated parallel attention:

  • Parallel self- and cross-attention routes use XOR gating: non-theme positions use self-attention, while theme positions use cross-attention in some layers.Source: 1) Gated parallel attention · pp. 5–6
  • Within theme regions, self-attention is turned off only in the top half of decoder layers, preserving its contribution to local musical continuity.Other tested placements of the XOR gates showed no notable differences.Source: 1) Gated parallel attention · p. 6

2) Theme-aligned positional encoding:

  • Separate positional encodings align decoder theme regions with the condition by resetting cross-attention positions to PE(1) after THEME-START and advancing them until THEME-END.This addresses the positional gap between a late generated theme and the earlier condition; positional encodings are injected in every layer.Source: 2) Theme-aligned positional encoding · p. 6
  • The proposed decoder combines parallel attention with XOR gating and separate positional encodings, and drops segment embeddings.Source: 2) Theme-aligned positional encoding · p. 6

VI. IMPLEMENTATION DETAILS

The implementation uses a filtered POP909 piano-cover dataset, separate piano and melody token representations, and matched six-layer theme-conditioned models. Training sequences are 512 tokens, while generated pieces are truncated at 64 bars for evaluation.

A. Dataset

  • After filtering POP909 piano covers for 4/4 meter and songs without key changes, the dataset contains 713 songs, with 29 reserved for testing.The BRIDGE track is omitted; note onsets and durations are quantized to quarter-beat multiples, ignoring triplets.Source: A. Dataset · p. 6

B. Token representation

  • The piano representation uses note, metric, and theme tokens; the prompt-based baseline omits theme tokens, and the vocabulary contains 730 unique tokens.Songs average about 95 bars and 5,249 tokens; 512-token training sequences average about nine bars, and two-bar conditions average 121.9 tokens. A separate melody representation supports contrastive retrieval; generated sequences have no end-of-song token and are truncated at bar 64 for evaluation.Source: B. Token representation · pp. 6–7, Table I · p. 6

C. Model settings

  • The basic seq2seq and Theme Transformer each use six-layer encoders and decoders with 11M parameters, while the six-layer prompt-based baseline has 5M parameters.Models train on 2,529 sequences; theme-conditioned models use sequences containing theme markers, while the prompt-based model uses random subsequences. At inference, theme-based models receive a THEME-START prefix.Source: C. Model settings · p. 7

VII. OBJECTIVE EVALUATION

The objective evaluation measures both generated-piece coherence and how the generated music relates to its theme condition. It uses general music metrics alongside newly proposed theme-specific metrics.

A. Evaluation Setup and Metrics

  • Each model generates 64-bar piano music for conditions retrieved from 29 POP909 test songs, evaluated for coherence and condition–generation relationships.The objective metrics are divided into general measures of generated music and theme-related measures.Source: A. Evaluation Setup and Metrics · p. 7
  • Pitch-class consistency, melody inconsistency, and grooving consistency assess key adherence, melodic stability, and rhythmic coherence, respectively.Melody inconsistency compares the opening two-bar melody embedding with the closest fragment among the latter 32 bars.Source: A. Evaluation Setup and Metrics · p. 7
  • Theme inconsistency compares melody embeddings across distinct theme regions, while theme uncontrollability measures their closeness to the condition.Theme gap measures the number of bars between successive theme starts.Source: A. Evaluation Setup and Metrics · p. 7

B. Main Result

The Theme Transformer improves objective theme following over comparison models, while longer sequences and separate positional encodings affect different evaluation metrics. Its theme timing still differs from original pieces.

B. Main Result

  • The Theme Transformer harnesses the condition better than the basic seq2seq model, but remains far from originals in theme uncontrollability and theme gap.The reported comparison suggests the theme-aligned positional encoding and gated parallel attention are effective.Source: B. Main Result · pp. 7–8
  • The Theme Transformer performs similarly to other models on pitch-class consistency and better on melody consistency; all models show overly high grooving consistency versus originals.Table II reports average results across the test songs; higher grooving consistency suggests less rhythmic diversity.Source: B. Main Result · p. 7, Table II · p. 8
  • The model has difficulty learning when to reuse a theme, possibly because 512-token training sequences limit learning long-range relations between theme occurrences.The authors identify this as a possible explanation for the theme-gap result.Source: B. Main Result · p. 8

1) On sequence length:

  • Increasing sequence length from 512 to 1024 improves melody inconsistency, theme inconsistency, and theme uncontrollability, with theme-gap statistics closer to originals.The authors suggest longer sequences or whole-song training with a linear-complexity Transformer as potential avenues for further improvement.Source: 1) On sequence length · p. 8, B. Main Result · p. 8

2) On segment embedding vs. theme-aligned positional encoding:

  • Using segment embedding instead of separate positional encodings degrades all metrics, especially theme uncontrollability; the ablation attributes the first two theme metrics to gated parallel attention and the latter to separate positional encodings.The ablation study probes model variants to validate design choices.Source: 2) On segment embedding vs.… · p. 8, C. Ablation Study · p. 8

3) On the number of trainable parameters:

The authors compare a parameter-matched prompt baseline and describe the setup for a human listening test. The test covers theme ratings and overall musical judgments across listener groups.

3) On the number of trainable parameters:

  • The prompt-based baseline is expanded to 12 decoder layers to match the Theme Transformer’s parameter count, but additional layers do not improve its result.The Theme Transformer has roughly twice as many trainable parameters as the original decoder-only baseline.Source: 3) On the number of trainable parameters · p. 8

A. Listening Test: Setup

  • Listeners rate theme controllability, repetition, timing, and variation on a five-point scale, with higher scores indicating better ratings.Theme controllability asks how strongly the piece is influenced by the given theme.Source: A. Listening Test: Setup · p. 8
  • The listening test also rates overall structureness and overall quality.Source: A. Listening Test: Setup · p. 8
  • The 50 recruited subjects come from the authors’ social circles, and 27 identify as male and 23 as female.Subjects self-report familiarity with Mandarin pop music; the authors found no significant relationship between musical experience and ratings.Source: A. Listening Test: Setup · p. 9
  • Only 12 of the 29 test songs are used in the listening test; songs are randomly selected and divided into four packs.Familiar listeners rate generated music, while unfamiliar listeners hear the original pieces in place of the seq2seq model’s output.Source: A. Listening Test: Setup · p. 9

B. Listening Test: Result

The listening tests favor Theme Transformer on theme-related and other reported metrics, while overall-metric gaps remain. Further analyses examine theme variation, theme development, and the limits of the theme-retrieval method.

B. Listening Test: Result

  • Theme Transformer outperforms the prompt-based and basic seq2seq models across all listening-test metrics, especially the first three theme-related metrics, and has similar MOS to original data on those metrics.The reported comparison uses both listener groups.Source: B. Listening Test: Result · p. 9
  • A gap remains between Theme Transformer and the original pieces on the last two overall metrics.The authors identify this as an area for future improvement; the evaluation’s original pieces are token-sequence-converted MIDI files, not the original POP909 MIDI files.Source: B. Listening Test: Result · p. 9

C. Comments on the Quality of the Generated Music

  • Generated theme regions vary with preceding melody: without preceding melody notes they tend to resemble the condition, and otherwise their variations relate to those notes.The authors interpret this pattern as evidence that the model learns melody and accompaniment distinctions in relation to theme variation.Source: C. Comments on the Quality of the… · p. 9
  • The model repeats the theme but does not properly develop it; random variations and short 512-token windows limit learning across theme occurrences and their positions.A 512-token sequence spans about 9 bars, versus an average 12.24-bar gap between themes in original pieces.Source: C. Comments on the Quality of the… · p. 9

A. On the Variance of the Generated Theme Regions

B. On the Quality of Theme Retrieval

  • 0.378 (CL) was the highest average F1 score, but differences were not significant except versus CL w/o Note Dur. (p-value<0.05).Averaged across six songs and three annotators; other averages were 0.220 (CL w/o Note Dur.), 0.336 (CL w/o Pitch Shift), 0.345 (CM), and 0.297 (COSIATEC).Source: B. On the Quality of Theme Retrieval · p. 10
  • CL, CM, and COSIATEC have different strengths, and CL’s adopted melody segmentation can leave theme regions incomplete or omit essential material.The authors note that contrastive-learning-plus-clustering can work with other segmentation methods; CM and COSIATEC sometimes retrieve musically better segments.Source: B. On the Quality of Theme Retrieval · pp. 10–11

X. CONCLUSION

The paper combines contrastive-learning-based theme retrieval with an encoder-decoder Theme Transformer using gated parallel attention and theme-aligned positional encoding. Evaluations report improved repetition with perceptible variations, while future work would extend the system beyond one extracted theme.

X. CONCLUSION

  • Theme Transformer combines contrastive learning with musically inspired augmentation and DBSCAN clustering to retrieve thematic material, then uses gated parallel attention and theme-aligned positional encoding in an encoder-decoder model.Source: §X · p. 11
  • Objective and subjective evaluations show better theme repetition with perceptible variations and improved generated-music quality than decoder-only Music Transformer and simple Seq2seq Transformer.Source: §X · p. 11
  • The current work conditions on one given theme; proposed extensions include multiple themes and user-created themes or themes generated from scratch.Source: §X · p. 11
  • Contrastive-learning similarity curves could characterize departures from and returns to the theme and inform future generation methods.Source: §X · p. 11
Loading 2111.04093v2…