Source-linked AI summary
Compound Word Transformer: Learning to Compose Full-Song Music over Dynamic Directed Hypergraphs
Wen-Yi Hsiao, Jen-Yu Liu, Yin-Cheng Yeh, Yi-Hsuan Yang
TL;DR
Music-generation Transformers must represent heterogeneous token types, but existing models generally treat them alike. This paper introduces type-specific prediction heads and compound-word grouping, interprets the resulting model as a dynamic directed-hypergraph learner, and applies it to full-song Pop piano. The model produces comparable perceived quality with much shorter training and inference time.
Problem
Existing music models mostly treat heterogeneous token types equally, although different token types may have different properties.
Method
The paper combines type-specific feed-forward heads with compound-word grouping that predicts related tokens together, using CP with a linear Transformer backbone and a dynamic directed-hypergraph interpretation.
Results
The model generates full-song expressive Pop piano with comparable perceived quality to a competing Transformer-XL model in much shorter training and inference time.
Takeaways & Limitations
Compound-word modeling provides shorter sequences while retaining separate type-specific token predictions for expressive full-song music generation.
Takeaways & Limitations
The paper does not explore whether the proposed model improves long-term repetitions or “structureness.”
Abstract
from arXiv · showhide
To apply neural sequence models such as the Transformers to music generation tasks, one has to represent a piece of music by a sequence of tokens drawn from a finite set of pre-defined vocabulary. Such a vocabulary usually involves tokens of various types. For example, to describe a musical note, one needs separate tokens to indicate the note's pitch, duration, velocity (dynamics), and placement (onset time) along the time grid. While different types of tokens may possess different properties, existing models usually treat them equally, in the same way as modeling words in natural languages. In this paper, we present a conceptually different approach that explicitly takes into account the type of the tokens, such as note types and metric types. And, we propose a new Transformer decoder architecture that uses different feed-forward heads to model tokens of different types. With an expansion-compression trick, we convert a piece of music to a sequence of compound words by grouping neighboring tokens, greatly reducing the length of the token sequences. We show that the resulting model can be viewed as a learner over dynamic directed hypergraphs. And, we employ it to learn to compose expressive Pop piano music of full-song length (involving up to 10K individual tokens per song), both conditionally and unconditionally. Our experiment shows that, compared to state-of-the-art models, the proposed model converges 5--10 times faster at training (i.e., within a day on a single GPU with 11 GB memory), and with comparable quality in the generated music.
Introduction
The paper addresses heterogeneous music tokens with type-specific Transformer heads and compound-word grouping, reducing sequence length while modeling full-song Pop piano.
- Motivation: Existing music models mostly treat heterogeneous token types equally, despite their differing properties and roles in representing musical aspects.
- Type-specific modeling: The model uses different feed-forward heads for tokens of different types within a Transformer sequence model.
- Compound-word representation: Compound words group consecutive related tokens and predict multiple token types simultaneously at one time step.
- Compound-word representation: CP supports type-specific loss functions, sampling policies, and embedding sizes while preserving separate prediction heads.
- Efficiency: Shorter CP sequences make training and inference faster and may facilitate learning long-range musical dependencies.
- Limitations and future work: The authors identify future study of long-term repetitions, or “structureness,” as an unexplored direction.
- Evaluation: Up to 10K individual tokens per song are modeled in unconditional and conditional expressive Pop piano generation, with comparable quality and faster training and inference than strong baselines.
Related Work
Prior Transformer-based music-generation work commonly uses MIDI-like or REMI-derived representations and varies in attention-window length and vocabulary size. The paper positions its approach as an early full-song Pop music study using a linear Transformer backbone.
- Background: Transformers have increasingly been applied to music generation by representing music pieces as sequences of discrete tokens analogous to text words.
- Representations: Most existing systems use MIDI-like or REMI-derived representations, sometimes adding track- or structure-related tokens.
- Representations: MIDI-like representations encode elapsed time with time-shift tokens, whereas REMI uses bar and position tokens on a metrical grid.
- Model scale: Existing work varies in attention-window length and vocabulary size, while this paper targets full-song Pop modeling involving 10K tokens per song and uses a linear Transformer backbone.
- Prior grouping: The authors note an early preliminary attempt to group tokens by Hawthorne et al. (2018b).
Methodology
The methodology represents music as compound words that group related, type-specific tokens, then models these shorter sequences with a Transformer using specialized prediction heads and embeddings.
- Sequence representation: A conversion function maps a music piece X to a time-ordered token sequence S, which a Transformer generates and an inverse function converts back into music.Different conversion functions produce representations with different sequence lengths T and vocabulary sizes M.
- Compound-word construction: Compound words group neighboring tokens that jointly define a musical event, including note-related and metric-related events.The procedure fills absent types with [ignore] tokens so every time step contains K predictions.
- Compound-word construction: The expansion-compression trick produces a sequence length TCP that is generally shorter than the original REMI length TREMI while retaining K token slots per compound word.The original sequence is expanded to KTCP individual tokens and compressed to TCP compound words, with TCP < TREMI < KTCP.
- Embedding construction: Each compound-word embedding concatenates type-specific token embeddings and a family-token embedding, then projects the result to the Transformer dimension before positional encoding.The embedding lookup tables and projection matrix support this compressive representation.
- Multi-head output module: CP uses different embedding sizes and feed-forward heads for different token types, with an additional head for the family token.A two-stage procedure predicts the family token first and then predicts the remaining tokens conditioned on it; inference uses type-specific temperature-controlled sampling policies.
Graph Interpretation
The paper interprets ordinary token sequences as walks on a fully connected graph and compound-word sequences as transitions over dynamic directed hypergraphs. This framing also suggests that compound words may apply beyond music when multiple tokens represent one event or entity.
- Static graph view: A vocabulary defines a fully connected graph whose nodes are tokens and whose edges connect token pairs, making a token sequence a graph walk.The graph has nodes V = {1, . . . , M} and edges E = V × V.
- Hypergraph construction: CP augments the vocabulary with [ignore] and family tokens, and each compound word contains K + 1 nodes representing its typed tokens and family token.A compound-word sequence transitions from one K + 1-node configuration to another at each time step.
- Hypergraph construction: The compound-word sequence forms a dynamic directed hypergraph in which each time step adds a directed hyperedge connecting the preceding and current compound-word nodes.Boundary hyperedges also include [BOS] and [EOS] tokens and therefore connect fewer nodes.
- Learner interpretation: The CP Transformer acts as a learner over this dynamic hypergraph by predicting the next hyperedge from embeddings updated using prior graph structure or prior positional input embeddings.This corresponds to predicting the next compound-word tokens and family token at each time step.
- Broader applicability: The compound-word idea is generic and may extend to other domains where multiple tokens are needed to represent a single event, entity, or object.
Implementation
The implementation evaluates CP for conditional and unconditional full-song Pop piano generation, using type-specific token representations, compound words, and linear Transformer backbones.
- Tasks: The experiments generate expressive Pop piano performances conditionally from lead sheets and unconditionally from scratch.The full-song sequences contain up to 10K individual tokens per song.
- Data: The dataset contains 1,748 Internet-collected Pop piano pieces averaging about 4 minutes and 108 total hours.Fifty songs are held out for testing.
- Representation: CP represents performances with six token types: pitch, duration, velocity, position/bar, tempo, and chord.Conditional generation adds track tokens; unconditional generation adds an EOS token.
- Comparisons: Sequences in CP are shorter than REMI, while the two representations have similar total vocabulary sizes.The implementation compares CP+linear with Pop Music Transformer-derived REMI+XL and REMI+linear baselines.
- Backbone: The CP+linear model uses a linear Transformer with attention window N equal to sequence length T, avoiding sequence segmentation.REMI+XL uses Transformer-XL with N = 512, while REMI+linear uses a linear Transformer with N = T.
Quantitative Evaluation
The quantitative evaluation measures efficiency and conditional-generation alignment under a resource-constrained setting. CP+linear trains and generates faster than REMI baselines, while producing comparable conditional quality and stronger unconditional subjective results.
- Setup: The experiments target a single-GPU, three-day resource constraint and average efficiency results over 50 held-out conditional test songs.The GPU has 11 GB memory.
- Efficiency: 0.6 days versus nearly 7 days: CP+linear reaches training NLL 0.27 much faster than REMI+XL.Both figures refer to reducing the training-data negative log-likelihood to 0.27.
- Efficiency: <30 seconds: CP+linear completes conditional song generation, while REMI+XL and REMI+linear are about 3x and 1.7x slower.The generated song averages about 4 minutes in the dataset, making CP+linear faster than real time.
- Efficiency: <20 seconds: CP+linear generates an unconditional full-length song, while REMI+XL is on average 7x slower.This comparison averages inference time over 50 songs generated from scratch.
- Objective metrics: The conditional evaluation measures melody and chord matchness between generated piano performances and their lead-sheet conditions.Unconditional objective evaluation remains an open issue in the cited discussion.
Qualitative Evaluation
The qualitative evaluation compares generated music from different models using piano-roll visualizations and human ratings in conditional and unconditional settings. CP+linear shows richer, more diverse unconditional content and performs slightly better across four metrics, while REMI+XL performs best conditionally with only a moderate gap.
- Qualitative comparison: Piano-rolls of the middle 64 bars show richer and more diverse content for CP+linear in unconditional generation.The figure compares random generated pieces from two models.
- Evaluation setup: Human ratings cover fidelity, richness, humanness, correctness, structureness, and overall quality on a five-point Likert scale.The study uses anonymous questionnaire responses and evaluates excerpts for conditional generation and full songs for unconditional generation.
- Human evaluation: CP+linear performs slightly better consistently across the four metrics in the unconditional setting.The comparison is based on average ratings from 18 subjects.
- Human evaluation: REMI+XL performs best in the conditional setting, but the performance gap between models is only moderate.The conditional task provides a lead sheet, while the unconditional task generates full songs from scratch.
Conclusion
The paper presents a type-aware Transformer that processes multiple consecutive tokens together, compresses sequences through integrated embeddings, and applies the model to music modeling. It also identifies generation-length and conditional-learning considerations in comparisons with REMI+XL.
- The proposed Transformer processes multiple consecutive tokens at each time step and customizes input and output modules according to token type.
- Sequence compression integrates token embeddings, which can be interpreted as forming a hyperedge over a dynamic graph.
- The paper reports that the new Transformer works well for modeling music.
- REMI+XL seldom generates [EOS] tokens for long music, so generation is stopped after 7,680 tokens.
- In conditional generation, a supplied melody fairly outlines the song's global structure, making learning from short segments appear sufficient.
Ethics Statement
The paper highlights copyright and broader social concerns associated with automatic music generation. It calls for care in training-data use and in generating music that may imitate existing artists.
- Automatic music generation may infringe copyright laws and raise concerns about the future role of human musicians.
- The paper calls for fair-use consideration when existing musical material is used to train models.
- It also raises concern about computer-generated music deepfaking an existing artist's style.