MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Hao-Wen Dong; Wen-Yi Hsiao; Li-Chia Yang; Yi-Hsuan Yang

Source-linked AI summary

MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment

Hao-Wen Dong, Wen-Yi Hsiao, Li-Chia Yang, Yi-Hsuan Yang

arXiv:1709.06298v2Sep 19, 2017eess.AScs.AIcs.LGcs.SDstat.ML

TL;DR

Symbolic music generation must capture temporal structure, interdependent tracks, and polyphonic note groupings without relying on unsuitable note orderings. MuseGAN proposes three GAN-based multi-track models and evaluates them with objective metrics and a user study. The authors report that the models learn aspects of music and can support multi-track generation and accompaniment, while remaining below human musical and aesthetic performance.

Problem
Symbolic music generation requires temporal modeling, coordinated interdependent tracks, and representations that handle polyphonic groupings without imposing chronological note order.
Method
MuseGAN proposes three GAN-based models for multi-track symbolic music, using temporal modeling, track-interaction strategies, bar-level composition, piano-rolls, and objective and subjective evaluation.
Results
The proposed models can learn aspects of music and generate coherent four-bar multi-track piano-roll music from scratch, while also generating accompanying tracks from a human-provided track.
Takeaways & Limitations
MuseGAN provides a framework for multi-track sequence generation that the authors hope can extend beyond music to other multi-track domains.
Takeaways & Limitations
The dataset is noisy and lacks a clear way to identify melody and accompaniment tracks, preventing categorization into melody, rhythm, and drum tracks.

Abstract

from arXiv · show

Generating music has a few notable differences from generating images and videos. First, music is an art of time, necessitating a temporal model. Second, music is usually composed of multiple instruments/tracks with their own temporal dynamics, but collectively they unfold over time interdependently. Lastly, musical notes are often grouped into chords, arpeggios or melodies in polyphonic music, and thereby introducing a chronological ordering of notes is not naturally suitable. In this paper, we propose three models for symbolic multi-track music generation under the framework of generative adversarial networks (GANs). The three models, which differ in the underlying assumptions and accordingly the network architectures, are referred to as the jamming model, the composer model and the hybrid model. We trained the proposed models on a dataset of over one hundred thousand bars of rock music and applied them to generate piano-rolls of five tracks: bass, drums, guitar, piano and strings. A few intra-track and inter-track objective metrics are also proposed to evaluate the generative results, in addition to a subjective user study. We show that our models can generate coherent music of four bars right from scratch (i.e. without human inputs). We also extend our models to human-AI cooperative music generation: given a specific track composed by human, we can generate four additional tracks to accompany it. All code, the dataset and the rendered audio samples are available at https://salu133445.github.io/musegan/ .

Introduction

Symbolic music generation remains challenging because music requires temporal structure, coordinated multiple tracks, and representations suited to polyphonic note groupings. MuseGAN addresses these challenges by targeting multi-track polyphonic music with harmonic, rhythmic, inter-track, and temporal structure.

Motivation: Music requires temporal modeling because higher-level structures such as phrases comprise smaller recurrent patterns such as bars.Listeners attend to coherence, rhythm, tension, and emotional flow over time.
Motivation: Multiple instruments or tracks interact closely while unfolding interdependently over time.This creates a need to model relationships among tracks rather than treating them as isolated sequences.
Motivation: Polyphonic notes form chords, arpeggios, and melodies, making chronological note ordering unsuitable and limiting direct transfer from monophonic generation.Prior methods therefore often simplified the task by using single tracks, ordered notes, or combinations of monophonic melodies.
Research goal: The paper aims to generate multi-track polyphonic music with harmonic and rhythmic structure, inter-track interdependency, and temporal structure.This goal explicitly avoids common simplifications in symbolic music generation.
Approach: MuseGAN uses temporal approaches for scratch generation or track-conditional accompaniment, three strategies for track interaction, and bars as compositional units.The interaction strategies use private generators, one joint generator, or private generators with shared inputs.
Contributions: The work proposes a GAN-based multi-track sequence model, track-conditional music generation, the Lakh Pianoroll Dataset, and intra-track and inter-track evaluation metrics.The authors also report a user study and release code, data, and rendered audio samples.

Generative Adversarial Networks

The paper represents music as multi-track piano-rolls and uses GANs to learn their distribution. It adopts WGAN-GP because the cited work reports more stable training, faster convergence, better optima, and less parameter tuning.

GAN framework: GAN training pits a generator mapping noise to data against a discriminator distinguishing real samples from generated ones.The discriminator receives real-data and generated-data examples during adversarial learning.
Representation: A multi-track piano-roll represents note presence across time for multiple instruments using binary-valued, scoresheet-like matrices.Figure 2 places time horizontally and notes from low to high pitch vertically; black pixels mark played notes.
WGAN-GP: Wasserstein distance and gradient penalty are used in WGAN-GP to address instability and mode collapse associated with the original GAN formulation.The passage contrasts gradient penalty with weight clipping for enforcing the K-Lipschitz constraint.
WGAN-GP: The model uses WGAN-GP because it is reported to converge faster to better optima while requiring less parameter tuning.This is the paper’s stated rationale for selecting WGAN-GP as its generative model.

Proposed Model

MuseGAN represents polyphonic music as multi-track piano-roll sequences and combines alternative GAN architectures with temporal generators to model track interactions and multi-bar coherence. Its hybrid and track-conditional designs preserve inter-track structure while allowing flexible track generation and accompaniment.

Data Representation: Multi-track piano-rolls encode each bar as a set of track-specific note matrices, supporting CNN-based modeling of fixed-size musical data.The representation treats bars as compositional units and stacks track piano-rolls into multi-track tensors.
Jamming Model: The jamming model uses one private generator and discriminator per track, with each generator driven by its own random vector.Each discriminator supplies backpropagated supervisory signals for its corresponding track generator.
Composer Model: The composer model uses one shared generator and discriminator to create and evaluate all tracks collectively from a shared random vector.The generator outputs a multichannel piano-roll whose channels represent individual tracks.
Hybrid Model: The hybrid model gives each track a private generator while sharing an inter-track random vector and one collective discriminator.This design combines coordinated generation with track-specific inputs and architectures.
Hybrid Model: The hybrid model can vary a specific track’s architecture or inputs without losing inter-track interdependency.This flexibility distinguishes it from the composer model’s single shared generator.
Temporal Models: Temporal modeling is required because the static multi-track models generate bars independently and may lack coherence across bars.Two temporal approaches are introduced: generation from scratch and generation conditioned on a human-provided track.
Generation from Scratch: The from-scratch temporal model maps noise to latent vectors carrying temporal information, then generates piano-rolls sequentially bar by bar.Its temporal structure generator feeds a bar generator that grows the phrase across bars.
Track-conditional Generation: The track-conditional temporal model encodes a human-provided track into inter-track features and generates the remaining tracks sequentially.The conditional generator learns the given track’s temporal structure while completing the other tracks.

Implementation

The preprocessing pipeline converts MIDI files into cleaned, segmented five-track piano-roll phrases for modeling, while acknowledging unavoidable dataset noise and track-label ambiguity.

Dataset construction: MIDI files are converted into multi-track piano-rolls with 128 pitches and 96 time steps per bar.The conversion uses pretty_midi and models temporal patterns such as triplets and 16th notes.
Dataset construction: Noisy web-scraped MIDI data are cleansed because user-generated files can impair learning.The authors use LPD-matched data and apply further cleansing steps.
Dataset construction: Tracks are merged into five families—bass, drums, guitar, piano and strings—to reduce sparsity and avoid empty bars.The merging introduces noise but is empirically preferred to retaining empty bars.
Dataset limitations: The dataset cannot reliably categorize tracks as melody, rhythm or drums because no clear melody-versus-accompaniment identification is available.This limits direct use of track categorizations adopted by some prior work.
Dataset construction: The preprocessing selects high-confidence Rock songs in 4/4 time, segments them into four-bar phrases, and produces 50,266 training phrases.Longer segments are pruned to the required phrase size.
Representation: Notes below C1 or above C8 are discarded, yielding a target tensor of 4 × 96 × 84 × 5.The dimensions correspond to bars, time steps, pitches, and tracks.

Objective Metrics for Evaluation

MuseGAN evaluates generated music with intra-track and inter-track statistics that compare structural, rhythmic and harmonic properties against real data.

Intra-track metrics: EB measures the percentage of empty bars, UPC counts used pitch classes per bar, and QN measures the percentage of qualified notes.A qualified note lasts at least three time steps, so QN indicates whether music is overly fragmented.
Inter-track and rhythmic metrics: DP measures the percentage of drum notes occurring in common 8- or 16-beat patterns for Rock songs in 4/4 time.The metric targets rhythmic regularity in drum tracks.
Inter-track and rhythmic metrics: TD measures harmonic distance between track pairs, with larger values indicating weaker inter-track harmonic relations.The metric compares tonal relationships across tracks.
Training-data analysis: Training data show UPC below 2.0 for bass and above 3.0 for guitar, piano and strings, reflecting melody-like and chord-like roles.Bass tends to play the melody, whereas guitar, piano and strings tend to play chords.
Training-data analysis: More than 88 percent of drum notes follow 8- or 16-beat patterns in the training data.TD is around 1.50 between melody-like and chord-like tracks and around 1.00 between two chord-like tracks.
Training-data analysis: Shuffling bars across tracks slightly increases TD, indicating that the metric captures inter-track harmonic relations.The comparison uses randomly paired bars from two specific tracks.

Experiment and Results

The proposed models generate multi-track piano-rolls with recognizable musical structure, while objective metrics reveal differences in track fidelity, harmonic coordination, and remaining fragmentation.

Generated Samples: Generated samples usually share a musical scale and exhibit chord-like intervals, bass melodies, drum patterns, and chordal guitar, piano, and strings.These observations indicate recognizable intra-track patterns and cross-track harmonic relations in the generated phrases.
Objective Evaluation: The jamming model tends to perform best on intra-track metrics, but its outputs have weaker cross-track harmonic relations than those of the composer and hybrid models.The authors attribute the intra-track advantage possibly to each jamming generator focusing on one track, while lower TD favors the composer and hybrid models.
Objective Evaluation: All non-ablated models perform well in DP, while generated music uses more pitch classes and fewer qualified notes than the training data.The authors associate the lower QN with overly fragmented notes, possibly caused by binarizing the generator’s continuous-valued outputs.
Objective Evaluation: Composer and hybrid models perform similarly across track combinations, suggesting that the hybrid model retains performance while offering greater flexibility.The hybrid model’s flexibility allows different architectures or inputs for individual tracks without losing inter-track dependency.
Training Process: During training, generated piano-rolls progress from noise to notes in proper pitch ranges, then acquire longer note durations in guitar, piano, and strings.The generator first learns pitch ranges, then develops lower bass clusters and longer note durations as training proceeds.
Training Process: The generator ultimately learns an appropriate number of pitch classes, but QN remains below the training-data value, indicating room for improvement.The authors use these objective metrics to inspect generated results before conducting subjective evaluation.
User Study: In the listening study, the hybrid model is preferred for scratch generation by both listener groups and for conditional generation by pros.Non-pros prefer the jamming model for conditional generation, while composer and hybrid models score higher on harmonious scratch generation than jamming.

Related Work

Related work spans recurrent, restricted-Boltzmann-machine, and GAN-based approaches for symbolic music, often simplifying polyphonic or multi-track generation.

Recurrent Models: Earlier symbolic-music systems used RNNs for monophonic melodies, four-voice chorales, and other musical formats.RNN-RBM also generated polyphonic piano-rolls, extending recurrent modeling beyond monophonic sequences.
Problem Simplifications: Prior work commonly simplified symbolic music by restricting tracks, imposing note orderings, or combining monophonic melodies.These simplifications made generation more manageable but did not directly address full multi-track polyphonic structure.
GAN-Based Music Generation: Recent GAN-based systems generated polyphonic or monophonic music as ordered note events or discrete token sequences.C-RNN-GAN introduced note ordering with recurrent generator and discriminator networks, while SeqGAN combined GANs with reinforcement learning.

Conclusion

The paper presents a GAN-based deep-CNN model for multi-track piano-roll generation and evaluates learning with objective metrics and a subjective study.

Conclusion: The objective metrics and user study show that the proposed models learn aspects of music, although they remain below human musicians musically and aesthetically.The authors identify desirable properties while leaving further improvement to follow-up research.
Conclusion: The model design is presented as a general approach to multi-track sequence generation beyond music.The conclusion emphasizes deep CNN implementation for multi-track piano-rolls while retaining a broader sequence-generation framing.

Appendix A Samples of the Training Data

The appendix presents randomly selected piano-roll samples from the training data.

Training Data Samples: Figure 10 shows randomly chosen sample piano-rolls from the training dataset.The figure provides examples of the data used for training.

Appendix B Implementation Details

The appendix specifies the generator, discriminator, and encoder architectures, training configuration, and post-processing used to produce rendered audio.

Input configuration: The system uses a fixed total random-vector length of 128, distributed as one vector, two 64-dimensional vectors, or four 32-dimensional vectors by model.The input length of Gbar is also 128.
Network architecture: The temporal generator uses transposed convolutions across inter-bar time, while the bar generator models intra-bar time and pitch successively.Batch normalization precedes each activation layer.
Network architecture: The discriminator contains five one-dimensional convolutional layers and one fully connected layer, using leaky ReLU units with negative slope 0.2.
Network architecture: The encoder reverses the generator, uses skip connections, and fixes each layer at 16 filters to compress inter-track interdependency.
Training: End-to-end training uses Adam with α = 0.001, β1 = 0.5, and β2 = 0.9, updating G and track-conditional E once per five discriminator updates.Each model trains in less than 24 hours on a Tesla K40m GPU.
Audio rendering: Generated piano-rolls are quantized to sixteenth notes, converted to MIDI, mixed, and rendered to stereo audio in an external digital audio workstation.

Appendix C Sample Generated Piano-rolls

The appendix presents randomly selected piano-roll samples from scratch and under track conditioning, with colors identifying the five instrument tracks. The conditional examples use strings to demonstrate flexibility beyond the piano conditioning used in the experiment.

From-scratch generation: Figures 11 and 12 show randomly selected piano-rolls generated from scratch by the composer and hybrid models, respectively.
Architecture reference: Table 4 tabulates the temporal generator, bar generator, discriminator, and encoder architectures, including layer and activation specifications.
Color encoding: The piano-roll colors encode bass as cyan, drums as pink, guitar as yellow, strings as blue, and piano as orange.This color mapping is used for the training examples and generated samples.
Output processing: In Figure 11(b), the composer-model output is binarized at zero because the generator’s final activation uses tanh.
Track-conditional generation: Figure 13 shows composer-model samples conditioned on the strings track.The strings track is used here instead of the piano track used in the Experiment section.