Source-linked AI summary
C-RNN-GAN: Continuous recurrent neural networks with adversarial training
Olof Mogren
TL;DR
The paper addresses whether adversarial training can learn distributions over sequential continuous data, a setting it evaluates through classical music. It proposes C-RNN-GAN, a recurrent generator–discriminator system using continuous musical events, and reports promising generated-music results, while noting that the output does not yet match training music by human judgment.
Problem
The paper investigates whether adversarial training can learn generating distributions for sequential continuous data, evaluated using classical music.
Method
C-RNN-GAN uses adversarially trained recurrent generator and discriminator networks to model and generate continuous music sequences.
Results
Adversarial training produces music with greater variability, larger tone and intensity spans, and scores more resembling real music than the baseline.
Takeaways & Limitations
The results support adversarial training as a viable approach for modeling distributions over sequences of continuous data.
Takeaways & Limitations
The generated music does not yet compare with the training music by human judgment, and more experimentation is needed.
Abstract
from arXiv · showhide
Generative adversarial networks have been proposed as a way of efficiently training deep generative neural networks. We propose a generative adversarial model that works on continuous sequential data, and apply it by training it on a collection of classical music. We conclude that it generates music that sounds better and better as the model is trained, report statistics on generated music, and let the reader judge the quality by downloading the generated songs.
1 Introduction
The paper investigates adversarial training for sequential continuous data and demonstrates C-RNN-GAN on classical MIDI music. It represents musical events continuously and evaluates generated music with distributional metrics.
- The study tests whether adversarial training can model sequential continuous data using classical music in freely available MIDI files.
- Unlike earlier RNN music models using symbolic representations, this approach uses continuous tone lengths, frequencies, intensities, and timing.
- C-RNN-GAN is a recurrent architecture trained adversarially to model the joint probability of sequences and generate new data.
- The system is evaluated with metrics including scale consistency and tone range on classical music MIDI sequences.
- The authors conclude that adversarial training is viable for learning distributions over sequences of continuous data.
2 C-RNN-GAN: A continuous recurrent network with adversarial training
C-RNN-GAN combines recurrent generator and discriminator networks in an adversarial zero-sum game for continuous music sequences. Its music representation uses real-valued event features and supports polyphonic outputs.
- The model uses two deep recurrent networks: a generator producing data and a discriminator distinguishing generated sequences from real data.
- Generator cells receive a random vector concatenated with the previous cell’s output.
- The discriminator is bidirectional, using an LSTM recurrent network to incorporate context from both sequence directions.
- Each music event contains real-valued tone length, frequency, intensity, and elapsed time, while C-RNN-GAN-3 can output up to three tones per cell.
3 Experimental setup
The study trains and evaluates C-RNN-GAN and related recurrent models on continuous classical-music MIDI data, using adversarial-training variants and statistics of generated output.
- Model architecture: Both generator and discriminator use two-layer LSTMs with 350 hidden units per cell; the discriminator is bidirectional and the generator unidirectional.The discriminator averages sigmoid outputs across time steps for its sequence-level decision.
- Models: The baseline recurrent network predicts the next tone event at each recurrence, unlike the adversarially trained generator.
- Dataset: The dataset contains 3697 MIDI files from 160 classical-music composers, with each note represented by frequency, duration, intensity, and timing.Data were normalized to a tick resolution of 384 per quarter note.
- Training: Freezing stops updates to whichever adversary becomes too strong, using a 70% training-loss threshold to preserve a usable generator gradient.
- Training: Feature matching trains the generator to match discriminator representations of real data, encouraging variance and reducing overfitting to the current discriminator.The representation is taken from the discriminator layer immediately before logistic classification.
- Evaluation: Evaluation measures polyphony, scale consistency, repetitions, tone span, and generated-music statistics across C-RNN-GAN, the baseline, and the three-tone variant.Polyphony counts tones whose start times are exactly simultaneous; tone span measures half-tone steps between the lowest and highest tones.
4 Results
Adversarial training produces generated music with greater variability and spans, while feature matching is judged to offer the best reported balance between structure and surprise.
- Quantitative evaluation: Adversarial training helps generate patterns with more variability, larger tone span, and larger intensity span.
- Quantitative evaluation: Allowing each LSTM cell to output more than one tone increases the generated music’s polyphony score.
- Listening impressions: Author and co-worker impressions favor feature matching for balancing musical structure and surprise over the other variants.The authors explicitly note that they had not yet conducted a thorough listening study.
- Listening impressions: Generated music files are available for download so readers can evaluate the samples directly.
5 Discussion and conclusions
The paper presents C-RNN-GAN as a promising approach for continuous sequential data, while acknowledging that its generated music remains below the training data by human judgment.
- Discussion and conclusions: C-RNN-GAN generates music with greater variation in the number of tones used and the span of tone intensities than the baseline.
- Discussion and conclusions: Generated music scores resemble real-music scores more closely with C-RNN-GAN than with the baseline.
- Discussion and conclusions: The generated music does not yet compare with the training music by human judgment, and the reasons remain unexplored.
- Discussion and conclusions: Allowing each LSTM cell to output up to three tones produces much higher polyphony scores than the standard C-RNN-GAN setting.The evaluation defines polyphony using tones beginning at exactly the same time.