Source-linked AI summary
MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
Li-Chia Yang, Szu-Yu Chou, Yi-Hsuan Yang
TL;DR
Most neural music-generation systems use RNNs, motivating a CNN-based approach for symbolic melody generation. MidiNet generates melodies bar by bar with a GAN and a conditional mechanism for prior musical information. In user studies, it was comparable to MelodyRNN in realism and pleasantness, while its melodies were reported as more interesting.
Problem
Most neural music-generation models use RNNs, while relatively few use CNNs for music generation, despite CNNs' demonstrated potential in audio-domain music generation.
Method
MidiNet combines a convolutional GAN that generates bar-level symbolic melodies with a conditioner CNN that incorporates prior musical information.
Results
MidiNet was comparable to MelodyRNN in pleasantness and realism, while its melodies were reported as much more interesting.
Takeaways & Limitations
MidiNet provides a flexible alternative to RNN-based designs for conditional symbolic music generation.
Takeaways & Limitations
GAN training remains subject to instability and mode collapse, requiring feature matching and one-sided label smoothing.
Abstract
from arXiv · showhide
Most existing neural network models for music generation use recurrent neural networks. However, the recent WaveNet model proposed by DeepMind shows that convolutional neural networks (CNNs) can also generate realistic musical waveforms in the audio domain. Following this light, we investigate using CNNs for generating melody (a series of MIDI notes) one bar after another in the symbolic domain. In addition to the generator, we use a discriminator to learn the distributions of melodies, making it a generative adversarial network (GAN). Moreover, we propose a novel conditional mechanism to exploit available prior knowledge, so that the model can generate melodies either from scratch, by following a chord sequence, or by conditioning on the melody of previous bars (e.g. a priming melody), among other possibilities. The resulting model, named MidiNet, can be expanded to generate music with multiple MIDI channels (i.e. tracks). We conduct a user study to compare the melody of eight-bar long generated by MidiNet and by Google's MelodyRNN models, each time using the same priming melody. Result shows that MidiNet performs comparably with MelodyRNN models in being realistic and pleasant to listen to, yet MidiNet's melodies are reported to be much more interesting.
1. INTRODUCTION
MidiNet addresses the limited use of CNNs in music generation by producing symbolic melodies bar by bar with a GAN and conditional mechanism. It supports generation from noise, chord progressions, or prior melodies, and is evaluated against MelodyRNN.
- Motivation: Most existing neural music-generation models use RNNs, while relatively few attempts use deep CNNs.WaveNet demonstrated CNN-based generation of realistic musical waveforms and motivated this symbolic-domain investigation.
- Approach: MidiNet generates symbolic melodies one bar after another using a 2-D note-by-time representation.This representation allows convolutions over the notes and time steps within each bar.
- Approach: The model combines a generator CNN, discriminator CNN, and conditioner CNN to generate melodies while incorporating temporal information from previous bars.The GAN transforms random noise into score-like matrices, while the conditioner incorporates previous-bar information into generator layers.
- Conditional generation: Random noise enables generation from scratch and produces different melodies from the same priming melody.The conditioner also supports represented prior knowledge such as chord progressions and starting notes.
- Flexibility: The architecture can support different conditions, control their influence, and extend to multiple MIDI tracks or parts.The authors describe this flexibility as an alternative to RNN-based designs.
- Evaluation: The evaluation compares eight-bar MidiNet and MelodyRNN melodies generated from identical priming melodies, with additional chord-conditioned and creative settings.Source code and pretrained models were released online for reproducibility.
2. RELATED WORK
Related work is dominated by RNN-based music-generation systems, while CNN and GAN approaches are less common. Existing models vary in domain, musical structure, conditioning, and the number of tracks they generate.
- Landscape: Recent neural music-generation systems include models for melodies, audio waveforms, accompaniment, and human-machine duets.These systems may follow priming notes or generate multiple musical parts.
- RNN-based models: MelodyRNN provides three prominent RNN-based symbolic-generation models, including lookback and attention variants for longer-term structures.Their source code and pretrained models are publicly available.
- RNN-based models: Song from PI uses hierarchical recurrent layers to generate melody, drums, and chords for multi-track pop songs.It requires prior musical-scale knowledge and melody profiles that MidiNet does not require.
- RNN-based models: DeepBach uses an RNN architecture for polyphonic four-part chorales and supports user-defined musical constraints.Its target style is composition in the manner of J. S. Bach.
- GAN and CNN models: C-RNN-GAN generates diverse melodies from random noise but lacks conditioning on priming melodies or chord sequences.The paper identifies it as the only prior GAN-based music-generation model known to the authors.
- GAN and CNN models: WaveNet is a CNN-based audio-domain model, whereas MidiNet focuses on symbolic-domain generation.WaveNet generates raw speech and music waveforms rather than MIDI note representations.
3. METHODS
MidiNet represents music as bar-level note matrices and uses a convolutional GAN with a conditioner CNN to generate melodies while incorporating prior musical context. The architecture also provides controls over conditional influence and output diversity.
- Symbolic Representation for Convolution: MidiNet divides MIDI into bars and represents each bar as an h-by-w matrix of note activity across pitches and time steps.Binary matrices omit velocity; multiple matrices can represent multi-track music.
- Generator CNN and Discriminator CNN: The generator transforms random noise into a note matrix, while the discriminator distinguishes authentic MIDI representations from generated ones.The generator uses fully connected and transposed convolution layers to upsample the noise-derived representation.
- Generator CNN and Discriminator CNN: GAN training uses feature matching and one-sided label smoothing to address instability and mode collapse.Feature matching adds L2 regularizers that encourage real and generated feature distributions to be close.
- Conditioner CNN: A conditioner CNN processes previous-bar matrices and supplies compatible intermediate features to the generator’s transposed convolution layers.The conditioner mirrors the generator’s filter shapes and is trained jointly with it through shared gradients.
- Conditioner CNN: MidiNet controls creativity and discipline by changing where conditions enter the generator and by adjusting feature-matching weights.Stronger feature matching makes generated music sound closer to examples in the training set.
4. IMPLEMENTATION
The implementation builds MidiNet from filtered, bar-segmented MIDI tabs, fixed-resolution melody and chord encodings, and several conditioning variants. The experiment generates eight-bar melodies from priming material and compares user judgments across models and settings.
- 4.1 Dataset: The dataset began with 1,022 two-channel TheoryTab MIDI tabs containing separate melody and chord channels.The channels enabled versions trained on melody alone or on melody paired with chords.
- 4.1 Dataset: Melodies use sixteenth-note resolution with w = 16, pitches restricted to C4–B5, and velocity omitted.Triplets, 32nd notes, and other shorter-note exceptions were excluded.
- 4.1 Dataset: Chords are encoded as 13-dimensional vectors and constrained to one chord per bar.Twelve dimensions mark the key and one marks major versus minor chord type.
- 4.1 Dataset: After filtering, segmentation, preprocessing, and key augmentation, training used 50,496 melody-and-chord bars from 526 MIDI tabs.The retained data comprised 4,208 original bars before circular transposition across 12 keys.
- 4.2.1 Model 1: Melody generator, no chord condition: The priming-melody model generates eight bars autoregressively, using generated bars as conditions after the initial real primer.Training samples two consecutive bars and conditions the second on the first.
- 4.2.2 Model 2: Melody generator with chord condition, stable mode: The stable chord-conditioned variant uses the present bar’s chord vector to generate a melody that follows the chord progression.Its architecture emphasizes chord conditioning and is designed to reduce notes violating chord constraints.
5. EXPERIMENTAL RESULT
A 21-participant user study compared MidiNet with three MelodyRNN models on eight-bar melodies generated from shared priming melodies. MidiNet matched MelodyRNN on pleasantness and realism, while its model 1 was rated more interesting and model 2 received the highest pleasantness and realism ratings.
- Study design: The study used 21 participants to rate automatically generated eight-bar melodies from 100 randomly selected priming melodies.Participants included people with and without musical backgrounds, and results were collected using five-point ratings for pleasantness, realism, and interestingness.
- User-study results: MidiNet model 1 received similar pleasantness and realism ratings to the MelodyRNN models.Model 1 used only the previous-bar condition, despite MelodyRNN models being able to exploit all previous bars during generation.
- User-study results: MidiNet model 1 was rated more interesting than the MelodyRNN models, with mean interestingness around 4 for musically trained participants and 3.4 for others.Professional participants’ ratings were mostly above 3.
- User-study results: MidiNet model 2 obtained the highest mean pleasantness and realism ratings for both participant groups.It also exceeded the three MelodyRNN models in interestingness, but remained below MidiNet model 1, especially among professionals.
- Interpretation: Professionals reported that MelodyRNN melodies could sound too repetitive or “safe,” whereas occasional music-theory violations in MidiNet could contribute to interestingness.The authors identify balancing realism and interestingness as a possible direction for further tuning.
- Qualitative examples: Figure 3 illustrates distinct conditioning effects: previous-bar conditioning repeats opening notes, chord conditioning emphasizes harmonic context, and balanced conditioning strengthens adjacent-bar connections.These examples correspond to MidiNet models 1, 2, and 3, respectively.
6. CONCLUSION
The paper presents MidiNet as a CNN-GAN model for MIDI generation with conditional control and a flexible architecture. It concludes that the model can serve as a powerful alternative to RNNs, while future work targets richer multi-track and musically informed generation.
- Conclusion: MidiNet combines a CNN-GAN architecture with a conditional mechanism for exploiting versatile prior musical knowledge.The architecture supports different music types depending on the inputs and specifications.
- Conclusion: The evaluation indicates that MidiNet can be a powerful alternative to RNNs for MIDI generation.
- Future work: Future work proposes multi-track generation, richer MIDI features such as velocity and pauses, larger training data, music-theory principles, and genre or emotion inputs.