PaperlayerView exact revision on arXiv

Source-linked AI summary

Music Generation with Deep Learning

Vasanth Kalingeri, Srikanth Grandhe

arXiv:1612.04928v1cs.SD

TL;DR

The paper targets pleasant music generation without musical-structure constraints, treating composed music as learning data. It compares LSTM architectures operating on raw audio frequency representations; the bilinear and 2D-convolutional LSTM models produced the best audio among those tested.

  • Problem

    The target is generating pleasant music without imposing musical-theory constraints or using note, notation, or chord features.

    Source: §1 · p. 1

  • Method

    The paper compares LSTM-based architectures that process raw audio frequency representations, including models with fully connected or convolutional components and no note or chord features.

    Source: §1 · p. 1, §3.1 · p. 2, §4 · p. 6

  • Results

    The bilinear architecture and LSTM with 2D convolutional layers produced the best audio among tested models, while the proposed models outperformed the base model.

    Source: §5 · p. 6, §4 · p. 6

  • Takeaways & Limitations

    The tested models provide insights into architectures suited to raw-audio representations.

    Source: §5 · p. 6

    The design relies on the assumption that only a subset of frequency components is essential for human perception.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

The use of deep learning to solve problems in literary arts has been a recent trend that has gained a lot of attention and automated generation of music has been an active area. This project deals with the generation of music using raw audio files in the frequency domain relying on various LSTM architectures. Fully connected and convolutional layers are used along with LSTM's to capture rich features in the frequency domain and increase the quality of music generated. The work is focused on unconstrained music generation and uses no information about musical structure(notes or chords) to aid learning.The music generated from various architectures are compared using blind fold tests. Using the raw audio to train models is the direction to tapping the enormous amount of mp3 files that exist over the internet without requiring the manual effort to make structured MIDI files. Moreover, not all audio files can be represented with MIDI files making the study of these models an interesting prospect to the future of such models.

1. Introduction

Because music composition is complex, the paper frames generation as a learning problem using composed music as training data. Its goal is pleasant, unconstrained music generated directly from audio waveforms, without notes, notation, or chords.

1. Introduction

  • The authors frame music generation as learning useful musical patterns from composed music because designing an algorithm to compose and perform music is not obvious.They describe music composition as complex and abstract.Source: §1 · p. 1
  • The goal is pleasant audio that need not resemble human performance and is generated without musical-theory constraints or note, notation, or chord features.The approach works directly with audio waveforms.Source: §1 · p. 1
  • Because audio waveforms vary over time with smooth transitions between neighboring fragments, the paper uses LSTM networks to model the signals.The authors motivate recurrent networks by their ability to share parameters across time.Source: §1 · p. 1

2. Related work

Earlier music-generation approaches use structured MIDI features or raw audio; the paper implements a frequency-domain LSTM method as its baseline. It identifies coherence and long-range-dependency problems in that method and changes the architecture to address them.

2. Related work

  • MIDI-based LSTMs predict successive notation vectors and can generate pleasant audio, but their low-dimensional output vocabulary restricts the audio they can produce [3,4,5].Their limited audio vocabulary also eliminates the possibility of noise during generation.Source: §2 · pp. 1–2
  • A deep dilated convolutional network generates raw audio one sample at a time, but takes about a minute to generate one second of audio on Google’s GPU clusters [2].It classifies each sample into one of 255 values and uses increasing dilation to capture long-range dependencies.Source: §2 · p. 2
  • Nayebi et al. [1] use an LSTM to predict the next timestep’s Fourier representation, processing groups of samples and serving as this paper’s baseline.The method trains with mean squared difference between predicted and true frequency representations.Source: §2 · p. 2
  • The baseline’s output has disturbance and lacks coherent structure, with long-range capture constrained by large feature size and hardware-limited cell-state size.The authors change the architecture to mitigate these issues and generate more coherent music.Source: §2 · p. 2

3. Approach

The approach models 16 kHz mono audio in the frequency domain with LSTMs, testing input transformations, multiple-LSTM designs, and convolutional context windows. The baseline predicts frequency vectors from prior timesteps; later variants seek richer representations or longer context.

3. Approach

  • The baseline uses 16 kHz mono audio and represents each Fourier vector’s real and imaginary components as a 2n-dimensional unrolled vector.The paper establishes Nayebi et al. as the baseline for comparison.Source: §3 · p. 2, §3.1 · p. 2

3.1. Baseline implementation

  • The baseline feeds a D-dimensional frequency vector to an LSTM, trains across T timesteps with mean squared error, and predicts autoregressively from an initial sequence.During prediction, each output is appended to the sequence and the last T steps are used for the next prediction.Source: §3.1 · pp. 2–3

3.2. Feed complex representations as input to the LSTM

  • A fully connected layer transforms each timestep’s input before the LSTM, while a 1D convolution plus max pooling shortened training but produced lower-quality music.The fully connected layer used ReLUs; the convolution used 12 filters of size 1×3 and 2×2 max pooling.Source: §3.2 · pp. 3–4

3.3. Using multiple LSTMs

  • The multilayer design passes the first LSTM’s output to a second LSTM for prediction, while the bilinear design merges two LSTM outputs by summation before a fully connected prediction layer.The bilinear LSTMs start from random initializations, and summation allows gradients to flow easily during backpropagation.Source: §3.3 · pp. 4–5

3.4. Adding convolutions to increase range

  • Feeding K groups of frequency samples per timestep lets a CNN compress 0.75 seconds of information for the LSTM; generated music was pleasant, but bilinear LSTMs were better in most cases.The groups are represented as a two-channel K×N image of real and imaginary values.Source: §3.4 · p. 5

4. Experiments

The experiments trained frequency-domain LSTM variants on a small piano-tune dataset and compared generated clips through blind ratings. Proposed models outperformed the baseline, with the bilinear model producing the best samples.

  • The dataset comprised 25 piano tunes; 20 were used for training and five as random prediction seeds.Tunes lasted two to three minutes and were converted to mono 16 kHz audio before frequency-domain processing.Source: §4 · p. 5
  • Models used 40 timesteps per iteration, corresponding to 10 seconds of audio, and songs were segmented into 10-second training samples.Source: §4 · p. 5
  • Thirty generated clips were rated from 1 to 5 by 10 testers, while training-loss plots were used for hyperparameter selection rather than judging music quality.Clips were generated at selected epochs and manually pruned before testing.Source: §4 · p. 6
  • All proposed models outperformed the baseline, and the bilinear model produced the best output samples in blind testing.Table 1 reports ratings from 10 testers on five generated samples picked from each model.Source: §4 · p. 6, Table 1 · p. 6
  • Multilayer LSTMs and single-LSTM models with fully connected or convolution layers all outperformed the base model, which the authors attributed to their greater parameter counts.The authors relate the improvement to learning from limited training data.Source: §4 · p. 6

5. Conclusion

The tested architectures improved music quality over the base model, with bilinear and 2D-convolution LSTMs producing the best audio among those tested. The authors propose exploring adversarial losses that better represent human music perception.

  • Bilinear and LSTM models with 2D convolutional layers produced the best audio among the tested models.The authors also report improved music quality over the base model.Source: §5 · p. 6
  • Future work will investigate replacing mean squared error with an adversarial network that models a loss function intended to better represent human music perception.Source: §5 · p. 6
Loading 1612.04928v1…