PaperlayerView exact revision on arXiv

Source-linked AI summary

Conditional LSTM-GAN for Melody Generation from Lyrics

Yi Yu, Abhishek Srivastava, Simon Canales

arXiv:1908.05551v2cs.AIcs.SDeess.AS

TL;DR

Lyrics-to-melody generation is constrained by limited paired lyrics-melody data with alignment information. The paper builds a 12,197-song paired MIDI dataset and proposes a lyrics-conditioned LSTM-GAN that jointly models melody and syllable-note alignment. Experiments report plausible, tuneful melodies and closer alignment and metric results to human compositions than baselines.

  • Problem

    Limited paired lyrics-melody datasets with alignment information hinder learning the relationship between lyrics and accompanying melody.

  • Method

    The paper combines a lyrics-conditioned LSTM generator and discriminator with lyric embeddings, noise inputs, and quantization for discrete MIDI sequences.

  • Results

    The model generates plausible and tuneful sequences, outperforms baselines across pitch-related attributes, and produces melodies closer to human compositions on each reported metric.

  • Takeaways & Limitations

    A large paired dataset and conditional LSTM-GAN provide a framework for modeling lyrics-conditioned melody and syllable-note alignment.

Abstract

from arXiv · show

Melody generation from lyrics has been a challenging research issue in the field of artificial intelligence and music, which enables to learn and discover latent relationship between interesting lyrics and accompanying melody. Unfortunately, the limited availability of paired lyrics-melody dataset with alignment information has hindered the research progress. To address this problem, we create a large dataset consisting of 12,197 MIDI songs each with paired lyrics and melody alignment through leveraging different music sources where alignment relationship between syllables and music attributes is extracted. Most importantly, we propose a novel deep generative model, conditional Long Short-Term Memory - Generative Adversarial Network (LSTM-GAN) for melody generation from lyrics, which contains a deep LSTM generator and a deep LSTM discriminator both conditioned on lyrics. In particular, lyrics-conditioned melody and alignment relationship between syllables of given lyrics and notes of predicted melody are generated simultaneously. Experimental results have proved the effectiveness of our proposed lyrics-to-melody generative model, where plausible and tuneful sequences can be inferred from lyrics.

I. INTRODUCTION

Lyrics-conditioned melody generation seeks to map lyrics to musical sequences, but prior approaches and limited paired data leave distributional fidelity and alignment challenging. The paper addresses these issues with a conditional LSTM-GAN, a large paired dataset, lyric embeddings, and quantization for discrete music generation.

  • I. INTRODUCTION: A conditional LSTM-GAN generates lyrics-conditioned melodies while using a discriminator to align generated and real melody distributions.The model uses deep LSTM generator and discriminator networks conditioned on lyrics.
  • I. INTRODUCTION: A syllable- and word-level LSTM learns joint lyric embeddings that capture syntactic and semantic information.The representation is intended to provide lyric structure for melody generation.
  • I. INTRODUCTION: A quantizer enables the conditional LSTM-GAN to generate discrete-valued music sequences.This design addresses the discrete nature of MIDI-derived music data.
  • I. INTRODUCTION: A paired dataset of 12,197 MIDI songs supports learning and evaluating alignment between lyrics and melody.The dataset is designed to demonstrate improved pleasantness and harmony relative to baseline methods.
  • II. RELATED WORKS: Earlier lyrics-conditioned systems use dynamic programming, n-gram models, random forests, or recurrent networks, while GANs offer distribution-oriented generation.Prior work includes rhythm and harmony constraints, pitch and rhythm prediction, and stylistic melody generation.

A. Melody

The paper represents melody as temporally ordered notes with pitch and duration, aligned to lyrics syllables through MIDI timestamps. Its end-to-end conditional LSTM-GAN predicts aligned MIDI attributes from lyrics.

  • A. Melody: Melody is a temporal sequence of notes whose attributes include pitch and duration.
  • A. Melody: MIDI timestamps synchronize lyrics and melodies, allowing the paper to extract their alignment.
  • A. Melody: The training representation uses one-syllable-to-one-note pairs, with each syllable associated with note, duration, and rest attributes.
  • A. Melody: The proposed system is an end-to-end lyrics-conditioned generative model that uses a generator and discriminator to learn sequential lyrics–melody alignment.
  • A. Melody: The model takes syllable sequences as input and predicts aligned melody sequences containing MIDI numbers, note duration, and rest duration.

B. Lyrics embedding

Lyrics are encoded at syllable and word levels to capture linguistic information before being used by the melody-generation model. The paper combines these representations with LSTM-based sequence modeling.

  • B. Lyrics embedding: The method trains skip-gram models on words and syllables to obtain lyrics representations that encode contextual linguistic regularities.
  • B. Lyrics embedding: Lyrics information is encoded at both syllable and word levels, with separate embedding models associating vectors with English syllables.
  • B. Lyrics embedding: Word-level and syllable-level embeddings are concatenated into a 20-dimensional representation for each syllable.
  • B. Lyrics embedding: LSTM networks provide internal memory and learn longer dependencies through input, forget, and output gates.
  • B. Lyrics embedding: The LSTM cell computes its current output from gated states using element-wise vector multiplication.

2) GAN:

GAN training pairs a generator that produces samples with a discriminator that distinguishes generated data from real training data. This adversarial objective encourages generated samples to match the training distribution.

  • 2) GAN:: A GAN jointly trains a generator and discriminator with conflicting objectives to model the training-data distribution.
  • 2) GAN:: The generator maps a random noise vector to an output sample while attempting to increase the discriminator’s error rate.
  • 2) GAN:: The discriminator distinguishes real samples from generated samples, forming the opposing side of the adversarial training objective.

3) Lyrics-conditional GAN:

The lyrics-conditional GAN generates melody attributes from syllable embeddings and noise, while an LSTM discriminator evaluates generated sequences against real melodies. Generated continuous attributes are quantized and tuned to discrete musical representations during validation and testing.

  • Lyrics conditioning: The generator and discriminator are both conditioned on lyrics encoded as sequences of 20-dimensional embedding vectors.This conditioning supplies textual context to both adversarial components.
  • Generator network: The generator maps syllable embeddings concatenated with random noise to 20-note melody sequences, using LSTM cells to learn lyrics–melody alignment.Noise supports diverse melodies while lyrics condition the generation process.
  • Discriminator network: The discriminator receives generated music attributes concatenated with syllable embeddings and outputs a real-or-fake decision through LSTM layers and sigmoid activation.Its 23-dimensional inputs combine three music attributes with a 20-dimensional syllable embedding.
  • Musical representation: Generated MIDI number, note duration, and rest duration values are constrained to their closest discrete musical values during validation and testing.The tuning procedure also checks whether generated notes fit the most likely standard scale and maps out-of-tune notes to nearby in-tune values.

V. LYRICS-MELODY DATA ACQUIREMENT

The study constructs a paired lyrics–melody dataset because no publicly available aligned dataset supported this task. MIDI files are parsed into aligned syllable and musical-attribute sequences for model training.

  • Data sources: The dataset combines 7,998 LMD-full MIDI files and 4,199 reddit MIDI files to create paired lyrics–melody data with sequential alignment.Together, these sources provide the reported 12,197-song collection.
  • Lyrics processing: The lyrics vocabulary used for skip-gram embeddings contains 20,934 unique syllables and 20,268 unique words from the two MIDI sources.Only paired lyrics–melody sequences from the LMD-full dataset are used to train the LSTM-GAN.
  • MIDI parsing: MIDI files are parsed into triplets containing MIDI number, note duration, and rest duration for notes aligned with English syllables.The parsing extracts pitch and timing information from note events and their syllable correspondences.
  • MIDI parsing: Note and rest durations are computed from note-on and note-off times, then constrained to the closest values in the representation sets.The operator φ maps each continuous duration-related value to the nearest supported discrete value.

3) Note duration:

The paper represents melody timing with note and rest durations and evaluates generated melodies against dataset distributions using MMD. The dataset’s attribute distributions emphasize common pitch, duration, and rest patterns.

  • Note duration: Note duration denotes how long a note sounds, while rest duration denotes the length of silence between notes.Only durations represented in the dataset’s duration table are considered when generated melodies are discretized.
  • Validation using MMD: Most dataset MIDI note numbers range from 60 to 80, quarter notes occur most frequently, and rest = 0 appears in most melodies.These distributions are presented as the dataset’s music-attribute statistics in Fig. 5.
  • Validation using MMD: The dataset is divided into training, validation, and testing sets in a 0.8/0.1/0.1 proportion before 400 training epochs.Generated attributes are discretized and checked against the most likely scale during validation and testing.
  • Validation using MMD: MMD measures distributional similarity between generated and reference sample sets, with a value of 0 indicating samples from the same distribution.The estimator is unbiased and uses an RKHS kernel with a bandwidth selected from cross-dataset distances.
  • Validation using MMD: The validation selects the model checkpoint with the lowest MMD between generated sequences and validation sequences.MMD is computed after each training epoch using melody sequences treated as sets of notes.

C. Comparison methods

The study compares the conditional LSTM-GAN with Random and MLE baselines using music-attribute, scale-consistency, transition, BLEU, and listening-related evaluations. Overall, the proposed model better matches pitch-related and human-composed melody characteristics, while baselines remain closer on temporal averages.

  • Comparison methods: The evaluation compares Random melodies sampled from dataset attribute distributions with a single-layer LSTM trained by maximum likelihood.The Random baseline generates 20-note melodies, while the MLE baseline uses mini-batch gradient descent over a log-likelihood objective.
  • Music quantitative evaluation: The proposed model outperforms the baselines on every pitch-related aspect and most aspects relative to MLE, generally approaching dataset averages.Its 2-MIDI-number and 3-MIDI-number repetition values remain below the corresponding dataset measurements but exceed the MLE baseline.
  • Music quantitative evaluation: For note-duration and rest-duration metrics, the baselines are closer to ground truth because they sample these attributes from the ground-truth distribution.
  • Music quantitative evaluation: 48.6% mean scale-consistency accuracy is achieved by the conditional LSTM-GAN, compared with 47.3% for MLE and 46.6% for Random.The authors attribute the modest accuracies partly to multiple possible scales for the same lyrics and overlapping notes between scales.
  • Music quantitative evaluation: The proposed model approximates the transition distribution of human-composed music, although the testing set has slightly more transitions to lower-pitched notes.
  • Music quantitative evaluation: The conditional LSTM-GAN achieves the highest BLEU scores among the proposed model, MLE baseline, and Random baseline.The paper interprets this as generating melodies of relatively higher quality.

1) Distribution of MIDI numbers:

The distribution experiments test whether lyrics conditioning affects generated MIDI numbers and duration-related sequences. Results indicate that lyrics conditioning improves pitch-distribution consistency and captures correlations between syllables and temporal attributes.

  • Distribution of MIDI numbers:: The lyrics-conditioned MIDI-number distribution is more consistent with ground-truth MIDI numbers than the unconditioned model’s narrower-band distribution.
  • Distribution of MIDI numbers:: Different lyrics produce different MIDI-number distributions, with one example yielding lower MIDI numbers and potentially conveying semantic information.For each of two lyrics, 1,000 songs are generated before estimating the MIDI-number distribution.
  • Distribution of MIDI numbers:: The observed distance between real and generated duration sequences is statistically lower than randomized alternatives, indicating learned correlation with syllable embeddings.The comparison uses note-duration or rest-duration matrices and 10,000 samples for each randomized variable.

G. Subjective evaluation

Subjective evaluation compares generated melodies with Random, MLE, and human-composed examples using listener ratings for overall melody, rhythm, and lyric fit. The proposed model is judged closer to human compositions, but remains below them and is affected by synthesizer limitations.

  • G. Subjective evaluation: Listeners rated the proposed melodies closer to human-composed melodies than either baseline across all three evaluation metrics.The metrics assess the entire melody, rhythm, and how well the melody fits the lyrics.
  • G. Subjective evaluation: The subjective results support the model’s ability to predict plausible sequential alignment between syllables and notes.
  • G. Subjective evaluation: The evaluation still finds gaps between generated and human melodies, leaving substantial room to improve neural melody generation.
  • G. Subjective evaluation: Low melody-evaluation scores may partly reflect the synthesizer’s limited capability for high pitches.
Loading 1908.05551v2…