Source-linked AI summary
Song From PI: A Musically Plausible Network for Pop Music Generation
Hang Chu, Raquel Urtasun, Sanja Fidler
TL;DR
Pop generation must combine melody with chords and other instruments while balancing randomness and compositional rules. The paper proposes a music-theory-informed hierarchical RNN and reports strong human preference for its songs over a recent Google method.
Problem
Pop songs combine melody with chords and other instruments, making generation broader than producing a single note sequence.
Source: §1 · p. 1Method
A scale-conditioned hierarchical RNN generates melody in lower layers and drums and chords in higher layers.
Source: §1 · p. 1Results
Human studies found strong preference for the generated songs over a recent method by Google.
Source: §1 · p. 2Takeaways & Limitations
The framework is presented in applications for neural dancing and karaoke, and neural story singing.
Source: §6 · p. 7, §7 · p. 8Scale conditioning helps the model learn musical regularities but does not constrain generated notes to the scale’s note subset.
Source: §4.1 · p. 3
Abstract
from arXiv · showhide
We present a novel framework for generating pop music. Our model is a hierarchical Recurrent Neural Network, where the layers and the structure of the hierarchy encode our prior knowledge about how pop music is composed. In particular, the bottom layers generate the melody, while the higher levels produce the drums and chords. We conduct several human studies that show strong preference of our generated music over that produced by the recent method by Google. We additionally show two applications of our framework: neural dancing and karaoke, as well as neural story singing.
1 INTRODUCTION
The paper targets pop songs that combine melody, chords, and other instruments, drawing on the idea that compositional rules can turn a random signal into pleasing music. It proposes a hierarchical RNN for these parts and reports human preference over a recent Google method.
1 INTRODUCTION
- Earlier neural music-generation approaches typically produced melody and chords, often as a single track with one note per time step.Boulanger-lewandowski et al. (2012) was an exception, generating simultaneous independent melodies.Source: §1 · p. 1
- The paper draws on Song from π, where a sequence of digits supplies a random base signal and musical rules such as scale and harmony shape it into music.The introduction identifies those rules as key to converting randomness into music.Source: §1 · p. 1
- A hierarchical RNN generates melody in its bottom layers and drums and chords in higher layers, with the model conditioned on scale type.The authors propose that drums and chords can compensate for the melody to produce pleasing music.Source: §1 · p. 1
- Human studies found that participants strongly preferred the generated songs over a recent Google method; the framework also supports neural dancing, karaoke, and story singing.The model was trained on 100 hours of MIDI containing user-composed pop songs and video game music.Source: §1 · p. 2
2 RELATED WORK
Prior music-generation work used both hand-crafted music-theory rules and neural networks, but the neural approaches often generated limited musical representations. The paper distinguishes its approach by combining music-theory-informed hierarchy with multi-track pop generation.
2 RELATED WORK
- Music generation drew interest from machine-learning researchers, music professionals, and enthusiasts, with real-time generation also explored for gaming.The passage cites Eck & Schmidhuber (2002), Boulanger-lewandowski et al. (2012), Chan et al. (2006), and Engels et al. (2015).Source: §2 · p. 2
- Earlier approaches encoded music theory through rules for stitching segments, while neural music generation has been explored since the 1980s.Chan et al. (2006) is given as a rule-based example; neural examples include Bharucha & Todd (1989), Mozer (1996), Chen & Miikkulainen (2001), and Eck & Schmidhuber (2002).Source: §2 · p. 2
- Neural systems varied in representation and scope: Mozer (1996) generated pitch, duration, and chord, while Eck & Schmidhuber (2002) used LSTMs for melody and chord.The passage reports that the LSTM captured more global song structure than Mozer (1996).Source: §2 · p. 2
- The paper’s stated novelty is a music-theory-informed hierarchical model that generates multi-track pop music—melody, chord, and drum—along with two applications.Earlier work includes Kang et al. (2012), Boulanger-lewandowski et al. (2012), and Huang & Wu (2016), with differing capabilities and comparison limits.Source: §2 · p. 2
3 CONCEPTS FROM MUSIC THEORY
This section introduces musical units and the theory concepts used to describe scales, chords, and progressions. Scales select note subsets, while the Circle of Fifths provides a way to favor nearby chord changes.
3 CONCEPTS FROM MUSIC THEORY
- A note is music’s basic unit, and a bar is a time segment spanning a specified number of beats, marked by vertical bar lines.The note system described uses a cycle of 12 tones.Source: §3 · p. 2
- A scale is a subset of notes defined by relative intervals from a starting note; the described system has 48 unique scales across four types and 12 starting notes.All scale types contain seven notes except Blues, which contains six; Major and Minor are treated as one type because they can share the same note set.Source: §3 · p. 2
- A chord is a group of notes that sound good together, and triads are described in six types: Major, Minor, Augmented, Diminished, Suspended 2nd, and Suspended 4th.The chord types are each defined by a starting note and a set of intervals.Source: §3 · p. 3
- The Circle of Fifths maps 12 chord starting notes to a circle, where moving to a nearby chord is often preferred for a strong-sounding progression.Source: §3 · p. 3
4 HIERARCHICAL RECURRENT NETWORKS FOR POP MUSIC GENERATION
The model generates pop songs with a hierarchy of recurrent layers: scale-conditioned melody is accompanied by chord and drum layers conditioned on the melody.
4 HIERARCHICAL RECURRENT NETWORKS FOR POP MUSIC GENERATION
- A hierarchical recurrent network encodes prior knowledge about pop composition, with lower layers generating melody and higher layers generating drums and chords.The model assumes drums and chords are independent given the melody.Source: §4 · p. 3
4.1 THE ROLE OF SCALE
- Major songs contain 94.66% within-scale notes; Harmonic Minor 87.16%, Melodic Minor 85.11%, and Blues 90.93%, motivating fixed song-level scale conditioning without restricting outputs to scale tones.The scale type is represented by one variable fixed for the whole song.Source: §4.1 · pp. 3–4
4.2 TWO-LAYER RNN FOR MELODY GENERATION
- A scale-conditioned key RNN generates notes, then a second RNN generates press durations; a clustered two-bar melody profile lets users steer melodic ups and downs.The press layer represents duration as a forward count, separating melody flow from how long each key is held.Source: §4.2 · pp. 4–5
4.3 CHORD AND DRUM RNN LAYERS
- 99.19% of observed chords fall into 72 classes and correlate with melody tones; the top 100 half-bar drum patterns account for 94.60% of observed patterns.Chord and drum layers are generated conditioned on the melody.Source: §4 · p. 3, §4.3 · p. 5
4.4 LEARNING
- Each layer is trained with cross-entropy using ground-truth inputs to the next layer, enabling parallel layer training with Adam over 10 epochs.Training uses a learning rate of 2e-3 with 0.99 decay after each epoch.Source: §4.4 · p. 6
4.5 MUSIC SYNTHESIS: PUTTING ALL THE OUTPUTS TOGETHER
- Synthesis samples a scale and melody profile, generates outputs layer by layer, shifts from C for other starting notes, and adjusts melody timing and chord progressions.Chords are generated at half-bars and selected using Circle-of-Fifths pairwise smoothing with dynamic programming; experiments use grand piano only.Source: §4.5 · p. 6
5 EXPERIMENTS
Human evaluations favor the full method over Magenta, while comparisons also examine melody-only generation and temporal alignment. The authors additionally test how much generated music matches the training data.
5 EXPERIMENTS
- Songs were normalized to a 0.5-second beat duration, and non-drum tracks were categorized as melody, chord, or other using note-count thresholds.Generation used 120 beats per minute and four time steps per beat; the MIDI drum channel was reserved for drums.Source: §5 · p. 6
- The survey asked 27 participants to choose between pairs of 30-second clips, with an option to vote neutral.The three comparisons were full method versus Magenta, melody-only versus Magenta, and full method versus no temporal alignment.Source: §5 · pp. 6–7
- Most participants preferred songs from the full method over Magenta, and most preferred the full method over its no-alignment version.Participants also preferred the melody-only method over Magenta, though its score was lower than the full method’s.Source: §5 · p. 7, Table 1 · p. 7
- With 1 hour of training data, the mean matching subsequence was 3.46s; with 100 hours, it was 4.65s.The authors interpret the short matching sequences as evidence that the model generates new music.Source: §5 · p. 7
6 APPLICATIONS
The applications extend the pop-music framework to dancing and karaoke, and to singing stories generated from photos. The dancing and lyric outputs are produced alongside music, while story singing uses generated text as input.
6 APPLICATIONS
- The section demonstrates two applications of the pop-music framework: neural dancing and karaoke, and neural story singing.The dancing application also produces a stickman and sing-along lyrics.Source: §6 · p. 7, §6.1 · p. 7, §6.2 · p. 8
6.1 NEURAL DANCING AND KARAOKE
- A dancing layer predicts one of 456 pose clusters per beat, with moving-average smoothing applied to the generated 3D pose sequence.Pose processing usually extracted reasonable poses, but tracking failures and video effects sometimes reduced quality.Source: §6.1 · p. 8
- The karaoke application generates one lyric word per beat, using lyrics aligned to MIDI through sentence-level timestamps.The lyrics collection contains 51 hours of text, including one hour of songs from Just Dance.Source: §6.1 · p. 8
6.2 NEURAL STORY SINGING
- For story singing, a generated photo story is arranged at one beat per word, then used to condition music and a virtual singer’s pitch.A short pause places each next sentence at a new bar; the singer’s final pitch is constrained to E3–G4.Source: §6.2 · p. 8
7 CONCLUSION AND FUTURE WORK
The paper presents a music-theory-informed hierarchical approach for multi-track pop-song generation, reports human-study strength against a baseline, and demonstrates dancing, karaoke, and story singing. It notes that note-level objectives may not suit music’s flexibility and unpredictability, and calls for deeper study of music theory.
7 CONCLUSION AND FUTURE WORK
- The conclusion presents the hierarchical, music-theory-informed approach as a way to generate multi-track pop songs, with human studies showing strength against an existing baseline.The paper also proposes neural dancing and karaoke, and neural story singing.Source: §7 · p. 8
- The note-level objective may be unsuitable for music’s flexibility and intentional unpredictability; the authors say their treatment of music theory only scratches the surface.They identify deeper study of music theory as future work.Source: §7 · p. 8