PaperlayerView exact revision on arXiv

Source-linked AI summary

PopMAG: Pop Music Accompaniment Generation

Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, Tie-Yan Liu

arXiv:2008.07703v1cs.SDcs.CLcs.IRcs.LGeess.AS

TL;DR

PopMAG addresses the challenge of modeling harmony among accompaniment tracks whose notes are not explicitly dependent on one another. It combines a single-sequence multi-track representation with techniques for long-term modeling, and reports favorable comparisons across three datasets.

  • Problem

    Previous multi-track accompaniment methods did not explicitly model dependencies between notes in different tracks, leaving harmony modeling as a challenge.

    Source: §1 · p. 1

  • Method

    PopMAG uses MuMIDI to represent multiple tracks in one sequence and combines note-attribute packing with extra-long context in a sequence-to-sequence model.

    Source: §1 · pp. 1–2

  • Results

    42%/38%/40% of generated pieces reached ground-truth quality on LMD, FreeMidi and CPMD, respectively, and PopMAG largely outperformed state-of-the-art accompaniment systems.

    Source: §1 · p. 2, §5.1 · p. 7

  • Takeaways & Limitations

    Experiments on LMD, FreeMidi and an internal Chinese pop dataset demonstrate PopMAG’s effectiveness for multi-track harmony and long-term dependency modeling.

    Source: §6 · p. 8

    Chord symbols assume each chord remains unchanged for half a bar and are placed only after <Pos_1> and <Pos_16>.

    Source: §2.4 · p. 3

Abstract

from arXiv · show

In pop music, accompaniments are usually played by multiple instruments (tracks) such as drum, bass, string and guitar, and can make a song more expressive and contagious by arranging together with its melody. Previous works usually generate multiple tracks separately and the music notes from different tracks not explicitly depend on each other, which hurts the harmony modeling. To improve harmony, in this paper, we propose a novel MUlti-track MIDI representation (MuMIDI), which enables simultaneous multi-track generation in a single sequence and explicitly models the dependency of the notes from different tracks. While this greatly improves harmony, unfortunately, it enlarges the sequence length and brings the new challenge of long-term music modeling. We further introduce two new techniques to address this challenge: 1) We model multiple note attributes (e.g., pitch, duration, velocity) of a musical note in one step instead of multiple steps, which can shorten the length of a MuMIDI sequence. 2) We introduce extra long-context as memory to capture long-term dependency in music. We call our system for pop music accompaniment generation as PopMAG. We evaluate PopMAG on multiple datasets (LMD, FreeMidi and CPMD, a private dataset of Chinese pop songs) with both subjective and objective metrics. The results demonstrate the effectiveness of PopMAG for multi-track harmony modeling and long-term context modeling. Specifically, PopMAG wins 42\%/38\%/40\% votes when comparing with ground truth musical pieces on LMD, FreeMidi and CPMD datasets respectively and largely outperforms other state-of-the-art music accompaniment generation models and multi-track MIDI representations in terms of subjective and objective metrics.

1 INTRODUCTION

PopMAG targets harmony in multi-track accompaniment by encoding tracks together and modeling their note dependencies; it also addresses the longer sequences this creates. It reports results across three datasets, including the share of generated pieces judged to reach ground-truth quality.

1 INTRODUCTION

  • Prior multi-track methods lacked explicit dependencies between notes across tracks or faced pianoroll training instability and data sparseness.The paper describes these as challenges to harmony modeling and generated-music quality.Source: §1 · p. 1
  • MuMIDI encodes multiple tracks in one token sequence, making each note depend on earlier notes across tracks and notes already generated at the same step.The sequence also conditions on earlier notes in the same track.Source: §1 · p. 1, Figure 1 · p. 2
  • To address the resulting long-sequence challenge, PopMAG models a note’s pitch, duration and velocity in one step and uses Transformer-XL for long-term dependencies.Its encoder reads conditional tracks such as chord and melody, while its decoder autoregressively predicts target tracks.Source: §1 · p. 2
  • PopMAG received 42%, 38%, and 40% of votes in comparisons with ground-truth musical pieces on LMD, FreeMidi2, and CPMD, respectively, and largely outperformed state-of-the-art music accompaniment generation systems.Source: §1 · p. 2

2 MULTI-TRACK MIDI REPRESENTATION

MuMIDI represents multi-track music as one compact symbol sequence so that cross-track harmony can be modeled within the sequence. Its symbols encode musical positions, tracks, notes, chords and metadata.

2 MULTI-TRACK MIDI REPRESENTATION

  • MuMIDI addresses the lack of explicit cross-track note dependencies in prior methods by encoding multi-track notes in one sequence, where harmony can be modeled inherently.The paper also notes that simply combining tracks as a pianoroll cannot ensure harmony and suffers from data sparseness.Source: §2 · p. 2

2.1 Bar and Position

  • Bars mark new measures, while one of 32 position symbols marks each timestep; sharing a position symbol across events shortens the sequence compared with REMI.Other events starting at that timestep follow its position symbol.Source: §2.1 · pp. 2–3

2.2 Track

  • Track symbols identify the track for subsequent notes, allowing notes from multiple tracks to appear in one sequence.The representation specifies melody, drum, piano, string, guitar and bass track symbols.Source: §2.2 · p. 3

2.3 Note

  • A single note symbol combines its attributes, including pitch, velocity and duration, instead of using the three or more tokens common in other representations.MuMIDI quantizes velocity into 32 levels and duration into 1–32 timesteps.Source: §2.3 · p. 3

2.4 Chord

  • Chord symbols encode 12 roots and 7 qualities, and the representation assumes each chord remains unchanged for half a bar.Chord symbols precede track symbols and are placed after <Pos_1> and <Pos_16>.Source: §2.4 · p. 3

2.5 Meta Symbol

  • Meta symbols can encode piece-level information such as tempo, tonality, style and emotion; this paper demonstrates tempo categories only.Tempo is classified as low, middle or high.Source: §2.5 · p. 3

3 MULTI-TRACK MODELING

PopMAG addresses the long sequences created by single-sequence multi-track modeling with note-level attribute modeling and extra-long context. Its sequence-to-sequence model uses recurrent Transformer encoder and decoder components to model long-term structure.

3 MULTI-TRACK MODELING

  • PopMAG shortens MuMIDI sequences by representing a note’s pitch, duration, and velocity in one step, and adds long context as memory in the encoder and decoder.The paper presents these as two enhancements for modeling long-term sequences.Source: §3 · p. 3

3.1 Modeling One Note in One Step

  • Note-level modeling sums attribute embeddings into one note representation and uses multiple output softmax matrices to predict its attributes.The authors state that the shorter input and output sequences can help capture long-term dependency and speed training and inference.Source: §3.1 · p. 3

3.2 Modeling Long-Term Structure

  • The recurrent Transformer encoder reuses hidden states from previous sequence steps as memory, and its outputs condition the decoder.The model treats one bar as one segment.Source: §3.2 · p. 4

3.3 Model Implementation

  • The implementation section introduces the model’s input and output modules and its training and inference procedures.These are the implementation topics identified by the paper.Source: §3.3 · p. 4, §3.3.2 · p. 5

3.3.1 Input and Output Module.

  • Input embeddings sum token, meta, position, and bar embeddings, while the output module predicts note attributes with separate categorical distributions.Bar and position embeddings replace vanilla Transformer positional encodings; the output module uses pitch, velocity, and duration distributions for notes.Source: §3.3.1 · pp. 4–5, Figure 3 · p. 5

3.3.2 Training and Inference.

  • Training uses teacher forcing and cross-entropy loss; inference generates target-side tokens one by one while updating bar and position embeddings when those symbols appear.The detailed procedures are described in supplementary materials.Source: §3.3.2 · p. 5

4 EXPERIMENTAL SETUP

The experiments use three processed pop-music datasets and evaluate accompaniment generation with subjective listening tests and objective metrics. The setup specifies model and training configurations, evaluation protocols, and measures of chord agreement, sequence fit, and musical-feature distributions.

4.1 Datasets

  • After processing, the datasets contain 21,916 LMD, 5,691 FreeMidi, and 5,344 CPMD pieces, with 100 validation and 100 test samples per dataset.Processing filters tracks and pieces, keeps 4/4 segments with melody and at least one other track, and infers two chords per bar.Source: §4.1 · pp. 5–6

4.2 Model Configurations

  • The recurrent Transformer encoder and decoder use 4 and 8 layers respectively, 512-dimensional embeddings, 512-token training inputs and memory, and 49M parameters.The encoder and decoder use 8 heads each.Source: §4.2 · p. 6

4.3 Training and Evaluation Setup

  • Training runs for 160k steps on two NVIDIA 2080Ti GPUs; the default task generates five accompaniment tracks conditioned on melody and chord, for up to 32 bars.Inference uses stochastic sampling to promote diversity.Source: §4.3 · p. 6

4.4 Evaluation Metrics

  • Objective metrics are averaged over 10 stochastic test-set runs with 95% confidence intervals, while validation perplexity is evaluated once with teacher forcing.The note specifies that PPL is measured on the validation set using the training process's teacher-forcing strategy.Source: §4.4 · p. 6, §4.4.1 · p. 6

4.4.1 Subjective Evaluation.

  • Fifteen participants evaluate 100 listening sets by choosing the piece they prefer for overall harmony, and preference scores average winning votes.Each set pairs pieces from several settings with the same conditional track, and all participants evaluate every set.Source: §4.4.1 · p. 6

4.4.2 Objective Evaluation.

  • Chord Accuracy measures whether generated-track chords match the conditional chords, while token-wise perplexity measures sequence fit on the validation set.Chord Accuracy is aggregated across tracks and chords; perplexity is a metric also used for music generation.Source: §4.4.2 · p. 6
  • Pitch, velocity, duration, and inter-onset-interval distributions are compared with ground truth using averaged overlapped area.Velocity, duration, and inter-onset interval values are quantized into 32 classes; kernel density estimation smooths the feature histograms.Source: §4.4.2 · p. 6

5 RESULTS AND ANALYSES

PopMAG reaches ground-truth-level subjective quality for a substantial share of pieces across three datasets and outperforms MuseGAN and alternative MIDI representations. Its analyses also report benefits from note-level modeling, encoder-decoder memory, and bar and position embeddings.

5.1 Overall Quality

  • 42%, 38%, and 40% of generated pieces reach ground-truth quality on LMD, FreeMidi, and CPMD, respectively, though a gap remains.The evaluation compares generated pieces with human-composed ground truth.Source: §5.1 · p. 7

5.2 Comparison with Previous Works

  • PopMAG outperforms MuseGAN on all subjective and objective metrics in the piano-to-others comparison.Both systems generate four bars under the same data splits and without chord conditioning; MuseGAN cannot generate longer pieces.Source: Table 3 · p. 7, §5.2 · p. 7

5.3.1 Comparison with Other MIDI Representation.

  • MuMIDI scores better than MIDI-like and REMI on subjective and objective metrics, indicating more harmonious generated pieces.The comparison uses the same accompaniment-generation model with context memory for all three representations.Source: §5.3.1 · p. 7

5.3.2 Analyses on Note-Level Modeling.

  • Note-level modeling shortens target sequences and is associated with faster convergence and generation than other MIDI representation modeling methods.Table 4 compares average target-token-sequence length, training time, and inference latency on LMD in the melody-to-others task.Source: Table 4 · p. 7, §5.3.2 · p. 7

5.3.3 Analyses on Memory in the Encoder and Decoder.

  • PopMAG performs best across the reported metrics with memory in both encoder and decoder; decoder memory is more important than encoder memory in most metrics.The authors report that both memories help long-term modeling and the harmony of generated pieces.Source: §5.3.3 · p. 7, Table 5 · p. 8
  • The combination of bar and position embeddings outperforms sinusoidal and relative position encodings, and the individual embeddings are also helpful.The authors report that subjective evaluations are consistent with these comparisons.Source: §5.3.4 · p. 8, Table 5 · p. 8, Figure 6 · p. 8

5.4 Extension

PopMAG supports accompaniment generation from scratch, extension of partially arranged songs, and recomposition; it can also be combined with singing-voice synthesis to produce a whole pop song.

5.4 Extension

  • From scratch, PopMAG generates multi-track accompaniment conditioned on melody and chord.This is one of the three described accompaniment-generation settings.Source: §5.4 · p. 8
  • For semi-manufactured music, it generates additional expressive accompaniment tracks conditioned on melody, chord, and a few existing tracks.Source: §5.4 · p. 8
  • For recomposition, PopMAG removes some tracks and generates other tracks.Source: §5.4 · p. 8
  • Combined with a singing-voice synthesis system, PopMAG can produce a whole pop song.Source: §5.4 · p. 8

6 CONCLUSION

The conclusion presents PopMAG as a framework for multi-track harmony and long-term dependency modeling, combining MuMIDI with an enhanced sequence-to-sequence model. Experiments across three datasets support its effectiveness, while future work includes finer-grained, controllable generation and other music tasks.

6 CONCLUSION

  • PopMAG combines MuMIDI, which encodes multi-track MIDI events as a single sequence, with a sequence-to-sequence model using note-level modeling and extra-long context.Source: §6 · p. 8
  • Experiments on LMD, FreeMidi, and an internal Chinese pop-song dataset demonstrate effectiveness for multi-track harmony and long-term dependency modeling.Source: §6 · p. 8
  • Future work includes fine-granularity and emotion- or style-controlled generation, large-scale generative pre-training, and applications to chord progression generation, singing-voice accompaniment, and MIDI classification.Source: §6 · p. 8

Algorithm 2 PopMAG Inference

The inference procedure initializes encoder and decoder memories, then iterates over bars to generate target tokens and update bar or position state when those tokens appear.

Algorithm 2 PopMAG Inference

A.1 Model Settings

The section identifies Table 6 as the location of PopMAG’s model settings and includes fragments of its training setup. The fragments specify conditional and target tracks, model components, and memory and epoch limits.

A.1 Model Settings

  • Table 6 presents the model settings for PopMAG.The supplied passage identifies the table but does not state its settings.Source: §A.1 · p. 10, Table 6 · p. 10
  • Training input consists of multi-track pieces in MuMIDI, with X as conditional tracks and Y as target tracks; the initialized components include recurrent Transformer encoder and decoder modules, I, P, and encoder and decoder memories.The memory initializations are shown as empty in the supplied fragment.Source: §A.1 · p. 10
  • The setup specifies maximum encoder and decoder memory lengths and a total training epoch count, and the fragments state that Enc, Dec, I, and P are optimized with a loss.The supplied passages do not give the parameter values or loss formula.Source: §A.1 · p. 10

A.2 PopMAG Training and Inference

At inference, PopMAG uses the first sampled token to decide whether the sequence step represents a note. For a pitch token it combines the next two tokens as velocity and duration; otherwise it keeps only the first token as a non-note symbol.

A.2 PopMAG Training and Inference

  • When the first sampled token represents pitch, PopMAG takes the next two tokens as velocity and duration and combines all three into a note symbol.The first token is sampled from categorical distribution D1 at each sequence step.Source: §A.2 · p. 10
  • When the first sampled token is not pitch, inference uses only that token as a non-note symbol.The passage describes this rule as the alternative to assembling a note symbol.Source: §A.2 · p. 10
Loading 2008.07703v1…