RL-Duet: Online Music Accompaniment Generation Using Deep Reinforcement Learning

• The proposed algorithm targets online accompaniment in which a human improvises melody and the machine generates counterpoint, but the paper focuses on algorithm design rather than system usability.System interactivity and usability studies are beyond the paper's scope.Source: Introduction · p. 1
• The authors argue that hand-crafted reward rules are difficult to make comprehensive and balance, and can conflict as music context changes.Rules may, for example, reward pitch variation while other rules encourage repetition.Source: Introduction · pp. 1–2
• RL-Duet learns a reward model from monophonic and polyphonic music that evaluates both inter-part and intra-part compatibility.The model avoids relying on hand-designed composition rules.Source: Introduction · p. 2
• The paper probes the reward model qualitatively and evaluates the method through a subjective user study.Source: Introduction · p. 2
• RL-Duet is presented as the first reinforcement learning model for online music accompaniment generation using an ensemble of reward models.Source: Introduction · p. 2
• Subjective evaluations found higher preference for RL-Duet music pieces than for the baseline method.Source: Introduction · p. 2

Symbolic Music Generation with Deep Learning

• Symbolic music generation uses sequential neural models, hierarchical structures, attention, and GANs, while Gibbs sampling can generate coherent chorales but is offline.Sequential-model examples include RNNs, LSTMs, and GRUs; hierarchical and attention approaches include (Chu, Urtasun, and Fidler 2017) and. GAN examples include (Yang, Chou, and Yang 2017; Dong et al. 2018; Mogren 2016; Yu et al. 2017).Source: Symbolic Music Generation with Deep… · p. 2

Interactive Music Generation Systems

• Interactive music systems include call-and-response and accompaniment; prior accompaniment systems often harmonize or render tracks offline, unlike RL-Duet's online response.MuseGAN follows a human-composed track offline, and the Bach Doodle generates other parts offline; RL-Duet responds online without delay.Source: Interactive Music Generation Systems · p. 2

Reinforcement Learning for Sequence Generation

• Music-generation RL faces reward-design challenges; SequenceTutor uses a trained neural reward alongside more than ten hand-crafted rules and omits inter-part harmonization.RL sequence-generation approaches also optimize long-term discounted reward (Guimaraes et al. 2017; Jaques et al. 2017).Source: Reinforcement Learning for Sequence… · pp. 2–3

Motivation for Reinforcement Learning

• Online generation faces exposure bias and a mismatch between token-level MLE training and music-wide coherence objectives, while beam search is inapplicable online.RL addresses these problems by learning from its own predictions and using data-learned rewards for temporal/rhythmic structure and human–machine harmony.Source: Motivation for Reinforcement Learning · p. 3
• RL-Duet better preserves temporal and rhythmic structures, diversity, and inter-part harmonization than MLE, whose greedy generations can be flat and monotonous.The comparison accompanies an MLE token-prediction accuracy of 83.75% on a Bach duet input.Source: Motivation for Reinforcement Learning · p. 3, Figure 2 · p. 3

Pitch, Duration, Articulation

• Notes are encoded as symbolic MIDI pitches at sixteenth-note steps, with hold symbols for longer durations; multi-hold representation produced more diverse music than single-hold.An additional rest symbol encodes rests.Source: Pitch, Duration, Articulation · p. 3

Model Concepts

• The state uses a sliding window of preceding human and machine notes, and the generation model predicts the next machine token from that context.Notes farther back than the window are ignored.Source: Model Concepts · pp. 3–4

RL Framework

• The actor-critic agent maximizes expected long-term discounted reward; with γ > 0, it accounts for future reward rather than only the current step.Generation and reward phases alternate: notes are produced sequentially, then rewards are computed and used to update the policy and value function.Source: RL Framework · p. 4

Reward Model

• An ensemble of deep reward networks replaces hand-crafted composition rules; specialists are intended to help reject high-scoring but unpleasant or boring “fooling instances.”The reward models consider horizontal temporal consistency and vertical harmony between the parts.Source: Reward Model · pp. 4–5
• Joint reward modeling scores a range of machine notes using pre- and post-context, then disentangles this view into horizontal machine-part consistency and vertical human–machine harmony.The horizontal model uses a Cloze task and can be trained with monophonic music; the vertical model uses the human part and central machine excerpt.Source: Reward Model · p. 5
• Rewards from the four neural submodules are averaged, optionally weighted by user preference, and combined with a −1 penalty for excessive repetition.The total per-step reward is the sum of model-based and rule-based rewards.Source: Reward Model · p. 5

Datasets

• The test comprised 460 duets from 37 held-out Bach chorales, with the human part and first two machine measures fixed before each model generated the remainder.Each generation model greedily generated one accompaniment for each human part.Source: Datasets · p. 5

Model Specification

• The generation and reward model (a) use embeddings, bidirectional GRUs, and temporal pooling and attention to summarize context and produce a note-probability distribution.Reward models (b), (c), and (d) use similar architectures but differ in the inputs to the Temporal Context Summarizer and output note-range probabilities.Source: Model Specification · p. 5, Figure 4 · p. 6

Training Process

• Reward models were trained with maximum likelihood, then the generation model was trained with reinforcement learning for 100K duets while the reward models were used only for inference.The pretrained joint-modeling reward model (a), trained with learning rate 0.01, initialized the generation model.Source: Training Process · p. 6

Objective Evaluation

• RL-Duet’s PC/bar, PI, and IOI were closer to the test set than the baselines; it was more similar on PCH and competitive on NLH.Table 1 reports differences from the test set for PC/bar, PI, and IOI, and earth moving distances for PCH and NLH; smaller values indicate better style imitation.Source: Objective Evaluation · p. 6, Table 1 · p. 6
• Across measures 1–20, RL-Duet was the most robust and had the closest metrics to ground truth, while MLE’s errors accumulated into metric drift.Metrics used a one-sided four-measure sliding window; all models began with the first two machine measures initialized from the ground-truth score.Source: Objective Evaluation · p. 6, Figure 5 · p. 7

Subjective User Study

• Around 58.42% of subjects preferred RL-Duet over MLE; the distinction was more obvious among listeners with greater classical-music familiarity.The result for listeners spending more than five hours per week listening to classical music may be unreliable because too few votes were available.Source: Subjective User Study · p. 7

Interactive Generation

• In an example with roles switched at measures 6 and 10, the machine part produced three motifs: one related to the human part and two to the machine part.The example’s human part remained fixed during generation.Source: Interactive Generation · p. 7

Conclusion

• Figure 7 summarizes subjective votes by musical-instrument skill and classical-music listening time; 58.42% of votes prefer RL-Duet.Source: Figure 7 · p. 8
• Figure 8 shows an RL-Duet accompaniment seeded with the first two machine-part measures, with three motifs highlighted.Source: Figure 8 · p. 8
• RL-Duet shows better global coherence than the MLE model, better style imitation than MLE and rule-based baselines, and higher subjective preference than MLE.The figure reports that 58.42% of votes prefer RL-Duet.Source: Conclusion · p. 8
• Its reward models assess machine notes against intra-part and inter-part contexts from both horizontal and vertical views.Source: Conclusion · p. 8