Source-linked AI summary
Counterpoint by Convolution
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, Douglas Eck
TL;DR
The paper addresses the mismatch between chronological music generation and composers’ nonlinear rewriting of musical ideas. It trains a convolutional orderless-NADE model to complete partial scores and samples with blocked Gibbs updates. Blocked Gibbs sampling improves sample quality over ancestral sampling, including with a cheap approximate procedure, according to likelihood and human evaluations.
Problem
Chronological single-pass generation does not match nonlinear composition, while musical counterpoint requires modeling strong dependencies among notes and voices.
Method
COCONET is a convolutional orderless-NADE model that completes partial scores through conditional prediction and blocked Gibbs resampling.
Results
Blocked Gibbs sampling produces better samples than ancestral sampling, and annealed independent blocked Gibbs is faster while producing better samples.
Takeaways & Limitations
Iteratively rewriting parts of a generated score lets the model revisit choices and improve sample quality for musical counterpoint.
Takeaways & Limitations
Some conditionals, especially those used early with very small contexts, are poorly modeled, making ancestral sampling sensitive to ordering.
Abstract
from arXiv · showhide
Machine learning models of music typically break up the task of composition into a chronological process, composing a piece of music in a single pass from beginning to end. On the contrary, human composers write music in a nonlinear fashion, scribbling motifs here and there, often revisiting choices previously made. In order to better approximate this process, we train a convolutional neural network to complete partial musical scores, and explore the use of blocked Gibbs sampling as an analogue to rewriting. Neither the model nor the generative procedure are tied to a particular causal direction of composition. Our model is an instance of orderless NADE (Uria et al., 2014), which allows more direct ancestral sampling. However, we find that Gibbs sampling greatly improves sample quality, which we demonstrate to be due to some conditional distributions being poorly modeled. Moreover, we show that even the cheap approximate blocked Gibbs procedure from Yao et al. (2014) yields better samples than ancestral sampling, based on both log-likelihood and human evaluation.
1. INTRODUCTION
The paper introduces COCONET, a convolutional model that completes partial musical scores and supports nonlinear score rewriting through blocked Gibbs sampling. It targets structured polyphonic composition, where local and distant note dependencies make conventional sequential generation difficult.
- Motivation: Counterpoint is challenging because each note strongly influences neighboring and more distant notes, while handcrafted rules can conflict or lead to musical dead ends.Statistical music models can help composers explore ideas more quickly.
- Model: COCONET is a deep convolutional model trained to reconstruct partial musical scores.The model uses convolutional structure to capture local patterns while retaining the ability to model larger-scale structure.
- Model: COCONET provides direct access to conditionals p(x_i | x_C) for completing any selected fragment of a score.It is an instance of deep orderless NADE, which learns multiple factorizations of the joint distribution.
- Sampling: Ancestral sampling from orderless NADE produces poor results for highly structured musical counterpoint because it relies heavily on a chosen ordering.The procedure samples variables one by one after randomly selecting an ordering.
- Sampling: Blocked Gibbs sampling repeatedly resamples subsets of notes from conditional distributions, allowing the model to iteratively rewrite and correct the score.The approach is presented as an alternative to a fixed causal direction of composition.
- Scope: COCONET addresses partial-score completion, including bridging fragments, temporal upsampling, extrapolation, rewriting, melodic harmonization, and unconditioned polyphonic generation.The experiments focus on four-voice Bach chorales.
2. RELATED WORK
Related work models polyphonic music with rule-based, optimization, sequence, undirected, recurrent, and Gibbs-sampling approaches. COCONET instead combines orderless NADE with approximate blocked Gibbs sampling for flexible partial-score completion and fast generation.
- Prior approaches: Computer-music systems have addressed polyphonic composition with Markov chains, rule-based methods, optimization, constraint-based methods, and recombination of existing musical fragments.Bach chorales are a principal corpus for full-fledged counterpoint generation.
- Sequence models: HMMs and RNNs model music sequentially, often requiring serialization or alternative representations to fit the single-stream sequence paradigm.This is difficult because music contains multiple interdependent streams of events.
- Neural models: Prior neural approaches represent polyphonic music with piano rolls, chord vocabularies, RBMs, NADEs, deep belief networks, or recurrent architectures.These methods model pitch vectors or serialized and structured musical contexts in different ways.
- Gibbs-based models: Undirected Markov models and DeepBach use Gibbs sampling to model note relationships and support constrained generation, while Markov assumptions can limit expressivity.Other work combines selective Gibbs sampling with optimization and simulated annealing to impose higher-level structure.
- Positioning: COCONET uses orderless NADE to train a mixture of directed models simultaneously and approximate blocked Gibbs sampling for fast generation.The resulting conditional distributions can complete arbitrarily partial musical scores.
3. MODEL
The model represents music as piano rolls and trains a deep convolutional orderless NADE to reconstruct masked score fragments. Because it provides conditionals for arbitrary contexts, it supports flexible completion of partial musical scores.
- Probabilistic model: The model learns pθ(x) as an autoregressive factorization of the joint distribution under orderings of the variables.NADE can use chronological orderings, but different orderings may differ in practice because of optimization and approximation effects.
- Probabilistic model: Orderless NADE trains across orderings and directly accesses pθ(x_i|x_C) for arbitrary contextual subsets, enabling completion of partial scores.Parameter sharing allows partial losses for multiple compatible orderings to be computed together.
- Network architecture: COCONET uses a deep convolutional neural network because contrapuntal rules are local and nearly invariant to translations in time and pitch.The experiments use 64 layers and 128 hidden channels, with batch normalization after convolutions.
- Data representation: Music is encoded as a binary three-dimensional tensor x ∈ {0, 1}^(I×T×P), with instruments, time steps, and pitches as its dimensions.Each instrument is assumed to play exactly one pitch at each time; experiments use four voices, MIDI pitches 36–88, 16th-note resolution, and 128-frame crops.
- Training objective: Training presents a strict subset x_C of a piano roll and trains the model to reconstruct its complement x_¬C.The input combines the masked context with its corresponding mask, and optimization uses a reweighted orderless-NADE loss with stochastic gradient descent.
- Training objective: The final activations are converted with softmax into pitch predictions for each instrument/time pair.The prediction loss is defined over variables outside the observed context.
4. EVALUATION
Evaluation estimates generative quality frame by frame while varying frame and instrument orderings. The procedure avoids teacher-forcing by conditioning on the model’s own prior predictions and averages results across repeated evaluations.
- Framewise evaluation: Framewise evaluation conditions on the model’s own predictions rather than injecting ground truth after each prediction.This makes the estimate sensitive to accumulated error and more representative of generative-model quality than notewise teacher-forcing.
- Averaging and orderings: The evaluation repeats likelihood computation across multiple orderings and averages over frames and examples.Chronological evaluation restricts sampled orderings to increasing frame order.
5. SAMPLING
COCONET samples music by choosing variables in random order, but this ancestral procedure produces poor samples. Blocked Gibbs sampling instead repeatedly resamples masked score regions, improving quality while independent blocked Gibbs offers a cheaper alternative.
- Orderless NADE Sampling: Ancestral sampling starts from an empty piano roll, randomly chooses an ordering, and samples variables one by one until the score is populated.
- Orderless NADE Sampling: The procedure generalizes to partial-score completion and melody harmonization by starting from a nonempty piano roll.
- Orderless NADE Sampling: Poor ancestral samples are attributed to accumulated errors and poorly modeled conditionals, especially when early predictions have little context.
- Gibbs Sampling: Blocked Gibbs sampling repeatedly masks and repopulates score regions, allowing the model to revisit earlier choices.
- Gibbs Sampling: Blocked sampling helps mixing because high temporal resolution creates strong correlations between consecutive notes.
- Gibbs Sampling: Independent blocked Gibbs uses an annealed masking probability that decreases over iterations, moving from fast mixing toward standard Gibbs resampling.
- Gibbs Sampling: Independent Gibbs requires O(N) model evaluations, fewer than the O(IT) required by plain ancestral sampling, and produces better samples than ancestral sampling.
6. EXPERIMENTS
Experiments evaluate COCONET on four-part Bach chorales using multiple temporal resolutions, ordering ensembles, likelihood measurements, and human judgments. Random orderings improve data likelihood, while blocked Gibbs sampling improves generated-sample quality over ancestral sampling.
- 6.1 Data Log-likelihood: COCONET is evaluated on four-part Bach chorales at quarter-note, eighth-note, and sixteenth-note temporal resolutions.The quarter-note representation is supplemented because it is considered too coarse to convey counterpoint accurately.
- 6.1 Data Log-likelihood: Temporal resolution has a dramatic effect on framewise log-likelihood, possibly because teacher-forcing rewards holding notes as chord changes become rarer.The metric conditions each prediction on the ground-truth frames previously predicted.
- 6.1 Data Log-likelihood: Fully random orderings achieve significantly better performance than chronological orderings when likelihoods are averaged across M = 5 orderings.The authors attribute this to the greater complementarity of the random ensemble.
- 6.2 Sample Quality: Increasing context inclusion probability ρ yields better likelihood for ancestral blocked Gibbs samples, supporting the hypothesis that small-context conditionals are poorly modeled.The experiment uses sample length T = 32 time steps and N = 100 Gibbs steps, with ρ = 0 corresponding to NADE.
- 6.3 Human Evaluations: MTurk participants rated independent Gibbs samples as more Bach-like than ancestral NADE samples, while independent Gibbs did not differ significantly from Bach.The overall model difference was significant, with χ2(2) = 12.23 and p < 0.001; the post-hoc ancestral-versus-independent-Gibbs comparison had p < 0.05/3.
7. CONCLUSION
The paper introduces a convolutional orderless-NADE approach for modeling musical scores and finds that rewriting through blocked Gibbs sampling improves generated samples. Annealed independent blocked Gibbs sampling is faster and produces better samples than ancestral sampling.
- COCONET models musical scores with a convolutional approach based on the orderless NADE framework.
- NADE ancestral sampling yields poor samples, which the authors attribute to some conditionals being poorly captured by the model.
- Blocked Gibbs sampling significantly improves sample quality by iteratively rewriting parts of the score.
- Annealed independent blocked Gibbs sampling is faster and produces better samples than ancestral sampling.