Source-linked AI summary
Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
Nicolas Boulanger-Lewandowski, Yoshua Bengio, Pascal Vincent
TL;DR
The paper asks how to model high-dimensional symbolic polyphonic sequences with multimodal next-step distributions and long temporal dependencies, and how such models can aid transcription. It introduces distribution estimators conditioned on recurrent networks, including the RNN-RBM, and evaluates them for music modeling. The models outperform traditional polyphonic-music approaches and improve transcription accuracy when used as symbolic priors.
Problem
High-dimensional polyphonic sequences require modeling multimodal conditional note distributions and temporal dependencies, while existing transcription methods rely primarily on audio signals.
Method
The paper combines recurrent neural networks with conditional RBM-based distribution estimators, generalizing the RTRBM as the RNN-RBM for symbolic music modeling and transcription.
Results
The RNN-based models perform better across a range of polyphonic-music datasets, and symbolic-prior transcription improves absolute frame-level accuracy by 1.3% to 10% over HMM smoothing.
Takeaways & Limitations
The model can learn harmonic and rhythmic probabilistic rules from polyphonic scores and immediately serve as a symbolic prior for polyphonic transcription.
Takeaways & Limitations
The transcription search uses a greedy chronological procedure rather than the HMM's global Viterbi search, and longer-term musical structure remains elusive in the unconstrained representation.
Abstract
from arXiv · showhide
We investigate the problem of modeling symbolic sequences of polyphonic music in a completely general piano-roll representation. We introduce a probabilistic model based on distribution estimators conditioned on a recurrent neural network that is able to discover temporal dependencies in high-dimensional sequences. Our approach outperforms many traditional models of polyphonic music on a variety of realistic datasets. We show how our musical language model can serve as a symbolic prior to improve the accuracy of polyphonic transcription.
1 Introduction
The paper targets high-dimensional polyphonic sequences whose next-step distributions are multimodal and whose dependencies can span long time lags. It combines recurrent temporal modeling with expressive per-time-step distribution estimators and applies the resulting model to music generation and transcription.
- Motivation: RNNs can represent long-term dependencies through internal memory, but efficient gradient-based training remains challenging.Hessian-free optimization was shown to help reduce these training difficulties.
- Motivation: High-dimensional sequence models should predict conditional distributions rather than only expected next-step values because these distributions are often multimodal.In polyphonic music, notes occurring simultaneously have correlated patterns, so one note changes the probabilities of others.
- Related work: Temporal RBMs condition expressive per-time-step distributions on sequence history, while RTRBMs provide exact inference and efficient contrastive-divergence training.The paper positions these models as attempts to combine RBM distribution modeling with recurrent temporal dependencies.
- Contributions: The paper introduces the RNN-RBM, a generalization of the RTRBM that allows more freedom in describing temporal dependencies.It also reports that RTRBMs outperform many traditional polyphonic-music models.
- Scope: The study focuses on symbolic music represented by timing, pitch, velocity, and instrumental information, modeling note simultaneities from preceding intervals.Model quality is characterized by temporal dependencies and chord-conditional distributions.
- Application: A musical language model can provide a symbolic prior that complements audio-based polyphonic transcription.This follows the motivation that musical language models can improve purely auditive music-information-retrieval approaches.
2 Restricted Boltzmann machines
RBMs model joint distributions over visible and hidden units with tractable conditional inference, making them useful distribution estimators for high-dimensional sequence modeling. The section describes their sampling and contrastive-divergence training, along with RBM samples for two musical datasets.
- RBM formulation: An RBM is an energy-based model defining a joint probability over visible inputs v and hidden units h.Its parameters are visible biases, hidden biases, and weights, with normalization by a partition function Z.
- RBM formulation: Given the visible vector, hidden units are conditionally independent, and the reverse conditional is likewise factorized.This conditional structure enables efficient alternating inference.
- Inference and training: RBM inference samples hidden units given visibles or visibles given hiddens using conditional Bernoulli distributions and block Gibbs sampling.Block Gibbs sampling alternates k steps of sampling h|v and v|h.
- Inference and training: Contrastive divergence estimates the negative log-likelihood gradient using opposing positive and negative phases.The negative phase can use a single sample obtained from a k-step Gibbs chain initialized at the training vector.
- Related estimator: NADE offers a related tractable distribution estimator whose visible-unit conditionals depend on preceding visible variables.The paper notes that RBMs can be replaced by NADEs using its exact gradient.
- Musical samples: RBMs can represent complicated musical distributions, producing context-sensitive samples for diverse piano music and more restricted simultaneities for JSB chorales.The samples are mean-field draws, with chord labels shown where analysis is unambiguous.
3 The RTRBM
The RTRBM is a sequence of conditional RBMs whose parameters depend on recurrently summarized history. Its mean-field hidden representation both conveys temporal information and supports exact, efficient recurrent inference.
- RTRBM structure: The RTRBM consists of one conditional RBM per time step, with parameters dependent on sequence history.The history A(t) contains preceding visible vectors and mean-field hidden states.
- Parameterization: In the considered RTRBM, only the biases depend on the previous mean-field hidden state.This restriction gives the model six parameters, while the fully general case is derived similarly.
- Recurrent state: Binary hidden units are used during inference and sampling, but their mean-field values are transmitted to successor time steps.This distinction makes exact inference of the transmitted states easy and improves training efficiency.
- Recurrent state: The transmitted-state recurrence is exactly the defining equation of a single-layer RNN with hidden units represented by mean-field values.The recurrence follows directly from the RBM conditional equations and the history-dependent parameterization.
4 The RNN-RBM
The RNN-RBM generalizes the RTRBM by separating recurrent hidden units from RBM hidden units, allowing more flexible temporal dependencies while retaining conditional RBM modeling.
- Model definition: The RNN-RBM combines a full RNN with the RTRBM graphical model, lifting the constraint that the same hidden units must model distributions and convey temporal information.Its joint distribution remains that of the RTRBM, but the recurrent hidden state is defined independently.
- Model definition: The model uses a single-layer RNN whose recurrent hidden units connect to their direct predecessor and the current visible vector.The RBM portion otherwise matches the RTRBM, with its parameters conditioned on the recurrent state.
- Training: Training propagates recurrent states, samples negative particles from conditional RBMs using k-step block Gibbs sampling, estimates CDk gradients, and backpropagates them through time.The resulting gradient reaches the RNN parameters through the hidden-to-bias connections.
- Generalization: The framework extends beyond RNN-RBMs: substituting NADEs for RBMs yields an RNN-NADE with exact gradient computation.The procedure applies to any RNN architecture and conditional distribution estimator whose parameters are supplied by the RNN.
- Relationship to the RTRBM: The single-layer RNN-RBM reduces to the RTRBM when W2 = W, W3 = W′, and bˆh = bh.Untying these connections does not increase computational cost, while separate hidden-unit counts can improve scaling and performance.
- Initialization: Initialization combines RBM pretraining for W, bv, and bh with RNN cross-entropy pretraining for recurrent and hidden-to-bias parameters.The authors report that RBM initialization produces less noisy filters and that pretraining affects overall performance.
5 Baseline experiments
Baseline experiments compare RTRBM and RNN-RBM prediction on bouncing-ball videos and human motion capture sequences, with the RNN-RBM achieving lower prediction error on both datasets.
- Bouncing balls: On bouncing-ball videos, the squared prediction error is 2.11 for the RTRBM and 0.96 for the RNN-RBM.The videos contain 128 frames at 15 × 15-pixel resolution; predictions use 50 block-Gibbs steps.
- Bouncing balls: The RNN-RBM learns localized edge detectors in nearly all of its first 48 hidden-unit filters on the bouncing-balls dataset.These receptive fields are visualized as input-weight images in Figure 3.
- Human motion capture: On human motion capture, the mean squared prediction test error is 20.1 for the RTRBM and 16.2 for the RNN-RBM.The sequences contain 49 real values per time step and are modeled with a Gaussian RBM variant.
6 Modeling sequences of polyphonic music
The paper evaluates probabilistic models for polyphonic music across four datasets, finding that RNN-based models address trade-offs faced by traditional models and that pretraining affects performance. Generated samples capture local musical structure, while long-term structure and meter remain elusive.
- Datasets and setup: Experiments use four polyphonic-music datasets, each containing at least seven hours, with approximately 67 hours total and average polyphony of 3.9.The representation uses 88 binary visible units spanning piano range A0–C8, aligned to integer fractions of the beat.
- Symbolic prediction: N-gram models perform reasonably on simple datasets but fail in realistic settings because of data sparsity, while note N-grams ignore harmonic dependencies.The optimal sequence order differs across these alternatives, reflecting a trade-off between history length, sparsity, and harmonic modeling.
- Symbolic prediction: RNN-based models robustly address traditional polyphonic-model trade-offs and perform better across a range of datasets.The comparison includes frame-level and temporal models evaluated using log-likelihood and expected frame-level accuracy.
- Training strategies: HF significantly improves RNN density-estimation and prediction performance, and pretraining and finetuning the RNN-RBM components materially affect prediction performance.Figure 4 compares SGD and HF pretraining strategies; all strategies except the baseline involve pretraining.
- Qualitative evaluation: Generated RBM-based samples learn basic harmony rules, melody lines, and local temporal coherence, but long-term structure and musical meter remain elusive.The qualitative evaluation uses generated sample sequences from the four-dataset music experiments.
7 Polyphonic transcription
The paper combines an acoustic polyphonic transcription model with its symbolic sequence model to score candidate note combinations. This symbolic prior improves frame-level transcription accuracy over HMM smoothing, while the search remains limited by its greedy chronological structure.
- Approach: The transcription system combines the Nam et al. (2011) acoustic model with symbolic sequence models to estimate audible note pitches at 10 ms intervals.The acoustic model supplies independent note probabilities, and the combined system evaluates promising fundamental-frequency candidates.
- Approach: The system selects the k = 7 most promising acoustic candidates and jointly evaluates all combinations of M candidates for M ≤ k.Candidate combinations are scored using a cost function combining acoustic and symbolic probabilities.
- Approach: The symbolic predictor uses an approximate sequence history formed from f0 estimates present in at least half the audio frames for each past symbolic time step.The symbolic model is treated as a product-of-experts component with α controlling its confidence.
- Results: 1.3%–10% absolute accuracy improvement is observed over the HMM approach when the best-performing symbolic model is used as a transcription prior.Figure 5 compares the Nam et al. (2011) model alone, after HMM smoothing, and with the symbolic prior across four datasets.
- Limitation: The search algorithm requires a greedy chronological search, whereas the HMM performs a global Viterbi search for the most likely f0 path.The authors identify this search difference as a limitation they are working to address.
8 Conclusions
The paper concludes that its RNN-based model learns probabilistic harmonic and rhythmic rules from polyphonic scores and substantially outperforms popular music-information-retrieval methods. It also improves transcription when used as a symbolic prior, although longer-term structure remains elusive.
- Conclusions: The RNN-based model learns harmonic and rhythmic probabilistic rules from polyphonic music scores of varying complexity.The conclusion reports this capability across the paper’s symbolic music experiments.
- Conclusions: Different strategies for describing temporal dependencies improve prediction accuracy in the evaluated models.The conclusion links prediction gains to temporal-dependency design choices.
- Conclusions: The model serves as a symbolic prior for polyphonic transcription and clearly improves state-of-the-art performance in that application.The conclusion presents transcription improvement as an immediate application of the musical language model.
- Conclusions: Longer-term musical structure remains elusive in the paper’s unconstrained representation.This is the stated scope boundary for the learned musical structure.