PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning Techniques for Music Generation -- A Survey

Jean-Pierre Briot, Gaëtan Hadjeres, François-David Pachet

arXiv:1709.01620v4cs.SDcs.LG

TL;DR

Deep learning music-generation research faces limited evaluation resources and diverse, specialized systems. This survey organizes and compares approaches across five dimensions, concluding that architecture and strategy choices depend on preferences and priorities rather than a general solution.

  • Problem

    Music-generation research lacks reference datasets for evaluating and comparing systems, while generated music has no preexisting objective evaluation measure.

  • Method

    The paper surveys and compares music-generation systems using a five-dimensional framework covering objective, representation, architecture, challenge, and strategy.

  • Results

    The analysis concludes that selecting architectures and strategies depends on preferences and priorities, with no general solution guaranteed.

  • Takeaways & Limitations

    The proposed conceptual framework is intended to help researchers understand issues and compare approaches for deep-learning-based music generation.

  • Takeaways & Limitations

    Music-generation systems are often hyper-specialized for specific objectives or corpora, limiting their general problem-solving scope.

Abstract

from arXiv · show

This paper is a survey and an analysis of different ways of using deep learning (deep artificial neural networks) to generate musical content. We propose a methodology based on five dimensions for our analysis: Objective - What musical content is to be generated? Examples are: melody, polyphony, accompaniment or counterpoint. - For what destination and for what use? To be performed by a human(s) (in the case of a musical score), or by a machine (in the case of an audio file). Representation - What are the concepts to be manipulated? Examples are: waveform, spectrogram, note, chord, meter and beat. - What format is to be used? Examples are: MIDI, piano roll or text. - How will the representation be encoded? Examples are: scalar, one-hot or many-hot. Architecture - What type(s) of deep neural network is (are) to be used? Examples are: feedforward network, recurrent network, autoencoder or generative adversarial networks. Challenge - What are the limitations and open challenges? Examples are: variability, interactivity and creativity. Strategy - How do we model and control the process of generation? Examples are: single-step feedforward, iterative feedforward, sampling or input manipulation. For each dimension, we conduct a comparative analysis of various models and techniques and we propose some tentative multidimensional typology. This typology is bottom-up, based on the analysis of many existing deep-learning based systems for music generation selected from the relevant literature. These systems are described and are used to exemplify the various choices of objective, representation, architecture, challenge and strategy. The last section includes some discussion and some prospects.

Introduction

Deep learning has expanded from conventional prediction and classification into music generation, where systems learn styles from musical corpora. This survey analyzes that domain through representative systems and a tentative multidimensional framework.

  • Introduction: Deep learning combines multilayer neural representations with techniques such as convolutions, pre-training, and LSTM networks.Its defining feature is hierarchical abstraction learned across multiple layers.
  • Introduction: Music generation is a growing deep-learning application that uses widely available corpora to learn musical styles and generate new content.The paper focuses on music alongside other generated media such as images and text.
  • Introduction: Computer-based music systems may autonomously create music or assist human composers, arrangers, and producers.The paper notes that most current deep-learning systems remain focused on autonomous generation, despite growing attention to control and interaction.
  • Introduction: The survey addresses a gap in comprehensive analysis by comparing architectures, systems, and experiments from numerous recent research works.Its proposed conceptual framework organizes design decisions around objective, representation, architecture, challenge, and strategy.
  • Introduction: The analysis is not exhaustive because the domain is active and new proposals continue to appear.The authors selected what they considered representative proposals and experiments.

Chapter 3 Objective

The chapter defines musical-generation objectives by the content produced, its destination and use, the degree of human intervention, and the training corpus’s style. It also introduces representation choices and encoding trade-offs that affect learning and generated-content quality.

  • Objective: Objectives specify the musical type, destination, use, generation mode, and style of the content to be produced.Types include melody, polyphony, accompaniment, counterpoint, chord progression, and lead sheets; destinations include audio systems, sequencers, and human performers.
  • Objective: The training dataset governs musical style, and its coherence, coverage, and scope are fundamental for good music generation.The dataset may specialize in a narrow style or provide broader coverage, with sparsity and coherence affecting its usefulness.
  • Objective: The chapter distinguishes monophonic melody, single-track polyphony, multitrack polyphony, accompaniment, and melody–chord associations.Accompaniment may be counterpoint or a chord progression, while a lead sheet combines melody with chord progression and may include lyrics.
  • Objective: Music generation can be autonomous and automated or interactive, with human users controlling the process through an interface.Most analyzed systems are not yet very interactive, making fully interactive support for musicians an important goal.
  • Representation: Representation choices configure architecture interfaces and may significantly affect learning accuracy and the quality of generated content.Audio may use spectrum or raw waveform representations, while symbolic representations may preserve or collapse enharmonic distinctions.
  • Representation: Value encoding is compact but sensitive to numerical approximation, whereas one-hot encoding is robust but can require many inputs.One-hot encoding is the most common strategy for symbolic representation, while value encoding is used rarely except for audio.

Chapter 5 Architecture

This chapter introduces neural-network foundations, from linear regression and gradient descent to multilayer architectures. It explains how nonlinear hidden layers overcome linear separability limits and how models are trained and evaluated.

  • Historical foundations: Neural networks evolved from the Perceptron, whose inability to classify nonlinearly separable domains motivated multilayer nonlinear architectures.Hidden layers and nonlinear units addressed this limitation, while backpropagation enabled their training.
  • Linear regression: Linear regression models a scalar output from explanatory variables using a bias and learned weight parameters.Training adjusts these parameters to minimize a cost measuring the distance between predictions and observed values.
  • Linear regression: A trained linear regression model uses its fitted hypothesis to estimate an output for a given input.The fitted line is adjusted to mostly match training examples before producing predictions for new input values.
  • Gradient descent: Gradient descent initializes parameters, computes predictions and cost, evaluates gradients, then updates parameters opposite the gradient until the error is minimized.The learning rate controls the update magnitude, and simultaneous updates are required for correct behavior.
  • Gradient descent: For convex linear-regression costs, gradient descent has one global minimum and therefore guarantees finding the optimal model.This guarantee is specific to the convex-cost setting described for linear regression.

5.3 Machine Learning

This section defines machine learning through experience, tasks, and accuracy, then introduces representation, evaluation, and optimization as core components. It surveys learning categories and the neural architectures and patterns used in later music-generation discussions.

  • Machine learning concepts: Machine learning improves accuracy on tasks through experience, with performance distinguished from computational efficiency and prediction accuracy.The text separates performance as a musician’s act, efficiency as computational ability, and accuracy as prediction or classification quality.
  • Learning categories: Supervised learning predicts labeled answers, whereas unsupervised learning extracts information such as features or compressed representations.The section identifies regression, classification, translation, feature extraction, and data compression as examples of these categories.
  • Machine learning components: Machine-learning algorithms combine representation, evaluation, and optimization to specify models, compare them, and search for the best parameters.Gradient descent is introduced as a basic optimization method for minimizing a cost function.
  • Architectures and patterns: The main deep-learning architectures introduced for music generation are feedforward networks, autoencoders, restricted Boltzmann machines, and recurrent networks.Convolutional, conditioning, and adversarial designs are presented as architectural patterns that can be applied to them.
  • Feedforward networks: A multilayer feedforward network consists of input, hidden, and output layers, and nonlinear hidden layers make it a universal approximator that overcomes linear separability limits.The universal approximation theorem does not guarantee that the network will learn the target functions.
  • Evaluation challenge: Music-generation tuning is more subjective than prediction or classification because no preexisting objective evaluation measure exists.The text gives manual evaluation of generated music by musicologists as an example of qualitative assessment.

5.6 Autoencoder

Autoencoders learn compressed representations by reconstructing their inputs, while variational autoencoders impose a prior distribution on latent variables to support sampling and controlled generation. The section emphasizes meaningful latent-space exploration for generating musical content.

  • Autoencoders: An autoencoder reconstructs its input through a hidden layer whose usually smaller dimensionality forces information compression and feature discovery.The encoder compresses information into latent variables, while the decoder reconstructs the original input.
  • Autoencoders: Autoencoder training is unsupervised because examples lack labels, but it uses supervised-learning techniques by setting each target equal to its input.This makes the network learn an identity function.
  • Controlled generation: Latent variables can be instantiated and decoded to generate new musical content corresponding to selected values.This provides high-level control over generated content through the compact representation learned from examples.
  • Sparse autoencoders: A sparse autoencoder constrains hidden units to be inactive most of the time, encouraging each unit to specialize as a feature detector.On images, such units learn edges at different positions and orientations; on audio or symbolic music, they learn domain-relevant features.
  • Variational autoencoders: A variational autoencoder constrains latent variables z to follow a prior probability distribution, usually Gaussian, enabling direct sampling through its decoder.Generation samples z from P(z), inputs it to the decoder, and produces an output following the learned conditional distribution P(x|z).
  • Latent representations: VAE training separates example-specific information into latent variables and common information into decoder weights for reconstruction.The model can consequently learn dimensions of variation in the dataset.
  • Latent-space exploration: Latent-space interpolation produces more meaningful and interesting melodies than interpolation in data space.Variational autoencoders are therefore described as promising models for generation with controlled variations.

5.7 Restricted Boltzmann Machine (RBM)

RBMs are stochastic, two-layer generative networks that learn input distributions and generate samples through alternating updates. Their musical relevance includes learning chord distributions from relatively few examples.

  • RBMs use visible and hidden layers without within-layer connections, with the visible layer also serving as input and output.
  • Unlike autoencoders, RBMs are stochastic, unsupervised, lack a decoder, and manipulate Boolean values in their standard form.
  • RBMs are useful for musical chord generation because they learn distributions efficiently from few examples despite the large combinatorial space of possible note combinations.
  • RBM training alternates feedforward encoding into hidden activations with backward prediction of visible-layer reconstructions.
  • After training, generation randomly initializes the visible vector and alternates hidden and visible updates until the energy stabilizes.
  • The energy function combines visible and hidden biases with weighted visible-hidden interactions, using vectors v and h, weight matrix W, and bias vectors a and b.

5.9 Convolutional Architectural Pattern

Convolution is an architectural pattern that extracts local features through sliding filters, nonlinearities, and pooling. For music, convolutions can share parameters across time, but recurrence remains more common because it models all time steps deeply.

  • A convolution slides a filter across an input, computes mapped dot products, and produces a feature map from the resulting sums.
  • Feature-map size depends on depth, stride, and zero-padding, while filters can detect features such as edges and curves or perform sharpening and blurring.
  • Convolutional architectures typically combine convolution, nonlinear rectification such as ReLU, and pooling to reduce dimensionality.
  • Pooling reduces feature-map dimensionality while retaining significant information and providing invariance to small transformations, distortions, and translations.
  • For musical applications, temporal convolutions can model temporally invariant motives by sharing parameters across neighboring time steps.
  • Unlike shallow temporal convolutional sharing, RNNs share parameters across all time steps and are more frequent in musical applications.
  • Conditioning architectures add information such as chords, genre, instruments, or constraints to control generation, globally across time or locally at individual steps.

6.1 Notations for Architecture and Representation Dimensions

The notation framework records architecture depth or multiplicity and representation size, then illustrates it through MiniBach’s symbolic counterpoint system. MiniBach matches a soprano melody with three accompaniment voices but has fixed-length, deterministic, one-shot generation.

  • Notation framework: Architecture-typen denotes an n-layer architecture, Architecture-type×n an n-instance compound architecture, and One-hot×n a multi-one-hot representation.
  • Notation framework: One-hot×n can encode time steps, voices, or both, as in One-hot×64×(1+3) for MiniBach’s 64-step input and output voices.
  • MiniBach: MiniBach uses a feedforward network to generate three-part counterpoint accompaniment for a soprano melody from paired melody-and-accompaniment examples.
  • MiniBach: Its symbolic piano-roll representation uses one-hot encodings for four measures of 4/4 music across successive time slices.
  • MiniBach: MiniBach is summarized as One-hot×64×(1+3) with a Feedforward2 architecture, and an example generated chorale is provided.
  • Limitations: MiniBach’s fixed-size representation prevents producing longer or shorter pieces, while deterministic single-step generation prevents variation and human intervention.
  • Limitations: The broader challenge list includes variability, expressiveness, consistency, control, originality, incrementality, interactivity, adaptability, and explainability.

6.4 Ex Nihilo Generation

Ex nihilo generation uses latent seeds, stochastic sampling, or related strategies to create music without a full input melody. Autoencoder systems provide controllable latent decoding, while sampling generates content from learned probability distributions.

  • Autoencoder generation: Decoder feedforward generation inserts a seed into an autoencoder’s hidden representation and propagates it through the decoder to produce new content.
  • Autoencoder generation: A stacked autoencoder places the seed at its bottleneck, allowing simple feature information to generate fixed-length musical content.
  • DeepHearM: DeepHearM generates four-measure ragtime segments by feeding random data into 16 bottleneck units and decoding the result.
  • DeepHearM: DeepHearM reported average similarity of 59.6% between notes in generated pieces and notes in training pieces, with the authors associating this plagiarism with the small bottleneck.
  • deepAutoController: deepAutoController applies a stacked autoencoder to Fourier-magnitude spectra from 8,000 songs across 10 genres and provides controls over bottleneck inputs and unit activations.
  • Sampling: Sampling generates elements iteratively from a stochastic model, accepting or rejecting candidates according to a probability density in methods such as Gibbs sampling.
  • Sampling: The RBM-based chord system produces samples whose columns represent simultaneous notes, with chord names shown when the analysis is unambiguous.
  • Sampling: The folk-rnn representation is decoded into playable MIDI melodies, with reported results described as convincing and clearly Celtic in style.

6.7 Expressiveness

Expressive music generation addresses the mechanical quality caused by fixed dynamics and exact quantization. The surveyed approaches use performance-based representations, augmentation, and controllable randomness to model or add expressive variation.

  • Motivation: Fixed dynamics and exact quantization make generated music mechanical, limiting expressiveness and nuance.
  • Performance-based representations: Performance-based representations capture human variations in tempo and dynamics rather than relying only on scores.
  • Representation trade-offs: Audio representations contain implicit expressiveness but make separate control of one instrument’s dynamics or tempo difficult because the representation is global.
  • Performance RNN: Performance RNN trains an LSTM on more than 1,400 skilled-pianist performances containing exact timing and note dynamics.Training examples are augmented with time stretching up to 5% faster or slower and transposition up to a major third.
  • Performance RNN: Its multi-one-hot representation encodes note starts, note ends, time shifts, and dynamics across discrete event values.The encoding includes 128 start values, 128 end values, 100 time-shift values, and 32 dynamics bins.
  • Generation control: Temperature controls generation randomness: 1.0 follows the predicted distribution, lower values increase repetition, and higher values decrease repetition.

6.10 Control

Control strategies determine how deep architectures process representations to generate music while satisfying desired requirements. The surveyed approaches range from sampling and conditioning to input manipulation, reinforcement, and unit selection, with trade-offs between variability, structure, and controllability.

  • Motivation: Deep architectures can induce musical style from a corpus without explicit modeling, but musicians also need control to adapt generated ideas to their objectives.The paper frames control as analogous to steering a powerful generative system.
  • Control strategies: Control strategies include sampling, conditioning, input manipulation, reinforcement, and unit selection.These strategies span bottom-up sampling and top-down structure imposition, with conditioning and reinforcement occupying intermediate positions.
  • Sampling: Constrained sampling selects valid solutions from random model samples, but it can be costly and offers no guarantee of satisfying constraints.The central difficulty is guiding the sampling process toward constraint fulfillment.
  • Sampling: Thresholded sampling removes notes whose probability falls below a threshold, reducing the risk that an unlikely note triggers cascading errors in later sequence generation.The adjusted distribution retains eligible probabilities and renormalizes them with z.
  • Sampling: DeepBach uses partial resampling to regenerate selected musical parts, restrict note ranges, and impose basic rhythm.The strategy supports localized user control rather than requiring regeneration of the entire piece.
  • Latent-space generation: VRAE samples latent vectors or interpolates between songs to generate melodies iteratively, although low corpus quality limits careful evaluation.The authors describe the result as positive but qualify the evaluation because of the corpus.
  • Adversarial generation: Feature matching in C-RNN-GAN achieves a better trade-off between structure and surprise than the other tested variants.The paper relates this result to feature-matching regularization as a creativity-control mechanism.
  • Conditioning: MidiNet’s creative mode balances following the previous melody against following chords, but generated chord violations can sound unpleasant and require handpicking.Stable mode is more chord-dominant, whereas creative mode permits occasional constraint violations.

6.11 Style Transfer

The paper surveys music style transfer across composition, timbre, and performance levels. It emphasizes that music’s multidimensional and anisotropic representations make direct transfer from image style transfer difficult, leaving disentanglement and performance transfer as open directions.

  • Transfer levels: Music style transfer is divided into composition, timbre, and performance transfer, corresponding respectively to score, sound, and performance-control representations.The paper stresses that these levels differ in nature and are interrelated.
  • Composition style transfer: Composition style transfer operates on symbolic representations, including imposing global structure, tonality, and meter on newly generated compositions.C-RBM implements these three constraint types separately.
  • Composition style transfer: Structure descriptors may be too low-level to define a composer’s style, but aggregating them could support higher-level descriptors such as recurring motive transpositions.The paper presents this aggregation as a possible route toward richer style representation.
  • Composition style transfer: DeepJ controls generation style by learning from user-specified musical examples and applying the learned style through conditioning.This differs from structure imposition because style is explicitly specified through examples.
  • Timbre style transfer: Audio timbre-transfer systems generally use spectrograms rather than direct wave signals as their input representation.The surveyed examples include systems by Ulyanov and Lebedev and by Foote et al.
  • Timbre style transfer: Ulyanov and Lebedev’s audio system uses one un pretrained layer with random weights, separating content and style through first- versus second-order activation correlations.The paper reports the resulting sound examples as compelling, while noting that audio transfer is less interesting than painting transfer.
  • Timbre style transfer: Directly transferring image style methods to music is difficult because global audio representations are anisotropic: time and note directions encode different kinds of correlation.Horizontal correlations represent temporal relationships, unlike the direction-independent correlations in natural images.
  • Timbre style transfer: Potential solutions include redefining style correlations to account for time or using a time-compressed representation learned by an RNN encoder-decoder.These are presented as hypothetical directions rather than established solutions.

6.12 Structure

Deep-learning music generators often struggle to learn or produce high-level musical structure. Hierarchical architectures such as MusicVAE are explored to capture longer-term organization while supporting multivoice generation and latent-space manipulation.

  • Structure: Most existing systems generate stylistically consistent music without clear global structure or a sense of direction.The paper distinguishes global forms such as AABA from finer-grain structures such as repeated and transposed melodic motifs.
  • Structure: Learning and inventing high-level structures remains an open challenge beyond imposing predefined forms.The paper identifies hierarchical temporal and structural processing as a natural research direction.
  • MusicVAE: MusicVAE uses a variational recurrent autoencoder with a two-level hierarchical RNN decoder for musical generation.Its decoder combines a high-level conductor RNN with a bottom-layer RNN; trio generation uses separate LSTMs for melody, bass, and drums.
  • MusicVAE: MusicVAE’s corpus includes monophonic melodies, drum patterns, and 16-measure trio sequences encoded as MIDI-event tokens.The reviewed examples cover 2- or 16-measure melodies and drum patterns, plus melody, bass, and drums in trio sequences.
  • MusicVAE: 27% error increase separates the flat and hierarchical MusicVAE architectures on 16-measure autoencoder reconstruction.The reported accuracies are 0.883 for the flat architecture and 0.919 for the hierarchical architecture; the flat model was accurate for 2-measure examples.
  • MusicVAE: MusicVAE supports latent-space interpolation and averaging, including generating a melody by averaging two melodies’ latent representations.The survey also reports attribute-vector operations for modifying selected musical attributes.

6.13 Originality

Originality is both an artistic and copyright concern in generated music. The survey contrasts post-generation similarity checks with more systematic control strategies and discusses CAN as a style-deviation approach to creativity.

  • Originality: Originality in generated music raises both creativity and copyright concerns.The paper separates artistic originality from the economic implications of copyright.
  • Originality: Post-generation originality control can detect whether generated music significantly copies an existing piece.The paper suggests adapting text-similarity algorithms to compare musical content.
  • Originality: A priori control aims to prevent recopying portions of the training corpus during generation, but deep-learning solutions are not yet available.A variable-order Markov-chain method uses minimum and maximum generation-order constraints to balance undesirable output and plagiarism.
  • Originality: MidiNet’s creativity controls remain ad hoc hyperparameter tuning through restricted conditioning and reduced feature-matching requirements.These adjustments reduce the requirement that generated and real-data distributions remain close.
  • Originality: CAN extends GANs by learning styles while deviating from style norms through realism and style-ambiguity signals.Its experiments used 81,449 WikiArt paintings tagged with 25 painting styles.
  • Originality: CAN assumes prior style classification and reduces creativity to exploring new styles.The survey notes that this assumption gives the approach some grounding in art history but constrains its interpretation of creativity.

6.14 Incrementality

Deep-learning generators differ in how incrementally they instantiate notes, from whole-content feedforward processing to recurrent iteration and incremental sampling. Incrementality enables selective regeneration, but strategies impose different structural and length constraints.

  • Incrementality: One-shot feedforward or autoencoder generation and recurrent time-slice generation are strong approximations to composers’ successive refinement of arbitrary parts.The survey presents this mismatch as a limitation of straightforward deep architectures.
  • Strategies: The three main note-generation strategies are single-step feedforward, iterative feedforward, and incremental sampling.They differ in whether processing covers a global representation, a local time step, or incrementally selected global cells.
  • Strategies: Incremental sampling can regenerate an arbitrary time slice or subset of voices without regenerating the entire musical content.This selective regeneration is represented by a user-selected zone in the strategy comparison.
  • DeepBach: DeepBach combines two LSTMs and two feedforward networks and models both forward and backward temporal directions.Its architecture is replicated for each of the four chorale voices and uses surrounding contextual notes and metadata.
  • DeepBach: DeepBach generates polyphony through incremental pseudo-Gibbs sampling of individual notes.Training repeatedly selects a note and its context, while generation samples note values following the learned distribution.
  • DeepBach: More than 1,200 human subjects found it significantly difficult to discriminate Bach chorales from DeepBach-generated chorales.The evaluation included experts and novices through a web questionnaire, with an additional live Dutch television evaluation.

6.15 Interactivity

Interactive music-generation systems let users control or selectively regenerate musical material, addressing the limitations of autonomous generation. The survey also highlights unresolved issues in adaptation, explainability, and trade-offs among generation strategies.

  • Interactive systems: Interactive systems are needed to support human composition, counterpoint, harmonization, analysis, arranging, and other musical tasks incrementally.FlowComposer is cited as an example of a companion system for human users.
  • Interactive systems: deepAutoController exposes decoder inputs and individual hidden-unit activations for user control of generation.Users can select an input, generate a random decoder input, or scale or mute a unit.
  • Interactive systems: DeepBach’s MuseScore plugin supports partial chorale regeneration under user-defined voice, phrase, range, and rhythm constraints.Users can freeze a soprano and resample other voices, alter fermatas, restrict note ranges, or impose rhythms through hold symbols.
  • Adaptability: Current deep-learning systems do not learn or adapt during generation, despite the possibility of user feedback about output quality.The survey distinguishes training-time learning from generation-time adaptation and discusses explicit and implicit feedback.
  • Adaptability: Adding positively selected outputs to the training set may cause overfitting, lose generalization, and does not provide direct negative feedback.The paper notes that retraining is not guaranteed to improve overall generation quality.
  • Explainability: Explainability remains difficult because deep-learning systems are black boxes whose learned representations and generation decisions are hard to justify.The survey frames explanation of what and how the system learned as an open issue.
  • Explainability: BachBot analyses associate particular network neurons with cadences, chord progressions, and phrase-ending patterns.The survey presents these correlations as preliminary evidence relevant to understanding learned musical features.
  • Discussion: Generation strategies trade off capabilities: DeepBach’s sampling is incremental but fixed-length, whereas iterative feedforward generation is variable-length but only forward incremental.The survey presents these limitations as potentially conflicting rather than independently solvable.

7.2 System Analysis

The analysis positions surveyed music-generation systems across objectives, representations, architectures, challenges, and strategies, then examines correlations among these choices. It identifies recurring design pairings while noting that the framework and correlation tables remain preliminary analytical tools.

  • The authors summarize systems across objective, representation, architecture, strategy, and challenge dimensions using tables that mark each system’s matching types.Additional notation conveys the number of occurrences of a type.
  • Correlation analysis: The correlation analysis examines relationships among representations, objectives, architectures, strategies, and challenges, but an absent table mark does not establish incompatibility.A mark indicates that at least one analyzed system matches both types; absence only means no such analyzed system was identified.
  • Temporal scope: Global temporal representations typically pair with feedforward or autoencoder architectures, whereas time-step representations typically pair with recurrent architectures and iterative feedforward strategies.The analysis also reports associations between audio versus symbolic representations and accompaniment objectives versus seed-based generation.
  • Global versus time-step representation: Global representations support arbitrary output and sampling-based incremental instantiation but do not allow variable-length generation or feedforward seed-based generation.Autoencoder decoders can provide seed-based generation, including in DeepHear.
  • Global versus time-step representation: Time-step representations allow variable-length generation, while recurrent iterative generation supports incremental instantiation only forward in time.CONCERT exemplifies variable-length generation.
  • Global versus time-step representation: RNN encoder-decoder architectures are presented as combining iterative variable-length generation with arbitrary output and manipulation of global latent variables.This combination is described as addressing advantages associated with both temporal-scope choices.
  • Convolution versus recurrent architectures: Recurrent networks are described as popular and accurate, while convolutional networks are faster to train and easier to parallelize but increase input volume and parameter counts.The discussion also contrasts local convolutional weight sharing with recurrent parameter sharing across all time steps.
  • Challenges and limitations: The surveyed systems are often hyper-specialized for particular objectives or corpora, and several challenges may require extending systems beyond their current designs.The authors also note that limitations and challenges differ in importance and difficulty.

8.7 Conclusion

The paper presents a survey and analysis of deep-learning approaches for musical-content generation through a five-dimensional conceptual framework. It compares systems from the literature to clarify design choices and support analysis of current and future approaches.

  • The paper proposes a five-dimensional framework covering objective, representation, architecture, challenge, and strategy for analyzing deep-learning music-generation systems.
  • The authors analyze and compare systems and experiments from the literature using a bottom-up conceptual framework and typology.
  • The framework is intended to help clarify issues and compare approaches for deep-learning music generation.
Loading 1709.01620v4…