Source-linked AI summary
Music Generation by Deep Learning - Challenges and Directions
Jean-Pierre Briot, François Pachet
TL;DR
Deep learning music generation can learn styles from arbitrary musical corpora, but direct generation leaves control, structure, creativity, and interactivity unresolved. The paper analyzes these challenges and reviews strategies and systems addressing them, while noting that a priori anti-copying remains unavailable for deep learning architectures.
Problem
Direct deep-learning music generation leaves control, structure, creativity, and interactivity insufficiently addressed.
Method
The paper analyzes these challenges and illustrates possible strategies through examples of recent music-generation architectures.
Results
The reviewed examples provide promising directions for addressing the identified challenges, including hierarchical modeling and interactive feedback.
Takeaways & Limitations
Deep-learning music generation requires strategies beyond direct generation to address control, structure, creativity, and interactivity.
Takeaways & Limitations
Deep learning lacks an equivalent solution for preventing copied training-corpus material a priori.
Abstract
from arXiv · showhide
In addition to traditional tasks such as prediction, classification and translation, deep learning is receiving growing attention as an approach for music generation, as witnessed by recent research groups such as Magenta at Google and CTRL (Creator Technology Research Lab) at Spotify. The motivation is in using the capacity of deep learning architectures and training techniques to automatically learn musical styles from arbitrary musical corpora and then to generate samples from the estimated distribution. However, a direct application of deep learning to generate content rapidly reaches limits as the generated content tends to mimic the training set without exhibiting true creativity. Moreover, deep learning architectures do not offer direct ways for controlling generation (e.g., imposing some tonality or other arbitrary constraints). Furthermore, deep learning architectures alone are autistic automata which generate music autonomously without human user interaction, far from the objective of interactively assisting musicians to compose and refine music. Issues such as: control, structure, creativity and interactivity are the focus of our analysis. In this paper, we select some limitations of a direct application of deep learning to music generation, analyze why the issues are not fulfilled and how to address them by possible approaches. Various examples of recent systems are cited as examples of promising directions.
1 Introduction
Deep learning extends beyond prediction and classification to music generation by learning styles from arbitrary corpora. The paper frames control, structure, creativity, and interactivity as unresolved limitations of direct generation.
- Technical progress, available datasets, and inexpensive computing helped drive deep learning’s reemergence.
- Deep learning uses multiple layers to automatically extract increasingly abstract representations from data.
- Deep learning systems can learn musical styles from arbitrary corpora and generate content from the learned distribution.
- Direct music generation remains limited by weak control, structure, creativity, and interactivity.
- The article analyzes these challenges, surveys possible strategies, and illustrates them with recent systems.
2 Control
Musicians need to adapt generated ideas through controls such as transposition or note minimization. Standard neural networks lack a direct operational entry point for imposing arbitrary constraints, so control relies on external intervention.
- Musicians may want to transpose ideas or minimize notes, requiring controllable generation.
- Standard neural networks lack the operational structure that lets Markov models attach constraints directly to generation.
- Distributed representations do not directly correspond to the structure of generated content.
- Control strategies therefore intervene externally at different entry points, including encapsulation or reformulation.
2.2 Sampling
Constraint sampling controls generation by selecting valid outputs from randomly generated model samples, but guiding the sampling process toward constraints is a central difficulty.
- Constraint sampling introduces output constraints during model sampling and selects valid solutions from generated random samples.
- A key challenge is guiding the sampling process so generated outputs fulfill the desired constraints.
2.3 Conditioning
Conditioning provides parameterized control by adding information such as chords, genres, instruments, or positional constraints to generation architectures. Anticipation-RNN uses a backward constraint model to help a melody anticipate required notes.
- Conditioning adds arbitrary information, such as labels or other modalities, to guide generation.
- Examples condition rhythm generation on bass lines or beats, MidiNet on chord progressions, WaveNet on genres or instruments, and Anticipation-RNN on positional constraints.
- Conditioning information is usually supplied through an additional input layer, providing parameterized control over generation.
- WaveNet distinguishes global conditioning shared across time steps from local conditioning specific to each time step.
- Anticipation-RNN uses a backward Constraint-RNN whose outputs summarize future constraints and are concatenated with Token-RNN inputs.
- On Bach chorales, the model generated melodies that adjusted their direction toward constrained lower- or higher-pitched target notes.
2.4 Input Manipulation
Input manipulation controls musical generation indirectly by optimizing an input or latent representation toward content, style, structural, or other target properties. The section presents DeepHear, style transfer, and C-RBM as examples of this strategy, while noting limitations in musical style transfer and exact constraint satisfaction.
- General strategy: Input manipulation incrementally modifies an initial or random input to maximize a target property, using back-propagation and gradient descent.The target may involve a visual unit, consonance, content and style, or musical structure.
- DeepHear: DeepHear generates ragtime melodies by decoding random seeds placed in a 16-unit bottleneck learned from 600 measures of Scott Joplin’s music.The same architecture and learned representation can also be used to harmonize a melody.
- Latent representations: Variational autoencoders also explore latent values to generate musical variations, but their latent-space exploration is user-directed.The comparison highlights a distinction between input manipulation and variational autoencoder generation.
- Style transfer: Style transfer separates content from style and guides an initially random input toward both descriptions through gradient following.In music, experiments using spectrograms produced effective but less distinctive results than visual style transfer.
- C-RBM: C-RBM generation combines learned local musical texture with constraints on self-similarity, tonality, and meter through constrained sampling.The process alternates gradient descent for high-level structure with selective Gibbs sampling, while simulated annealing reduces exploration as solution variance decreases.
- C-RBM: C-RBM results are described as quite convincing but not exact, unlike the exactness associated with Markov-constraint approaches.The approach imposes high-level musical structure while sampling from an RBM learned on Mozart sonatas.
2.5 Reinforcement
Reinforcement reframes melody generation as sequential decision-making, combining a learned musical objective with user-defined constraints. The cited RL-Tuner results are preliminary but convincing, and the strategy supports potential adaptation to user feedback.
- Objectives: The strategy combines the output of a trained recurrent network with user-defined constraints as objectives for musical generation.Examples include tonality rules from music theory.
- Formulation: Reinforcement learning treats partial melody generation as a state and selection of the next note as an action.The agent receives rewards and learns a policy intended to maximize cumulative gain.
- RL-Tuner: RL-Tuner uses Q Network and Target Q Network components alongside Note RNN and Reward RNN architectures.Reward RNN is a fixed copy of the trained Note RNN and serves as a reference during reinforcement learning.
- Reward design: The reward combines adherence to the learned model with fulfillment of user-defined constraints such as tonality consistency and avoiding excessive repetition.The learned-model component compares the selected next note with the note predicted by Reward RNN in a similar partial-melody state.
- Results: Results are described as preliminary but convincing, with potential for adaptive generation through user feedback.The adaptive-generation potential follows from incorporating user feedback into the reinforcement strategy.
2.6 Unit Selection
Unit selection generates melodies by retrieving and concatenating musical units according to learned semantic relevance and concatenation cost. Because these criteria can be extended with user-defined constraints, the approach provides explicit entry points for control.
- Overview: Unit selection queries successive musical units from a database and concatenates them into sequences according to user characteristics.The strategy was adapted from text-to-speech systems for melody generation across jazz, folk, and rock corpora.
- Feature extraction: The preparation phase extracts 9,675 features from 10 manually handcrafted musical-unit feature types.The features follow a bag-of-words approach, including pitch-class counts, rhythmic tuples, and ties to previous measures.
- Candidate selection: Successor selection ranks candidates using semantic relevance predicted by one LSTM and concatenation cost predicted by another LSTM.Semantic relevance concerns the ideal next unit, while concatenation cost evaluates the transition from the current unit’s last note to the successor’s first note.
- Generation: A heuristic dynamic-ranking process combines the two criteria to generate an arbitrary-length melody measure by measure.Generation iterates over successor choices until the desired sequence is formed.
- Control: The selection framework offers control entry points because user-defined constraints can extend semantic-relevance and concatenation-cost criteria.This makes the selection stage an explicit location for adding control conditions.
3 Structure
Deep-learning music systems often reproduce style without a clear sense of direction, motivating methods that impose or model musical structure across multiple temporal levels. MusicVAE addresses longer-range organization with a hierarchical recurrent decoder, outperforming a flat architecture on long sequences.
- Motivation: Generated music can match a learned style yet wander without the higher-level organization typical of human-composed music.Examples of musical form include classical movements, jazz AABA or AAB structures, and song refrains, verses, or bridges.
- Motivation: Musical structure also appears at finer grain, such as repeated melodic patterns transposed to fit a new harmonic structure.
- Approaches: Structure can be enforced through constraints, abstract unit sequences, or explicit modeling of multiple temporal and structural hierarchies.The article identifies reinforcement, structure imposition, and unit selection as routes toward this goal.
- MusicVAE: MusicVAE combines a bidirectional encoder with a hierarchical two-level recurrent decoder containing a high-level Conductor and a bottom-layer RNN.The Conductor produces embeddings, which initialize and accompany the lower-level RNN as it generates subsequences.
- MusicVAE: A flat architecture incurred a 27% reconstruction-error increase on 16-measure examples, while listeners showed a very significant gain for the hierarchical architecture.The flat model remained accurate for 2-measure examples but became inaccurate for 16-measure examples.
4 Creativity
Deep-learning music generation raises creativity and copyright concerns because generated material may imitate existing music. The section contrasts similarity checking, constrained generation, heuristic creativity controls, and Creative Adversarial Networks, while identifying important scope limitations.
- Motivation: Creativity in generated music is also an economic and copyright concern, not solely an artistic one.
- Avoiding copying: A posteriori control can check whether generated music copies a significant amount of notes from an existing piece.The text suggests adapting tools that detect similarities in texts.
- Avoiding copying: A priori avoidance is more systematic but challenging; unlike a variable-order Markov solution, deep-learning architectures lacked an equivalent solution in the surveyed work.The Markov approach constrains generation order using minimum and maximum orders to balance junk and plagiarism.
- MidiNet: MidiNet explores creativity by restricting conditioning or weakening feature-matching regularization, but the approach remains ad hoc hyperparameter tuning.
- Creative Adversarial Networks: Creative Adversarial Networks add style-classification ambiguity to the standard GAN signal, encouraging samples close to existing art distributions yet stylistically original.The discriminator’s second signal rewards items that are difficult to fit into established styles.
- Creative Adversarial Networks: CAN assumes a prior style classification and narrows creativity to exploring new styles.
5 Interactivity
Most systems generate music automatically with little support for local revision, unlike musicians’ successive partial refinement. Incremental and locally selectable generation, exemplified by DeepBach, enables users to regenerate only affected musical regions or voices.
- Motivation: Most existing systems offer whole-piece regeneration rather than local modification, conflicting with musicians’ successive partial refinement and adaptation.The section identifies incrementality and locality as requisites for interactivity.
- Generation strategies: Common note-instantiation strategies are single-step global generation and iterative time-slice generation.The former generates all time steps at once; the latter generates one time slice repeatedly with an RNN.
- Generation strategies: Incremental variable instantiation progressively refines a global representation in a nondeterministic order, allowing selective regeneration of intervals or voices.It avoids regenerating the entire musical content.
- DeepBach: DeepBach combines recurrent and feedforward networks, uses bidirectional temporal context, and generates chorales through incremental pseudo-Gibbs sampling.The architecture is replicated for each of the four chorale voices and predicts a current note from surrounding context.
- DeepBach: When a user changes one or two soprano measures, DeepBach can resample only the corresponding counterpoint voices in those measures.
- DeepBach: DeepBach’s interface supports interactive global or partial chorale regeneration for non-expert users and integrates with MuseScore.
6 Conclusion
The article reviews open challenges in deep-learning music generation and surveys strategies and systems addressing control, structure, creativity, and interactivity. It presents these examples as possible directions for the broader research agenda.
- Control, structure, creativity, and interactivity remain open challenges that standard deep-learning techniques do not directly address.
- The article analyzes these challenges, discusses possible strategies, and illustrates them with examples of actual architectures.