PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Deep Music Generation: Multi-level Representations, Algorithms, Evaluations, and Future Directions

Shulei Ji, Jing Luo, Xinyu Yang

arXiv:2011.06801v1cs.SDcs.LGeess.AS

TL;DR

Automatic music-generation surveys have often emphasized algorithms without clearly comparing tasks across music-representation levels. This paper organizes deep music generation into score, performance, and audio levels, surveys tasks and methods within them, and synthesizes representations, datasets, evaluations, challenges, and future directions.

  • Problem

    Previous reviews generally classify music-generation research by algorithms rather than clearly comparing composition tasks across score, performance, and audio representation levels.

  • Method

    The paper provides a task-oriented survey that divides music generation into score, performance, and audio levels and reviews methods, representations, datasets, evaluations, and challenges.

  • Results

    The survey maps diverse deep-learning composition tasks and their development across the three music-generation levels.

  • Takeaways & Limitations

    The task-focused classification helps researchers locate related methods and understand the field’s tasks, development, shortcomings, and challenges.

  • Takeaways & Limitations

    Objective music-quality metrics do not consistently correlate with human judgment, so subjective evaluation remains indispensable.

Abstract

from arXiv · show

The utilization of deep learning techniques in generating various contents (such as image, text, etc.) has become a trend. Especially music, the topic of this paper, has attracted widespread attention of countless researchers.The whole process of producing music can be divided into three stages, corresponding to the three levels of music generation: score generation produces scores, performance generation adds performance characteristics to the scores, and audio generation converts scores with performance characteristics into audio by assigning timbre or generates music in audio format directly. Previous surveys have explored the network models employed in the field of automatic music generation. However, the development history, the model evolution, as well as the pros and cons of same music generation task have not been clearly illustrated. This paper attempts to provide an overview of various composition tasks under different music generation levels, covering most of the currently popular music generation tasks using deep learning. In addition, we summarize the datasets suitable for diverse tasks, discuss the music representations, the evaluation methods as well as the challenges under different levels, and finally point out several future directions.

1 Introduction

The paper frames automatic music generation as three representation levels—score, performance, and audio—and reviews prior work by algorithms rather than by composition tasks. It proposes a task-oriented survey that organizes methods, datasets, representations, evaluations, and challenges across these levels.

  • Music generation levels: Music generation comprises score generation, performance generation, and audio generation, with lower levels conditionable on higher-level results.Score generation produces symbolic scores; performance generation adds expressive characteristics; audio generation renders or directly generates sound.
  • Score generation: Score generation is the most established level and includes melody, polyphonic music, accompaniment, style transfer, interactive generation, and music inpainting.Score representations are discrete and capture features such as pitch, duration, and chord progression.
  • Survey perspective: The survey classifies research by music-generation level and specific composition task instead of grouping tasks primarily by algorithm.This organization is intended to help researchers locate work relevant to a particular subtask.
  • Survey scope: The paper also covers datasets, music representations, evaluation methods, challenges, and future directions for diverse generation tasks.Its organization includes dedicated discussions of representations, task classifications, datasets, and conclusions.

2 Related Work

Prior automatic music-generation research spans rule-based, probabilistic, evolutionary, neural, and deep generative approaches, while existing reviews commonly emphasize algorithms or broad histories. Deep learning research has evolved from recurrent models toward VAE-, GAN-, Transformer-, and raw-audio systems across score, performance, and audio generation.

  • Field development: Automatic music generation has developed for more than 60 years through rule-based, probabilistic, evolutionary, neural-network, and deep-learning methods.Research has addressed varied generation tasks and datasets throughout this development.
  • Related surveys: Existing reviews cover histories, AI methods, computational intelligence, algorithmic composition, and functional taxonomies, but generally do not clearly divide generation levels and subtasks.The survey positions its representation-level and task-level organization as a distinction from these approaches.
  • Traditional methods: Traditional approaches include grammar rules, probabilistic models, evolutionary algorithms, intelligent agents, physical or acoustic modeling, and concatenative synthesis.These methods differ across symbolic composition, expressive performance, and audio-generation settings.
  • Neural sequence models: RNNs were early neural models for music, while LSTMs and later architectures addressed sequence memory and long-term structure to varying degrees.Examples include early monophonic melody generation, RNN-RBM, Melody RNN, and Anticipation-RNN.
  • Deep generative models: Deep music-generation research expanded to hierarchical VAEs, CNN-based GANs, Transformers, expressive-performance RNNs, and WaveNet-family raw-audio models.These models target symbolic music, performance rendering, singing voice, and audio synthesis.
  • Applications: Commercial systems demonstrate applications including AI-assisted composition, improvisation, and professional music production.Examples include Iamus, GarageBand, and Band-in-a-Box.

3 Representation

The survey distinguishes symbolic and audio representations, then details sequence, MIDI-like, pianoroll, and alternative encodings with their modeling trade-offs. Representation choices determine how rhythm, timing, dynamics, and harmonic structure can be expressed.

  • Symbolic representations encode discrete concepts such as pitch, duration, and chords, while audio representations are continuous.
  • 1D representation: MIDI-like encodings use events such as note-on, note-off, pitch, velocity, and tempo, with time shifts determined by beat resolution or fixed intervals.REMI replaces note-off with duration and time shifts with bar-position events to improve rhythmic modeling and provide a metric grid.
  • 1D representation: Sequence-based representations align separate pitch, duration, chord, and bar-position sequences to corresponding musical events.
  • 2D representation: Pianoroll represents score information as a two-dimensional matrix, commonly binary but sometimes storing velocity, duration, or multiple note attributes.Its image-like structure is easy to understand and widely used.
  • 2D representation: Pianoroll omits note-off information and discretizes time, making repeated short notes indistinguishable from long notes and limiting expressive timing.Proposed remedies include hold/replay markers, separate onset and offset steps, continuation symbols, and Conlon pianoroll.
  • 2D representation: A univariate categorical representation permits arbitrary timing and explicitly models onsets and offsets, distinguishing repeated eighth notes from a quarter note.It also avoids the prohibitively fine unified grid that can make complex music difficult to learn.
  • Others: Learned vector representations can capture tonal and harmonic relationships, including functional chord relations, circle-of-fifths structure, and key relationships.
  • Audio representation: Amplitude spectra organize audio by frequency and amplitude but require phase reconstruction, whose iterative Griffin-Lim procedure is not generally real-time and can yield local-minimum artifacts.

4 Multi-level Deep Music Generation

The survey organizes deep music generation by the hierarchical and multimodal nature of music representations. It reviews score, performance, and audio generation, further subdividing each level by task and comparing methods chronologically.

  • Music generation is divided into score, performance, and audio levels corresponding to symbolic structure, expressive timing and dynamics, and continuous acoustic characteristics.
  • Within each level, the survey further classifies generation tasks and compares deep learning methods and architectures for the same task in chronological order.

4.1 Top-level: Score Generation

Score generation works with symbolic representations of musical features and spans melody, polyphony, multi-track music, and orchestral arrangement tasks. Deep-learning approaches include recurrent, variational, adversarial, and hybrid architectures, while generation becomes more difficult as musical dependencies expand.

  • Score generation encodes abstract features such as pitch, duration, chords, tonality, and repetition, producing symbolic music that must be converted to audio for listening.
  • Monophonic/Melody: Melody generation is typically monophonic, requiring prediction of one note at each time step, with RNNs, VAEs, GANs, and unit-selection systems among the main approaches.
  • Polyphony: Polyphonic generation must model both temporal sequences and harmonic relationships among simultaneous notes, making it more complex than monophonic generation.
  • Polyphony: Transposition-invariant architectures address RNNs’ reliance on absolute note relationships, but BALSTM can shift styles within a piece when trained on multiple styles.
  • Polyphony: JSB four-part chorales became a standard benchmark for polyphonic modeling, supporting comparisons across LSTM, GRU, and related architectures.
  • Multi-track/Multi-instrument music: Conditional RBM enabled automatic orchestral arrangement from real-time piano input by learning relationships between piano scores and composers’ orchestrations.
  • Multi-track/Multi-instrument music: Multi-track generation faces interdependence and temporal-structure challenges; GAN-based systems can produce fragmented notes, unstable results, and insufficient harmony or coherence.

4.1.2 Conditioning

Conditioning guides score generation with additional information such as chords, melody, style, or existing tracks. These conditions support harmonization, arrangement, accompaniment, and rhythm-coherent multi-track generation.

  • Conditioning: Conditional generation adjusts music creation using information such as melody, chords, music style, or other tracks instead of generating sequences without constraints.
  • Melody generation: Chord-conditioned melody generation uses harmonic context to guide note selection and improve the structure of generated melodies.
  • Melody harmonization: Melody harmonization generates chord or polyphonic accompaniment for a given melody, addressing the difficulty of creating suitable chords without composition expertise.
  • Melody harmonization: BLSTM harmonization improved over HMM evaluation but constrained outputs to one triad chord per measure, while MTHarmonizer also predicts chord functions to reduce monotonous progressions.
  • Arrangement: Lead-sheet arrangement converts a lead sheet into piano-rolls for multiple accompanying instruments using conditional convolutional GANs.
  • Multi-track conditioning: Track-conditional generation extends harmonization by allowing any one or multiple tracks to condition accompaniment, while drum generation can use metrical, melodic, and instrumental information for rhythm coherence.

4.2 Middle-level: Performance Generation

Performance generation adds expressive timing, dynamics, articulation, and related characteristics to symbolic scores or composes music with those characteristics directly. The survey distinguishes rendering a given score from composing expressive performances, while highlighting data requirements and modeling limitations.

  • Middle-level: Performance Generation: Performance generation supplements score information with expressive timing, dynamics, articulation, and other changes that make performances vivid rather than metrically rigid.
  • Performance characteristics: Performance modeling uses score and performance features including tempo, onset deviation, velocity, pedal, rests, articulation markings, and dynamic directions.
  • Render expressive performance: Rendering expressive performance generates performance features for an existing score without re-modeling already specified pitch and duration.
  • Render expressive performance: The score-to-performance mapping is not one-to-one because musicians inject individual style, which GenreNet models through score-conditioned dynamics generation.
  • Render expressive performance: NES-MDB pairs four-instrument scores with dynamic and timbre properties because datasets lacking expressive details can yield rigid and boring generated music.
  • Compose expressive performance: Composing expressive performance jointly models score features and performance characteristics, offering greater relevance to improvisation than rendering a specified work.
  • Compose expressive performance: Performance RNN produces expressive local phrasing but lacks overall consistency, whereas Music Transformer outperforms it for expressive piano-performance modeling.
  • Compose expressive performance: NoteTuple groups note attributes into one event to shorten sequences, while transformer-based models can additionally control global structure and style.

4.3 Bottom-level: Audio Generation

Audio generation produces audible music directly, spanning waveform, frequency-domain, vocoder, and synthesizer approaches; singing voice synthesis conditions audio generation on linguistic and musical score features.

  • 4.3.1 Audio synthesis: WaveNet introduced a breakthrough in audio synthesis by using dilated causal convolutions to generate conditioned speech and music waveforms.Its generated clips were realistic but lacked long-term consistency across genre, instrumentation, volume, and sound quality.
  • 4.3.1 Audio synthesis: SynthNet learned mappings from pianoroll notes to instrument timbres, training faster while generating audio with quality close to real samples.Unlike WaveNet autoencoder, it learned timbre from an entire song rather than separately marked notes.
  • 4.3.1 Audio synthesis: Audio models including WaveNet, SampleRNN, and NSynth require many parameters, long training, and abundant examples while offering limited interpretability.Alternative latent-space and scattering-based approaches were explored to simplify synthesis and model perceptual timbre structure.
  • 4.3.2 Singing Voice Synthesis(SVS): Singing voice synthesis generates voices from lyrics and musical features, requiring models to represent dynamic F0 patterns and spectral changes unlike ordinary speech.Typical systems align score and acoustic features, predict frame-level acoustics, generate trajectories, and vocode them into waveforms.
  • 4.3.2 Singing Voice Synthesis(SVS): Adversarial training alleviates over-smoothing in predicted acoustic parameters and improves the naturalness of synthesized singing compared with DNN-based methods.Traditional training struggles with the multi-modal distribution of acoustic features, often producing over-smoothed outputs.
  • 4.3.2 Singing Voice Synthesis(SVS): Unconditional or weakly conditional singing generation relaxes the requirement for pre-assigned scores and lyrics by using noise or accompaniment-based inputs.These studies extend SVS beyond conventional score-and-lyrics-conditioned settings.

4.4 Fusion Generation

Fusion generation combines multiple music representations, often linking symbolic structure with audio synthesis to obtain coherent and expressive music across levels.

  • 4.4.1 Score-to-Audio: Symbolic-to-audio systems address the high computational cost and weak long-term dependence of direct audio synthesis by inserting structured musical representations.Biaxial LSTM outputs can condition WaveNet, while Wave2Midi2Wave uses MIDI to span musical timescales from 0.1 ms to 100 s.
  • 4.4.1 Score-to-Audio: PerformanceNet maps pianoroll symbols to spectrograms, predicts expressive tempo and pitch changes, and outperforms a WaveNet-based model and two existing synthesizers in subjective evaluation.It supports any instrument but requires a separate model for each instrument.
  • 4.4.2 Audio-to-Score: Fusion generation also includes audio-to-score transcription, where encoder-decoder models predict polyphonic notes and instruments and can rearrange timbre representations.These approaches reverse the usual direction by deriving symbolic structure from audio.
  • 4.4.3 Lyrics-to-Melody: Lyrics-to-melody generation composes melodies from text while aligning generated notes with lyrics, including one-to-many syllable-to-note relationships.Songwriter uses an end-to-end sequence-to-sequence model that generates sentence-level melody segments and merges them.

4.5 Style Transfer

Style transfer generates new music by separating and recombining content and stylistic characteristics across symbolic and audio representations. The surveyed methods address genre, rhythm, texture, timbre, and other audio styles, but face alignment, long-term structure, and data limitations.

  • Genre style transfer: Genre transfer commonly uses unsupervised frameworks because aligned cross-genre data are scarce, with MIDI-VAE changing pitch, dynamics, and instruments.MIDI-VAE uses a style classifier and shared latent space to manipulate existing songs and generate multi-instrument polyphonic music.
  • Rhythm style transfer: Rhythm transfer disentangles pitch and rhythm representations, but early methods were limited to short segments or required splicing for longer pieces.EC2-VAE reconstructs 8-beat melodies conditioned on chords by combining rhythm features with pitch latent vectors.
  • Other style transfer: FusionGAN combines multiple genres through adversarial learning and Wasserstein distance, and experiments showed effective merging of two music genres.The framework uses a three-way GAN and dual learning to create a new domain from existing genre patterns.
  • Audio style transfer: Audio style transfer operates on spectrograms or time-domain signals, but phase reconstruction can omit fine temporal characteristics and limit audio quality.Direct time-domain optimization reduced noise relative to Ulyanov’s method, while meaningful audio required combinations of real, imaginary, and magnitude inputs.
  • Audio style transfer: Parallel use of mel and CQT spectrograms captures rhythmic and harmonic characteristics, producing the most convincing reported style-transfer examples.Mel spectrograms capture rhythmic information better, whereas CQT spectrograms represent harmonic style.
  • Timbre style transfer: Timbre-transfer models learn disentangled pitch and timbre representations, but some require external timbre selection, synthetic MIDI audio, or scarce frame-wise labels.UnetAE better created new instruments without losing pitch, while later GM-VAE work enabled controllable multi-instrument transfer without domain-specific decoders.

4.6 Interactive Generation

Interactive music generation places human input within the generation loop through call-and-response, accompaniment, browsing, inpainting, and audio-control interfaces. The surveyed systems increasingly support real-time interaction, although some remain offline or offer limited performance.

  • Call and response: Deep autoencoder embeddings support call-and-response by transforming input representations and combining generation with nearest-neighbor unit selection.The system reconstructs transformation vectors and selects appropriate responses through embedding-space search.
  • Accompaniment: RL-duet enables real-time accompaniment without delay by learning rewards for horizontal and vertical compatibility between generated notes.Unlike Bach Doodle’s offline Gibbs sampling, RL-duet uses learned rewards rather than manually defined composition rules.
  • System limitations: Some interfaces remain constrained by offline generation or weak audio quality, limiting spontaneity, reactivity, or practical interaction.Bach Doodle generates accompaniment offline, FlowComposer lacks real-time audio and MIDI playback, and one stacked-autoencoder system has poor performance.
  • Interactive browsing: MusicVAE-based interfaces let users browse latent spaces and define trajectories for continuous morphing of melody loops and drum beats.The interface supports user-directed exploration of two-bar melody and drum-beat sequences.
  • Multimodal interaction: Interactive systems also accept humming, singing, speaking, images, or emojis as inputs for generating music and audio.Composer4Everyone generates music from recordings, while other systems organize input, generation, and output components around conversion and indexing databases.
  • Drum interaction: Real-time drum systems combine generation models with performer interaction or intuitive plugin controls, extending interactive generation beyond melody and accompaniment.MANDI uses a temporal convolution network to improvise drum patterns with human performers.

4.7 Music Inpainting/Completion

Music inpainting and completion fill missing score or audio information while preserving musical continuity. The task differs from conventional left-to-right generation because it can use context on both sides of a gap, but long-range structure remains difficult.

  • Score inpainting: Score inpainting generates a sequence between past and future contexts, matching human completion of existing fragments more closely than purely continuous generation.The formulation uses C_p and C_f to generate C_i that connects both contexts musically.
  • Score inpainting: Existing score-completion systems include selective note regeneration, context-based chorale completion, and convolutional completion of partial scores.Anticipation-RNN, DeepBach, and Coconet represent different approaches to filling missing symbolic material.
  • Audio inpainting: Audio inpainting restores local lost information through interpolation, extrapolation, or waveform substitution, with short-gap recovery as one major setting.The task requires new audio to remain semantically compatible with music’s underlying structure and long dependencies.
  • Audio inpainting: Time-frequency context encoders improve musical-audio inpainting for gaps of tens of milliseconds, while instrumental-audio performance remains inferior.The model uses surrounding signals as context through convolutional and fully connected layers.

4.8 Generation with Emotion

Emotion-conditioned generation addresses the difficulty of steering automatically generated music toward desired emotional states. The surveyed mLSTM approach controls polyphonic generation through emotion-related neurons, and listeners recognized its outputs at human-like accuracy.

  • Motivation: Emotion-conditioned generation is motivated by the difficulty of controlling deep music models to produce music with specific emotions.The surveyed discussion focuses on simple emotion categories.
  • Emotion control: An mLSTM controls polyphonic music emotion by optimizing neurons associated with emotional signals and classifying hidden-state representations.The approach inserts logistic regression into the mLSTM to classify emotions in symbolic music.
  • Evaluation: Listeners recognized the emotions of generated works as accurately as those of human-created music.This is the reported perceptual outcome of the emotion-controlled generation experiment.

4.9 Application

The survey frames music-generation systems as tools for human–machine cooperation, spanning interactive applications and creative assistance for both non-musicians and professionals.

  • Current generative music research should progress beyond modeling quality toward systems usable in actual creative scenarios.The paper presents convincing music generation as necessary before practical use, rather than as the ultimate goal.
  • Interactive applications include accompaniment, song creation, real-time performance, drum-pattern generation, collaborative editing, and emotion-matching composition.
  • Magenta and Flow Machine develop tools that assist musicians through melody, drumbeat, interpolation, and related creative workflows.
  • Commercial systems target video soundtracks, professional composition, films, businesses, games, and television, but some support only fixed styles or rule-based modifications.
  • These applications aim to encourage human–machine cooperation, make non-musicians creative, and increase musicians’ effectiveness.

5 Datasets

The survey organizes music-generation datasets by storage format, modality, and task suitability, while highlighting differences in scale, annotation, expressive content, and data quality.

  • Dataset choice is closely related to the generation task, with collections spanning folk, pop, jazz, MIDI, pianoroll, and audio formats.The survey notes that data quantity and dataset properties both affect deep-learning music generation.
  • MIDI stores descriptive note and instrument events rather than waveform data, making files compact and suitable for score-oriented analysis.
  • The JSB Chorales collection contains 402 four-part choruses but is small and lacks expressive information.
  • VGMIDI combines 95 sentiment-labeled piano pieces, 966 four-bar phrases, and 728 unlabeled pieces annotated using valence–arousal judgments from 30 subjects.
  • LMD contains 176,581 unique MIDI files, including 45,129 aligned with the Million Song Dataset, but spans heterogeneous genres, instruments, periods, and expressive characteristics.
  • The Lakh Pianoroll Dataset converts MIDI into multi-track pianorolls and addresses sparsity by merging tracks from similar instruments.

6 Evaluation

Evaluation of generated music combines model-level, music-specific, and human assessments because objective metrics alone do not reliably capture artistic quality or human judgment.

  • Objective metrics include model statistics, task-specific music measures, audio reconstruction and transfer measures, and singing-voice synthesis measures.Examples include loss, perplexity, BLEU, precision, recall, F1, RMSE, LSD, MCD, MMD, KNN, and F0-related measures.
  • Model metrics compare generated and original data distributions or classification behavior, but may measure data processing rather than the quality of musical generation.
  • Music metrics compare descriptive statistics of generated music with real or other generated music, using measures such as rhythm, pitch repetition, tonal distance, chord coverage, and harmonic consistency.
  • Subjective evaluation remains indispensable because quantitative music-quality measures have limited correlation with human judgment.Listening tests and expert analysis assess properties such as salient themes and styles that automated metrics may not capture.
  • Online listening tests can recruit many participants and included a 2,336-participant Bachbot discrimination test, but they cannot ensure that respondents complete tasks authentically or validly.

7 Challenges and Future Directions

Deep music generation has progressed rapidly, but challenges remain in structure, creativity, emotion, controllability, evaluation, representation, and expressive performance. The survey therefore identifies directions spanning longer and more controllable music, richer performance and audio synthesis, and broader cultural applications.

  • Challenges: Generated music still lacks long-term structure, coherent closure, innovation, emotional expression, and flexible user control.The survey identifies recurring weaknesses in themes and repetition, endings, extrapolative creativity, emotion modeling, and constraint handling.
  • Challenges: Evaluation remains difficult because researchers use diverse metrics, and objective scores may disagree with subjective judgments.The survey notes that results can differ across metrics and that strong quantitative performance may coexist with poor human evaluation.
  • Challenges: Audio representations trade off waveform flexibility against packet alignment, phase, spectrum-leakage, or data-efficiency challenges.Fourier-based approaches face alignment and leakage issues, whereas autoregressive waveform models avoid packet bias but require larger models and more data.
  • Future Directions: Future score-generation work should target long-term themes, self-repetition, controllable closure and form, innovation, style transfer, and emotion.The survey also proposes using generated music for emotional adjustment and practical applications.
  • Future Directions: Performance-generation research should model richer expressive details, use aligned score-performance datasets, support more instruments, and incorporate emotional expression.Suggested details include piano pedal and note-closing speed, extending beyond piano and drums.
  • Future Directions: Audio-generation research should condition synthesis on high-level musical information, model musical form, and make singing reflect lyric meaning more emotionally.The survey also points toward singing voice conversion with background music and virtual singers with varied timbres.
  • Future Directions: Future work should adapt representations, architectures, and hyperparameters for culturally specific genres such as Peking Opera and opera.Such research may support the inheritance of national culture even when broad commercial application is difficult.

8 Conclusion

The survey organizes deep music generation by three representation levels and reviews tasks, representations, evaluation methods, and datasets across them. It concludes that progress is substantial but generation remains limited by structural, creative, emotional, interactive, performance-modeling, and evaluation challenges.

  • Conclusion: The paper divides automatic music generation into score, performance, and audio levels, then reviews deep-learning tasks under each level.This task-centered organization is intended to help researchers locate relevant methods and improvement directions.
  • Conclusion: The survey additionally introduces music representations, evaluation methods, and commonly used datasets across the field.
  • Conclusion: Deep-learning music generation has progressed rapidly, but generated music remains far from expected quality.The conclusion highlights missing structure and innovation, limited expressive-performance modeling, difficult emotion capture, restricted interaction, and the absence of unified evaluation standards.
  • Conclusion: Proposed directions include reinforcement learning, unified evaluation metrics, emotion-aware generation, stronger human-AI interaction, and real-world applications.
Loading 2011.06801v1…