Source-linked AI summary
Music transcription modelling and composition using deep learning
Bob L. Sturm, João Felipe Santos, Oded Ben-Tal, Iryna Korshunova
TL;DR
The paper addresses how deep learning can model music transcriptions in ways useful for composition. It trains character- and token-based LSTMs on approximately 23,000 ABC transcriptions and generates new material. The outputs reproduce several training-data conventions and assist composition, while still requiring human curation and further stylistic evaluation.
Problem
The paper examines how music transcription models can support composition within and outside the stylistic conventions represented in training data.
Method
The authors train character- and token-based LSTM networks on approximately 23,000 ABC transcriptions and sample their output to generate new transcriptions.
Results
Generated outputs reproduce training-data patterns including pitch distributions, tonic endings, and AABB structure, and the system assists composition with a composer's guidance.
Takeaways & Limitations
The systems produce musically meaningful and sometimes nearly session-ready transcriptions, but composers or curators remain needed to select, correct, or improve outputs.
Takeaways & Limitations
The work has not yet assessed the generated transcriptions through discussions with the thesession.org community about stylistic and performance conventions.
Abstract
from arXiv · showhide
We apply deep learning methods, specifically long short-term memory (LSTM) networks, to music transcription modelling and composition. We build and train LSTM networks using approximately 23,000 music transcriptions expressed with a high-level vocabulary (ABC notation), and use them to generate new transcriptions. Our practical aim is to create music transcription models useful in particular contexts of music composition. We present results from three perspectives: 1) at the population level, comparing descriptive statistics of the set of training transcriptions and generated transcriptions; 2) at the individual level, examining how a generated transcription reflects the conventions of a music practice in the training transcriptions (Celtic folk); 3) at the application level, using the system for idea generation in music composition. We make our datasets, software and sound examples open and available: \url{https://github.com/IraKorshunova/folk-rnn}.
1 Introduction
The paper applies deep learning to music modelling and composition, enabled by increasingly accessible data, computation, training methods, and reproducible tools. It develops LSTM-based transcription generators trained on homogeneous ABC notation data and evaluates them statistically, stylistically, and as composition aids.
- Motivation: Deep learning is increasingly applied to analysing, modelling, and generating music as data, software, and compatible hardware become more accessible.The paper situates its work within broader advances in deep learning and open experimentation.
- Approach: The models use three-layer LSTMs with 512 blocks per layer, trained on approximately 23,000 ABC transcriptions from stylistically homogeneous session music.The authors use ABC notation because it is high-level, available, and associated with Celtic, Morris, and related session practices.
- Approach: The system includes character-based and token-based generators that model preceding textual context and produce transcriptions resembling the training material.Character models condition on the previous 50 characters, whereas token models condition on all preceding transcription tokens.
- Evaluation: The evaluation compares training and generated statistics, examines stylistic conventions in Celtic folk music, and uses the system for composition outside those conventions.These perspectives address population-level behaviour, individual generated transcriptions, and practical composition use.
2 Background
The background explains how recurrent networks model sequences through feedback and why LSTMs are useful for music generation. Earlier music systems used varied encodings and generally smaller architectures, while prior work also exposed difficulties with long-term musical structure.
- Recurrent networks: Recurrent neural networks use feedback from previous inputs to predict sequential symbols, while deeper stacks can represent higher-level short- and long-term relationships.Training modifies network parameters to reduce prediction error, but basic RNNs can suffer exploding or vanishing gradients.
- LSTM motivation: LSTMs address basic recurrent-network training problems and have achieved success on sequential tasks including handwriting, speech recognition, and machine translation.The paper presents these developments as motivation for applying recurrent models to music.
- Earlier music models: Earlier music models encoded pitch, duration, harmony, meter, key, and downbeats using representations ranging from local note units to distributed features.These systems generated melodies or accompaniments at note-level or quantised time steps.
- Earlier music models: Prior work applied evolutionary constraints to melody generation and recurrent models to long-term Irish folk conventions, including metrically related historical inputs.The examples show different strategies for controlling style, rhythm, pitch, and temporal context.
- Open challenge: RNN-generated polyphonic music in earlier studies was reported to lack long-term structure, motivating further work on music transcription modelling and generation.The paper frames this limitation alongside the varied encodings and architectures of prior systems.
3 Creating our generative LSTM networks
The authors construct two LSTM generators over ABC transcriptions: a character model trained on continuous text and a token model trained on complete transcriptions. They preprocess a large folk-music repository, use shared deep architectures, and generate sequences by sampling model output.
- Model types: The two systems are char-rnn over single characters in continuous text and folk-rnn over transcription tokens in complete transcriptions.Both use LSTMs but differ in vocabulary and training organisation.
- Training data: The source repository contains ABC transcriptions from traditional music sessions, including Celtic and Morris material, with varied formatting and occasional multiple voices.Entries may include ornamentation, grace notes, slurs, chords, missing bar lines, redundant accidentals, and miscounted measures.
- Training data: The char-rnn corpus retains five ABC fields and produces a 13,515,723-character text file with 135 unique characters.The retained fields are title, meter, key, unit note length, and transcription.
- Training data: The folk-rnn preprocessing transposes 23,636 transcriptions to C and represents seven token types, including meter, key, measure, pitch, grouping, and duration.The resulting corpus contains 4,056,459 tokens and 137 unique vocabulary tokens.
- Architecture and training: Each LSTM has three hidden layers of 512 blocks, one-hot vocabulary inputs and outputs, dropout, gradient clipping, and 100 training epochs.Both models use 95% of the data for training and 5% for validation, while folk-rnn batches parsed transcriptions using bucketing for variable lengths.
- Data limitations: The preprocessing leaves documented transcription errors uncorrected, including incorrect measure lengths, unpaired repeat signs, and incorrect variant endings.The reported errors include 78,338 incorrect-length measures, 4,761 unpaired repeat signs, and 3,057 incorrect variant endings.
- Generation: The models generate transcriptions by repeatedly sampling from the output probability distribution, optionally after initialising hidden state with a valid seed sequence.Each sampled character or token becomes the next input for a chosen number of timesteps.
4 Demonstrations of our generative LSTM networks
The generative LSTM systems produce transcriptions that capture several statistical and stylistic conventions of the training music, while also supporting iterative composition and idea generation. Their outputs include plausible musical structures, but also contain errors and passages requiring compositional revision.
- Statistical analysis of outputs: 6,101 folk-rnn transcriptions had meter and mode proportions close to the training dataset, while token-length distributions did not match and ending-pitch distributions appeared to match.About 68% of generated transcriptions used an AABB structure with two 8-bar sections, compared with 54% of training transcriptions.
- Statistical analysis of outputs: The generated outputs included formatting errors: 16 repeated |1 first endings, 6 incomplete ending specifications, and 3 incompletely specified chords.The chord errors involved a closing bracket without its accompanying opening bracket; these problems had been corrected in the training data.
- Musical analysis of outputs: “The Mal’s Copporim” was judged nearly session-ready, with correct measure durations, repeated 8-bar tune and turn sections, melodic repetition and variation, and implied harmonic structure.The analysis also identified an unbalanced repeat sign, awkward harmonic moments, and aimless or weakly prepared passages for revision.
- Musical analysis of outputs: The analysis found that characteristic passages in “The Mal’s Copporim” were not simply copied from the training transcriptions, including its ending measure.Some related patterns occurred in training examples, but often in different harmonic or melodic contexts.
- Music composition with the generative systems: The composition workflow seeded the char-rnn system with a short idea, selected and edited outputs, and reseeded it with expanded material.This process produced “The March of Deep Learning,” including variation, modulation to the V of V, and a transposed reprise, sounding different from the training music.
5 Discussion and reflection
The systems model ABC folk transcriptions and can produce musically meaningful material, including outputs that support composition beyond the training style. Their usefulness remains bounded by subjective curation, expert assessment, and limited evaluation of transcription modelling.
- Discussion and reflection: A char-rnn-assisted composition, “The March of Deep Learning,” sounds quite different from the training transcriptions.The figure documents use of the system for composition outside the training material’s stylistic conventions.
- Discussion and reflection: The study evaluates generated material through population statistics, individual transcription analysis, and assisted composition.These perspectives cover dataset-level modelling, stylistic and musical assessment, and practical idea generation.
- Discussion and reflection: The folk-rnn statistics match training-data pitch distributions and learn measure counting, tonic endings, and repeated 8-bar AABB structures.These observations concern production of valid ABC output and do not by themselves establish compositional quality.
- Discussion and reflection: The systems generate transcriptions showing repetition, variation, melodic contour, structure, progression, and resolution, although output quality varies.Many outputs appear plausible, but others are much less plausible and require human selection or revision.
- Discussion and reflection: The systems assist composition within restricted workflows, helping generate material in directions composers might not otherwise explore.The composer guides the process, including when the input departs from stylistic conventions in the training data.
- Discussion and reflection: The reported work has examined the ability of deep-learning methods to model ABC transcriptions, with stylistic evaluation and user exploration left for further work.Planned next steps include discussion with session musicians, expert interviews, and a user interface for exploring the system.
6 Conclusion
The work extends music modelling with substantially larger LSTM systems and datasets, exploring their usefulness for composition within and beyond stylistic conventions. The authors find plausible variation and meaningful transformations, while emphasizing that composers remain responsible for creative judgment.
- The models scale prior music-generation work to thousands of units, tens of thousands of training examples, and tens of thousands of generated transcriptions.Earlier work used a few dozen units, hundreds of examples, and only a few example sequences.
- The authors evaluate generated transcriptions through statistical comparison, stylistic convention analysis, and composition-oriented critical perspectives.These perspectives address composition within and outside particular musical conventions.
- The generated transcriptions show consistency in plausibility and meaningful variation, including transformations of transcribed musical ideas.The authors describe this balance as relevant to developing a composer’s inner monitor and style.
- The system may support traditional Celtic composition by recombining familiar elements, but curation, composition, performance, and rule-breaking remain the composer’s responsibility.The authors explicitly reject treating the models as substitutes for human creativity.