Source-linked AI summary
This Time with Feeling: Learning Expressive Musical Performance
Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, Karen Simonyan
TL;DR
Music-generation systems have often separated score creation from performance, even though expressive timing and dynamics are central to heard music. This paper generates direct MIDI performances with notes and expressive information jointly, using an LSTM-based recurrent model and performance-oriented data. The resulting samples show strong local musical structure, while professional feedback identifies limited long-term structure and inconsistent stylistic continuity.
Problem
The paper addresses the gap between score-based music generation and expressive performance generation, where timing and dynamics materially shape the heard result.
Method
The authors directly generate MIDI performances by predicting notes together with expressive timing and dynamics, using a representation suited to polyphonic events and an LSTM-based recurrent model.
Results
The system generates audio with strong local structure in phrasing and dynamics, and several professional musicians treated or questioned some samples as human performances.
Takeaways & Limitations
Direct performance generation is presented as an effective space for modeling expressive musical timing and dynamics together with notes.
Takeaways & Limitations
Professional feedback indicates that the generated music lacks long-term structure and can shift inconsistently across rhythmic structures, genres, and styles.
Abstract
from arXiv · showhide
Music generation has generally been focused on either creating scores or interpreting them. We discuss differences between these two problems and propose that, in fact, it may be valuable to work in the space of direct $\it performance$ generation: jointly predicting the notes $\it and$ $\it also$ their expressive timing and dynamics. We consider the significance and qualities of the data set needed for this. Having identified both a problem domain and characteristics of an appropriate data set, we show an LSTM-based recurrent network model that subjectively performs quite well on this task. Critically, we provide generated examples. We also include feedback from professional composers and musicians about some of these examples.
Preamble/Request
The paper asks readers to listen to linked audio because its motivation, data, results, and conclusions are most effectively understood perceptually.
- Preamble/Request: The authors argue that the paper’s actual results are most effectively perceived in the audio domain.They request that readers listen to the linked examples for context.
1 Introduction
The paper proposes generating improvised performances directly, jointly modeling musical notes with expressive timing and dynamics. It motivates this choice by showing that scores omit performance details that strongly affect musical perception, while MIDI performances encode richer timing and loudness information.
- 1 Introduction: The central thesis is to generate expressive timing and dynamics concurrently with musical notes.The paper frames this as direct generation of improvised performances rather than creating or interpreting scores.
- 1.1.1 Scores: Scores specify relative note timing and some dynamics, but performers routinely vary timing, loudness, and articulation beyond those markings.Examples include rubato, swing, unequal note dynamics, and voice-leading articulation in polyphonic piano music.
- 1.1 Scores, Performances and Musical Abstraction: A score is a highly abstract representation whose realization depends on conventions, traditions, and individual creativity.Chopin’s rubato and jazz lead-sheet swing illustrate information that is not fully written into the score.
- 1.1.2 MIDI: MIDI encodes note timing, duration, and velocity, but not timbre, and can represent performances as piano-roll events over time and pitch.Velocity is a discrete loudness-related value, while piano-roll rectangles encode note events and durations.
- 1.1.2 MIDI: The paper distinguishes MIDI Scores, rendered on the metrical grid without dynamics, from MIDI Performances, encoded from performed music.This distinction separates direct score rendering from expressive human performance data.
2 Factoring the Music Generation Process: Related Work
Prior work spans composition, expressive-performance modeling, and sound rendering, but existing systems often rely on rules, restricted representations, or separate prediction of expressive dimensions. The paper positions its approach within this pipeline while targeting direct performance generation.
- 2 Factoring the Music Generation Process: Related Work: The music-generation pipeline can be factored into composition, performance, sound rendering, and perception.The figure uses a score, MIDI piano roll, synthesizer-rendered audio, and human listening as successive stages.
- Music Generation: Existing music-generation research includes deterministic rule systems, random or aleatory procedures, and computational approaches to musical perception, interpretation, and generation.The paper situates its work within a long history spanning formal rules, chance, and AI-based methods.
- Expressive Musical Performance: Expressive Musical Performance systems model the second pipeline stage, commonly using hand-engineered features, rules, or conventional statistical and neural methods.The surveyed literature includes linear models, PCA, regression, KCCA, performance rules, and neural networks.
- Expressive Musical Performance: Earlier learned systems often predict expressive dynamics from micro-timing rather than jointly predicting timing and dynamics.The paper contrasts those systems with its joint prediction objective.
- Expressive Musical Performance: Prior performance-rendering systems frequently use complex heuristic rules, motivating approaches that learn parameters or expressive outputs more directly.Gaussian-process and other learned approaches were proposed partly to reduce rule-generation and parameter-optimization difficulties.
3 Choosing Assumptions and a Problem Domain
The paper chooses direct performance generation to jointly produce musical notes with expressive timing and dynamics, avoiding score-to-performance separation. Its representation remains MIDI-grounded while allowing polyphony, but long-term structure and evaluation remain difficult.
- 3.1 Assumptions: The model uses absolute time intervals rounded to 8ms instead of metric units that can omit rubato and swing.
- 3.1 Assumptions: Notes are specified one at a time while allowing arbitrary numbers of simultaneous notes, avoiding assumptions about chords, voices, or shared tempo variation.
- 3.1 Assumptions: The approach uses MIDI information—velocity, timing, and duration—without requiring time signatures, voice leading, or inferred chords.
- 3.2 Problem Domain: Simultaneously Composing and Performing: Direct performance generation jointly predicts composition and expressive timing and dynamics, rather than generating scores that require separate performance.The generated performance can be rendered to audio through a piano sample library.
- 3.2 Problem Domain: Simultaneously Composing and Performing: Long-term coherent structure remains elusive, and current symbolic MIDI systems do not effectively learn structure spanning many minutes.
- 3.2 Problem Domain: Simultaneously Composing and Performing: Generated-music metrics are limited, making listening-based evaluation necessary but difficult for score-generation systems because composition and performance quality become entangled.
4 Data
The study uses homogeneous piano-performance data designed to capture expert human expression consistently. This choice supports coherent modeling and feasible audio synthesis from MIDI.
- 4 Data: The dataset contains MIDI captures of roughly 1400 performances by skilled pianists playing a Disklavier.The Disklavier records MIDI events corresponding to performers’ actions through internal sensors.
- 4 Data: All recordings are classical solo instrumental music, consistently performed on piano.The homogeneous material is intended to improve output coherence and match the solo-instrument generation setting.
- 4 Data: Every performance was produced by humans rather than synthesized or hand-synthesized examples.
- 4 Data: The performers were experts because the system’s learned human performance should match listeners’ concept of expert playing.
- 4 Data: Piano is advantageous because synthesizing piano audio from MIDI can be done quite well, unlike some instruments and vocals.
5 RNN Model
The model generates expressive piano performances directly as event sequences, using an LSTM RNN and a temporally efficient representation that preserves timing detail. The system includes note, timing, velocity, and pedal-related events, with training augmented through pitch and time transformations.
- Model architecture: The performance data is modeled with a three-layer LSTM recurrent network, with 512 cells per layer.The network was not particularly sensitive to this layer-size hyperparameter.
- Event representation: The 413-event vocabulary contains pitch-specific NOTE-ON and NOTE-OFF commands, 125 TIME-SHIFT events, and 32 VELOCITY events.Velocity events apply to subsequent notes until another velocity event occurs.
- Event representation: Fine-grained 8 ms quantization combined with skippable time shifts preserves expressive timing while substantially shortening sequences.Consecutive shifts can represent intervals longer than the maximum single shift of 1 second.
- Predicting pedal: Including sustain pedal information produced a significant subjective improvement in output quality and helped predict grouped note releases and their preceding delays.Pedal was represented by extending note lengths and delaying corresponding NOTE-OFF events.
6 Results
Generated performances are evaluated through audio examples, informal expert reactions, and log-loss comparisons, revealing expressive and structurally coherent moments alongside weaknesses in long-term compositional consistency.
- 6.1 Examples: Generated clips include jointly produced note timing, duration, and velocity, then use piano samples for audio synthesis.This makes the model’s output directly listenable as a performance rather than only as an abstract score.
- 6.1 Examples: RNN Sample 4 combines a natural G-minor progression, a tempo shift, and inner-voice emphasis through left-hand dynamics.The excerpt moves from a slower segment through the dominant into a faster section beginning with G major.
- 6.1 Examples: RNN Sample 7 exhibits appropriate rubato, phrase swells, and pauses before isolated left-hand notes without memorizing the apparent Schubert style.Its phrasing includes a beginning-of-phrase rubato and slight pauses near 10 and 12.5 seconds.
- 6.2 Log-likelihood: Objective evaluation is fundamentally difficult, and log-loss comparisons can be misleading because variants differ in prediction difficulty.RNN-NV achieved the best log-loss while avoiding velocity prediction, making that variant inherently easier.
- Composer & Professional Musician: Listeners also identified inconsistent rhythmic structure, genre, and style, while describing long-term compositional structure as a continuing challenge.Sample 4 was judged the most stylistically consistent, whereas other samples mixed influences such as Mozart, Walton, Scott Joplin, Mahler, and free jazz.
- Composer & Professional Musician: Professional listeners consistently perceived a human quality, with one reporting greater expressiveness than playback features used in composition software.The feedback was informal and explicitly did not constitute a Turing test.
7 Conclusion
The paper proposes generating music directly in the space of MIDI performances and demonstrates a system that does so effectively. Professional feedback suggests strong local musical structure but limited long-term structure, with some samples mistaken for human performances.
- The authors propose MIDI performance generation as an effective space for generating music.
- The system demonstrates this approach using a data set designed for effective MIDI performance generation.
- Professional composers and musicians found strong local structure, including phrasing and dynamics, but not yet strong long-term structure.
- Several musicians assumed or asked whether generated samples had been performed by a human.