Source-linked AI summary
PianoKontext: Expressive Performance Rendering from Deadpan Context
Dmitrii Gavrilev
TL;DR
Expressive performance rendering must preserve note sequences while modeling timing that fixed-duration audio editing overlooks. PianoKontext uses DTW-aligned deadpan score contexts with latent flow matching to generate variable-length piano performances, outperforming an unsupervised bridge method in fidelity and content preservation.
Problem
Existing audio editing models mainly process synchronized samples of equal duration, limiting expressive timing, while audio models may omit or hallucinate notes.
Method
PianoKontext uses DTW-aligned deadpan MIDI-audio contexts and jointly models context and performance embeddings in DiT blocks for variable-length latent rendering.
Results
PianoKontext outperforms an unsupervised bridge method in audio fidelity and content preservation, with lower deviation from the score and controllable tempo variation.
Takeaways & Limitations
The model demonstrates controllable generation of variable-tempo piano performances while preserving musical structure, harmony, melody, and score content.
Takeaways & Limitations
This proof of concept is limited to piano, requires more rigorous musicality evaluation, and does not yet address full-length performances.
Abstract
from arXiv · showhide
Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes. However, flow matching audio editing models manipulate only synchronized music samples of the same duration, limiting their understanding of expressive timing. We introduce PianoKontext, a flow matching rendering model for classical piano music that generates variable-length performances in the latent space of a pretrained Music2Latent model. We synthesize MIDI scores into deadpan audio and employ Dynamic Time Warping (DTW) in the latent space to construct paired data for training. The aligned embeddings are concatenated in DiT blocks, allowing for a simple and effective learning of the dependencies between the score and performances. Audio samples are available at our demo page: https://realfolkcode.github.io/pianokontext_demo/.
1. Introduction
The introduction identifies same-duration constraints and score unfaithfulness as open problems in controllable music generation. PianoKontext addresses these challenges by rendering variable-length piano performance segments from deadpan latent context using a latent flow matching framework.
- Music-editing methods predominantly modify pairs of musical samples with the same duration, limiting controllable music generation.
- Audio models may hallucinate or omit notes, making them less faithful to the score than symbolic music models.
- PianoKontext is a latent flow matching model that renders variable-length piano performance segments from a deadpan latent context.
- Its framework models dependencies between context and targets with self-attention in Diffusion Transformer blocks, inspired by FLUX Kontext.
2. Related Work
Related work spans latent-space audio generation and two main approaches to expressive performance rendering: symbolic modeling of note-level expressive parameters and direct audio-domain synthesis. End-to-end latent- or audio-domain EPR remains underresearched, with RenderBox and GuitarFlow illustrating existing alternatives.
- Latent-Space Audio Generation: Latent audio generation models use either autoregressive decoding of neural-codec tokens or diffusion and flow matching over continuous autoencoder embeddings.MusicGen represents the autoregressive discrete-token paradigm, while other work models continuous embeddings.
- Latent-Space Audio Generation: Iterative latent denoising supports controllable generation, including timbre transfer through Music2Latent inversion to retrieve a source-audio semantic code.Classifier-free guidance is one technique enabling control during iterative denoising.
- Expressive Performance Rendering: Expressive performance rendering has two branches: modeling note-level MIDI attributes such as timing, velocity, and articulation, or synthesizing performances directly in audio.Symbolic models are limited because they do not represent an instrument’s or performance space’s acoustic properties.
- Expressive Performance Rendering: End-to-end EPR in latent or audio domains remains underresearched, with RenderBox conditioning a finetuned Stable Audio Open model on MIDI and text.RenderBox is presented as an example of latent- or audio-domain EPR.
- Expressive Performance Rendering: GuitarFlow uses synthesized deadpan audio instead of MIDI as input, avoiding the need to learn a symbolic music encoder.The passage introduces GuitarFlow as another end-to-end EPR example.
3. Method
PianoKontext formulates expressive performance rendering as conditional audio-to-audio translation in Music2Latent space and trains it with guided flow matching. The method aligns deadpan-score and performance latents with DTW, samples aligned variable-length segments, and conditions a DiT architecture using 2D positional embeddings.
- Flow matching: Flow matching continuously interpolates between noise and data distributions through intermediate marginals and an ODE induced by a learned velocity field.Because directly regressing the marginal velocity is intractable, the method regresses the conditional velocity field instead.
- Data pipeline: Music2Latent encodes score and performance audio into 64-dimensional latent sequences sampled at approximately 11 Hz, which DTW aligns for corresponding musical content.DTW is precomputed once, so it adds no computational overhead during training.
- Conditional rendering: Given deadpan latent context y, the model generates expressive latent performance x while preserving y’s musical content by learning p(x|y) with guided flow matching.Both context and performance are latent sequences from the pretrained Music2Latent model.
- Training data: Training samples random DTW subpaths whose segment lengths stay below the maximum sequence length S, with random minibatch lengths and a lower bound preventing extremely short samples.The sampled score and performance segments are temporally aligned.
- Architecture: The architecture uses DiT enriched with 2D RoPE and adds an axis separating context from performance elements in relative positional encoding.Element position is encoded as (i, s), where i ∈{0, 1} indicates context or performance.
4. Experiments
The experiments build a paired deadpan–expressive piano dataset, compare PianoKontext with CFG Bridge, and evaluate generated performances using audio-fidelity and alignment metrics.
- Dataset: The paired dataset combines expressive MAESTRO performances with deadpan audio synthesized from ASAP scores using a YDP Grand Piano soundfont.ASAP provides scores for a subset of MAESTRO performances, enabling paired deadpan–expressive data construction.
- Baselines: CFG Bridge is an unsupervised trajectory-inversion baseline trained conditionally on combined ASAP and MAESTRO data with deadpan and expressive labels.Source-label inversion and target expressive-label denoising are guided with classifier-free guidance.
- Implementation: PianoKontext and CFG Bridge use the same 8-block DiT architecture with hidden size 512 and MLP expansion ratio 1.PianoKontext halves CFG Bridge’s 16 attention heads to increase per-head dimensionality for an additional RoPE axis.
- Evaluation: The evaluation reports FAD, KAD, Pitch DTW, Alignment Precision, and Alignment Recall.FAD and KAD measure embedding-distribution discrepancies, while DTW chromagram similarity and alignment metrics assess correspondence between deadpan and generated performances.
- Generation: Five performances are generated per test-set score using 64 Heun ODE solver steps, with PianoKontext supporting varying latent lengths and CFG Bridge using fixed 11-second context.CFG Bridge uses guidance scales 1.0 for inversion and 2.0 for denoising.
5. Results and Discussion
PianoKontext outperforms CFG Bridge in audio fidelity and score-content preservation, while demonstrations show controllable tempo variation from deadpan context. The work is presented as a proof of concept, with future directions including broader instrumentation, more rigorous musicality evaluation, longer sequences, and outpainting.
- Evaluation: PianoKontext outperforms CFG Bridge in audio fidelity and content preservation.The reported measures are FAD, KAD, Pitch DTW, Alignment Precision, and Alignment Recall.
- Evaluation: Human-level Pitch DTW indicates preserved structure and harmony, while precision and recall suggest fewer score deviations than CFG Bridge.Human performances provide upper bounds because the transcription model is imperfect.
- Limitations and future work: The paper presents a proof of concept for expressive polyphonic music conditioned on deadpan context sequences.The authors identify broader instrumentation and more rigorous musicality evaluation as future directions.
- Limitations and future work: Extending sequence length and incorporating outpainting could enable generation of full-length performances.These are proposed extensions rather than evaluated capabilities in the presented work.