PaperlayerView exact revision on arXiv

Source-linked AI summary

CPR: Combining global composing, local performing and full-sequence refining in piano rendering with continuous autoregressive modelling

Chong Jing, Junan Zhang, Zhizheng Wu

arXiv:2609.18216v2cs.SDcs.AI

TL;DR

Prompt-conditioned piano rendering must follow target MIDI while matching a reference recording’s timbre, but existing autoregressive and flow-matching approaches have distinct representation and semantic-structure limitations. CPR combines continuous autoregressive prediction with local flow matching and full-sequence refinement, using BREPA and MT-RoPE for semantic and temporal alignment. The authors report state-of-the-art piano MIDI-to-Music performance, with gains over MIDI-VALLE across three metrics and a mixed comparison with P-MUSE.

  • Problem

    Piano MIDI-to-Music rendering must follow target notes while preserving reference timbre, but discrete-codec autoregression can lose acoustic detail and flow matching can lack semantic structure.

    Source: §1 · p. 1

  • Method

    CPR combines continuous autoregressive Composer hidden states, local flow-matching Performer patch generation, and a Refiner that upsamples 24 kHz audio to 48 kHz, with BREPA and MT-RoPE for semantic and temporal alignment.

    Source: §1 · p. 1, §2.2.1 · p. 2, §2.2.2 · p. 2

  • Results

    CPR outperforms MIDI-VALLE on all three evaluation metrics; versus P-MUSE, it has 2.7 percentage points higher onset F1 and the lowest FADclap, but slightly lower timbre similarity.

    Source: §3.4 · p. 4

  • Takeaways & Limitations

    The authors conclude that CPR combines global autoregressive modeling, local flow matching, and full-sequence refinement, with BREPA and MT-RoPE enhancing semantic and cross-modal temporal alignment.

    Source: §4 · p. 4

Abstract

from arXiv · show

Prompt-conditioned piano MIDI-to-Music rendering aims to faithfully render target notes while reproducing the timbre of a reference recording. Existing approaches primarily follow two paradigms: autoregressive (AR) modeling and flow matching (or diffusion). Discrete-codec AR models provide causal temporal modeling, but quantization can discard acoustic detail. Flow matching better preserves acoustic structure in the cost of full-sequence attention costs and worse semantic structure. Continuous autoregressive models operate directly on continuous representations. It not only combines the condition-following ability of AR models and distribution-modeling capacity of flow matching but also bypasses the quantization bottleneck with lower computational costs. Building on this principle, we present Composer--Performer--Refiner (CPR) framework. Composer autoregressively predicts continuous hidden states, Performer generates 24kHz acoustic latents through local flow matching and Refiner then upsamples the waveform to 48 kHz. We further introduce Bottlenecked Representation Alignment (BREPA) and Modality--Time RoPE (MT-RoPE) to strengthen musical semantic structure in Composer hidden states and temporal alignments across modalities. Codes are available at https://github.com/FEAfeatherTHER/CPR_official

1. INTRODUCTION

Prompt-conditioned piano rendering must follow target MIDI while transferring a reference recording’s timbre and expressive performance. CPR combines continuous autoregressive modeling with local flow matching, adding BREPA and MT-RoPE to strengthen semantic structure and cross-modal temporal alignment.

1. INTRODUCTION

  • Prompt-conditioned piano MIDI-to-Music rendering must follow target note onsets and dynamics, preserve reference timbre, and produce a musically expressive performance.The task uses prompt audio, its aligned MIDI, and target MIDI.Source: §1 · p. 1
  • Discrete-codec autoregressive models face quantization limits, while full-sequence flow matching captures acoustic detail but can lack naturalness, musicality, and semantic structure.Prior work is cited as showing that pure flow-matching supervision struggles to recover semantic structure without auxiliary alignment objectives.Source: §1 · p. 1
  • CPR jointly trains Composer–Performer as a continuous autoregressive model on 24 kHz audio, then trains a separate Refiner to upsample generated waveforms to 48 kHz.Source: §1 · p. 1, Figure 1 · p. 2
  • BREPA aligns compressed Composer hidden-state projections with self-supervised features to strengthen musical structure while retaining capacity for acoustic detail.The framework introduces BREPA as auxiliary semantic supervision because flow matching does not directly supervise Composer hidden states.Source: §1 · p. 1
  • CPR achieves state-of-the-art performance on the piano MIDI-to-Music generation task.The framework combines complementary strengths of autoregressive and flow-matching modeling.Source: §1 · p. 1
  • MT-RoPE adapts rotary position embeddings along modality and time axes to improve temporal alignment between audio and MIDI.It is introduced to enhance condition following.Source: §1 · pp. 1–2

2. METHOD

The method groups continuous acoustic tokens into patches, predicts patch-level hidden states autoregressively, and generates acoustic patches with local flow matching. Its stages also encode MIDI-audio timing, condition patch generation on prompt and history, refine 24 kHz audio to 48 kHz, and align Composer states with self-supervised features.

2.1. Formulation

  • CPR groups P adjacent continuous tokens into a patch and uses a local flow-matching head to model the current patch distribution.This replaces token-by-token regression and a discrete-token classification head.Source: §2.1 · p. 2

2.2.1. Composer–Performer–Refiner

  • Composer autoregressively predicts continuous acoustic-patch hidden states, while Performer generates the next patch with local bidirectional flow matching conditioned on CLAP, Composer states, and history.Five 20-ms mel frames form a 100-ms patch; the separately trained Vocos-based Refiner upsamples generated 24 kHz waveforms to 48 kHz.Source: §2.2.1 · p. 2

2.2.2. Composer

  • MT-RoPE allocates half the rotary dimensions to modality identity and half to temporal position, indexing MIDI and audio independently on a common physical-time axis.MIDI frame r at 25 Hz and audio patch i at 10 Hz use temporal coordinates 10/25r and i, respectively.Source: §2.2.2 · pp. 2–3

2.2.3. Performer

  • Performer models a linear path between clean data and standard Gaussian noise, conditioning on clean historical context and concatenated CLAP, Composer-state, and mel features.CLAP provides a global timbre prior; hierarchical condition dropping is used to encourage timbre cloning through history and Composer states.Source: §2.2.3 · p. 3

2.2.4. Refiner

  • The Refiner converts reconstructed 24 kHz audio to 48 kHz to compensate for missing high-frequency harmonic components.The continuous autoregressive model is trained at 24 kHz because 48 kHz latents contain too much high-frequency information for semantic modeling.Source: §2.2.4 · p. 3

2.2.5. Bottlenecked Representation Alignment

  • BREPA upsamples Composer hidden states from 10 Hz to 25 Hz through a 1024-to-768 bottleneck and aligns them with MuQ features using cosine similarity loss.The bottleneck is intended to prioritize semantic structure while leaving remaining dimensions available for residual acoustic detail.Source: §2.2.5 · p. 3

3. EXPERIMENTS

Experiments evaluate CPR on paired-prompt piano generation, comparing objective and subjective outcomes against baselines and testing BREPA and MT-RoPE through controlled ablations. CPR improves MIDI adherence and overall generation quality relative to the reported baselines, with a timbre-similarity trade-off for P-MUSE and Refiner.

3.1. Dataset

  • P-MUSE-eval contains 100 paired-prompt piano-family samples; training combines MAESTRO, Slakh, and rendered Lakh MIDI into about 5,000 hours of 3–30-second clips.The test samples cover diverse piano-family timbres.Source: §3.1 · p. 3

3.2. Implementation Details

  • Composer and Performer are jointly trained; Refiner is separately pretrained and fine-tuned, and inference samples each patch with four Euler steps before decoding to 24 kHz and refining to 48 kHz.The Refiner uses random bandwidth degradation with full-band audio targets during training.Source: §3.2 · p. 3

3.3. Metrics

  • The evaluation reports Sim, onset F1, and FADclap for timbre preservation, MIDI adherence, and overall generation quality, plus MOS for timbre similarity, MIDI accuracy, and naturalness.Onset F1 matches pitch and onset within 50 ms; offsets are not considered.Source: §3.3 · pp. 3–4

3.4. Results

  • CPR beats MIDI-VALLE on all three metrics; versus P-MUSE it gains 2.7 onset-F1 points, has slightly lower timbre similarity, and achieves the lowest FADclap.The objective evaluation uses P-MUSE-eval.Source: Table 1 · p. 4, §3.4 · p. 4
  • NFE performance peaks near 4; subjective results show Refiner raises MIDI accuracy and naturalness but reduces timbre similarity, and cannot restore detail lost during low-resolution generation.The subjective evaluation used 14 samples and 13 listeners; the study leaves improved Refiner recovery for future work.Source: §3.4 · p. 4, Table 2 · p. 4

3.5. Ablations

  • Training with REPA improves onset F1 and FADclap, the bottleneck adds gains, and replacing MT-RoPE with standard RoPE causes larger degradation despite BREPA.The ablations change one module at a time while holding other settings fixed.Source: §3.5 · p. 4, Table 3 · p. 4

4. CONCLUSION

CPR combines global autoregressive modeling, local flow matching, and full-sequence refinement. BREPA and MT-RoPE enhance semantic structure and cross-modal temporal alignment, while future work includes higher-rate refinement, continuous representations, and richer conditioning.

4. CONCLUSION

  • CPR combines global autoregressive modeling, local flow matching, and full-sequence refinement.Source: §4 · p. 4
  • BREPA and MT-RoPE enhance semantic structure and cross-modal temporal alignment.Source: §4 · p. 4
  • Future work may explore low-resolution observations for efficient higher-rate refinement, continuous autoregressive music representations, and richer conditioning for instruction following.Source: §4 · p. 4

5. COMPLIANCE WITH ETHICAL STANDARDS

Listeners voluntarily participated in the subjective listening evaluation, provided informed consent, and had their ratings collected anonymously.

5. COMPLIANCE WITH ETHICAL STANDARDS

  • Listeners voluntarily participated in the subjective listening evaluation.Source: §5 · p. 5
  • Listeners provided informed consent before the evaluation.Source: §5 · p. 5
  • Ratings were collected anonymously.Source: §5 · p. 5

Related papers

Loading 2609.18216v2…