PaperlayerView exact revision on arXiv

Source-linked AI summary

CANTABILE: Learning Expressive Dynamics for Robotic Piano Performance

Woosik Kim, Wonhyeok Choi, Sunghoon Im

arXiv:2609.18213v1cs.RO

TL;DR

Robotic piano benchmarks largely measure pitch and onset, leaving expressive dynamics unrepresented. CANTABILE connects score intensity to embodied key motion and couples velocity fidelity with onset coverage. The framework improves dynamics on EXPRESSIVE-51, while its current scope remains limited to velocity and simulation.

  • Problem

    Standard robotic piano benchmarks measure key-press accuracy and timing, leaving musical dynamics neither rewarded nor evaluated.

    Source: §I · p. 1

  • Method

    CANTABILE anticipates score velocity goals, maps key motion at onset to MIDI velocity, couples velocity fidelity with onset coverage, and adapts a frozen base through a finger-only residual.

    Source: §I · p. 2, A. Score-to-Contact Dynamics Interface · p. 3, C. Structure-Preserving Residual Control · p. 4

  • Results

    CANTABILE yields substantially more faithful dynamics on EXPRESSIVE-51, and its intensity-conditioned policy follows a commanded intensity at run time.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The approach may extend expressive objectives beyond piano by exposing a sensing channel, defining an event-level reward, and attaching a residual to pretrained dexterous controllers.

    Source: §VI · p. 8

    The framework models velocity but not tempo or articulation, does not yet generalize across repertoire, and has only been evaluated in simulation.

    Source: §V · p. 8

Abstract

from arXiv · show

Robotic piano playing has emerged as a standard benchmark for dexterous bimanual manipulation, yet progress on it has been measured almost entirely by note accuracy -- which keys are pressed (pitch) and when (onset) -- leaving the musical dynamics essential for expressive performance neither rewarded nor evaluated. We propose CANTABILE, a dynamics-aware framework for robotic piano performance that (i) closes the score-to-contact loop by conditioning the policy on upcoming velocity goals and mapping each key's angular velocity at onset back to MIDI velocity, (ii) couples a velocity-fidelity reward with an onset-coverage reward, so that dynamics cannot be improved by omitting difficult notes, and (iii) refines a frozen dynamics-aware base policy with an alpha-scaled, finger-only residual that localizes strike-intensity adaptation away from nominal note execution. On EXPRESSIVE-51, a dynamics-rich 51-song subset of RoboPianist, CANTABILE raises Velocity F1 -- jointly measuring pitch, onset, and intensity within a +/-8 MIDI-velocity tolerance -- from 0.06 to 0.34 over the RoboPianist baseline, improves all 51 songs, more than halves matched-note velocity error, and reduces log-mel distance to reference audio by 8%. Intensity-randomized training further enables runtime control of performance intensity without retraining.

I. INTRODUCTION

Robotic piano benchmarks have advanced repertoire, transfer, and motion realism, but standard note-accuracy measures leave musical dynamics weakly represented. CANTABILE connects score-specified intensity to physical key motion and evaluates dynamics alongside note coverage.

I. INTRODUCTION

  • Standard evaluation scores pitch and onset but renders every detected keystroke at a fixed MIDI velocity, leaving impact-speed differences out of the objective.Piano loudness is strongly influenced by hammer impact speed.Source: §I · p. 1, Figure 1 · p. 1
  • CANTABILE measures realized intensity at note onset and couples velocity fidelity with onset coverage, so selective note omission cannot make dynamics appear accurate.Its score-to-contact interface anticipates velocity targets and maps key-onset motion to MIDI velocity.Source: §I · pp. 1–2
  • CANTABILE separates strike-intensity adaptation from nominal note execution through localized finger corrections to a frozen piano-playing policy.The framework conditions control on upcoming velocity goals and maps each key’s onset motion to MIDI velocity.Source: §I · p. 2
  • EXPRESSIVE-51 is a 51-piece suite selected for substantial variation in notated MIDI velocity, and Velocity F1 jointly assesses pitch, onset, and intensity.The work also identifies runtime intensity control through intensity randomization as a contribution.Source: §I · p. 2

II. RELATED WORK

Prior piano benchmarks and related expressive-performance work treat velocity differently: benchmark policies focus on key presses and timing, while rendering models predict symbolic velocity. CANTABILE brings velocity into embodied control through an interface, event objective, and residual controller.

II. RELATED WORK

  • RoboPianist and follow-up benchmark work evaluate which keys were pressed and when, while MIDI velocity appears in neither their reward nor evaluation metric.Concurrent work [16] adds score-velocity matching for single-hand policies; this paper targets bimanual repertoire-scale evaluation with onset accuracy.Source: §II · p. 2
  • Expressive rendering models treat velocity as a first-class symbolic output, whereas physical piano performance must realize velocity through motor commands and contact.ScorePerformer supplies matched-note event evaluation conventions adopted for the embodied setting.Source: §II · pp. 2–3
  • CANTABILE uses residual learning for a new objective—expressive dynamics—while a frozen base retains note accuracy, with α-scaled, finger-only constraints limiting interference.Prior piano residual approaches correct imitation errors or close the sim-to-real gap, [8].Source: §II · p. 3

III. METHOD

  • CANTABILE links score-specified intensity to physical realization with anticipatory targets and onset-motion sensing, a coverage-aware objective, and a finger-only residual controller.The three components are the score-to-contact interface, combined velocity-supervision and onset-coverage objective, and residual controller.Source: §III · p. 3

A. Score-to-Contact Dynamics Interface

The score-to-contact interface supplies upcoming velocity goals before a strike and measures realized key motion at onset. It converts that motion to MIDI velocity for rendering and training rewards.

A. Score-to-Contact Dynamics Interface

  • Each score onset carries a target MIDI velocity that encodes intended strike intensity, linking the score-level target to physical execution.The target is associated with the relevant key and control step.Source: A. Score-to-Contact Dynamics Interface · p. 3
  • Velocity-goal look-ahead adds upcoming intensity targets to the standard observation, allowing motion adjustment before contact; the chosen horizon is nvel=2.The look-ahead is the only feature added to the standard policy observation, and control remains closed-loop through robot and key states.Source: A. Score-to-Contact Dynamics Interface · p. 3
  • At an executed onset, the policy reads key angular speed and maps it to realized MIDI velocity, which is used for rendering and training reward computation.The mapping linearly interpolates between empirically selected onset-speed endpoints as a monotonic approximation.Source: A. Score-to-Contact Dynamics Interface · p. 3

B. Coverage-Aware Event Learning

CANTABILE addresses sparse velocity supervision by pairing matched-onset velocity fidelity with onset coverage. It also uses commanded velocities for anticipatory targets and reward, with runtime intensity changes available through the command.

B. Coverage-Aware Event Learning

  • Sparse velocity supervision can reward omission, so the method couples matched-onset velocity fidelity with a reward for realizing target onsets at the correct step.The complete reward retains the RoboPianist task reward and uses λvel=0.2 and λonset=0.5.Source: AND uj, ˙qj THE ACTUATOR TORQUE AND… · p. 4
  • Velocity agreement is measured only for matched onsets, using a Gaussian tolerance; without coverage, a policy can avoid velocity penalties by missing difficult notes.A well-matched onset receives positive reward, a poor match is penalized, and a step with no executed onset receives zero.Source: AND uj, ˙qj THE ACTUATOR TORQUE AND… · p. 4

AND uj, ˙qj THE ACTUATOR TORQUE AND JOINT VELOCITY. A CONSTANT

  • Commanded velocities feed both velocity-goal look-ahead and the velocity-fidelity reward; changing intensity command c at inference requests a different intensity without policy updates.Training can randomize a global intensity command c per episode.Source: AND uj, ˙qj THE ACTUATOR TORQUE AND… · p. 4

C. Structure-Preserving Residual Control

CANTABILE separates nominal note execution from strike-intensity refinement in a two-stage controller. A frozen base policy supplies the nominal action, while a bounded residual modifies only finger-action dimensions.

C. Structure-Preserving Residual Control

  • Jointly learning note execution and strike intensity could perturb acquired coordination, motivating a two-stage, coarse-to-fine controller.The first-stage policy is trained with the composite reward, then frozen as the base for residual learning.Source: C. Structure-Preserving Residual Control · p. 4
  • The residual is conditioned on the observation and base action, then applies an α-bounded correction only to finger-action dimensions.The base remains fixed while the residual alone is optimized with the composite reward; the structure also allows a frozen RoboPianist base.Source: C. Structure-Preserving Residual Control · p. 4

IV. EXPERIMENTS

The experiments assess CANTABILE against compute-matched baselines and examine its rewards, residual design, runtime intensity commands, and robustness.

IV. EXPERIMENTS

  • The evaluation compares dynamics fidelity and note accuracy against compute-matched baselines.The setup is described in Section IV-A and the comparison in Section IV-B.Source: §IV · p. 4
  • Ablations examine the event rewards and residual design.These are reported in Section IV-C.Source: §IV · p. 4
  • Further experiments evaluate generalization across runtime intensity commands and robustness to design and evaluation choices.These are reported in Sections IV-D and IV-E, respectively.Source: §IV · p. 4

A. Experimental Setup

The setup evaluates dynamics on EXPRESSIVE-51, defines metrics for note, velocity, joint timing-and-loudness, and audio similarity, and compares compute-matched methods.

A. Experimental Setup

  • EXPRESSIVE-51 retains 51 of RoboPianist’s 150 pieces with velocity-sequence standard deviation of at least 10, excluding pieces with notes at MIDI velocity ≤4.The selection targets meaningful dynamics variation and excludes notes considered effectively inaudible under the benchmark soundfont.Source: A. Experimental Setup · p. 4
  • Table II reports the main comparison on EXPRESSIVE-51, averaging 51 songs with one deterministic rollout per song and method; all methods are compute-matched at 11×10^6 steps, with relative changes against RoboPianist [3].Source: Table II · p. 5
  • Velocity MAE measures error over matched onsets only, while Velocity F1 jointly requires same-step score onset and velocity within τ units.At τ=8, Velocity F1 penalizes spurious keystrokes and missed score onsets through precision over keystrokes and recall over score onsets.Source: A. Experimental Setup · p. 5
  • Audio similarity is measured by log-mel spectrogram L2 distance and correlation between frame-wise RMS loudness curves in dB.Reference and performance are rendered through the same synthesizer and soundfont.Source: A. Experimental Setup · p. 5

B. Dynamics Realization and Note Accuracy

Across EXPRESSIVE-51, CANTABILE improves dynamics while preserving the comparison’s focus on note execution. The residual also outperforms end-to-end training at comparable note accuracy, while the dynamics-aware base is needed for the full improvement.

B. Dynamics Realization and Note Accuracy

  • The four compute-matched comparisons are RoboPianist, End-to-End, and residuals applied to either RoboPianist or the dynamics-aware πvel base.Aggregate results are reported over EXPRESSIVE-51.Source: B. Dynamics Realization and Note… · p. 5
  • Velocity F1 rises from 0.063 for RoboPianist to 0.342 with CANTABILE, which improves every song and reduces velocity MAE from 37.1 to 17.2.RoboPianist’s note F1 is 0.779, despite its low Velocity F1.Source: B. Dynamics Realization and Note… · p. 5
  • The residual beats end-to-end training at comparable note accuracy; on RoboPianist, it reaches Velocity F1 0.198 with almost no note-accuracy loss, but the dynamics-aware base is needed for the full gain.End-to-End and the residual on πvel are both substantially closer to the reference audio than RoboPianist, and comparable to each other.Source: B. Dynamics Realization and Note… · p. 5
  • Widening the residual beyond the fingers slows learning and converges to higher velocity error than the frozen-base reference.The figure reports Nocturne results over 5×10^6 residual steps and three seeds.Source: Figure 5 · p. 6

C. Ablation Studies

The ablations show that velocity fidelity and onset coverage address different failure modes, while restricting residual adaptation to the fingers supports faster learning and lower final velocity error.

C. Ablation Studies

  • Combining velocity and onset-coverage rewards achieves the highest Velocity F1; either reward alone leaves a complementary failure mode unresolved.Velocity-only supervision can yield low matched-onset error while covering few target events; onset coverage alone gives the highest onset F1 but no intensity supervision.Source: C. Ablation Studies · p. 6
  • Finger-only residuals reduce velocity error fastest and reach the lowest final error; wrist- or forearm-expanded residuals learn more slowly and have lower final Velocity F1.Final Velocity F1 is 0.41 for fingers only, 0.35 with the wrist, and 0.33 with the forearm; the base retains proximal coordination while the residual adapts finger motion.Source: C. Ablation Studies · p. 6
  • Figure 6 compares achieved intensity and Velocity F1 across commands, with both axes normalized to each song’s written loudness.The dashed diagonal marks exact realization; the reference is the same policy trained at fixed intensity.Source: Figure 6 · p. 6
  • Figure 7 shows commanded velocity against performed velocity and velocity distributions at the softest and loudest commands; the distributions shift but overlap.The command-specific means climb along the diagonal only under randomization.Source: Figure 7 · p. 6

D. Generalization Across Intensity Commands

Intensity randomization supports in-range runtime control, though the loudest command is only partly realized. Robustness tests identify settings for reward weights, residual scale, evaluation tolerance, and velocity look-ahead.

D. Generalization Across Intensity Commands

  • The randomized policy responds monotonically to intensity commands (slope +0.54), while the fixed-intensity reference stays nearly constant; note F1 is nearly unchanged across the tested range.Testing covers 0.5×–1.5× commands on four songs, matching the training support; extrapolation was not evaluated.Source: D. Generalization Across Intensity… · pp. 6–7
  • The softest-to-loudest commands shift the realized-velocity distribution by about one and a half notated dynamic levels, but the loudest command is only partially realized.The results demonstrate in-range generalization, not exact tracking or extrapolation beyond the training range.Source: D. Generalization Across Intensity… · p. 7

E. Robustness to Design and Evaluation Choices

  • Reward-weight screening exposes a trade-off: smaller velocity weights favor note accuracy but weaker dynamics, while the adopted (0.2, 0.5) setting leads on metrics counting every scored note.Raising λvel to 0.5 costs 0.06 note F1 and 0.14 onset F1; lowering λonset to 0.2 costs onset and Velocity F1 without buying note accuracy. The selected setting avoids relying on matched-onset MAE, which can improve when difficult notes are dropped.Source: E. Robustness to Design and Evaluation… · p. 7, Figure 8 · p. 7
  • Residual scales α≤0.2 form the useful range; at α=0.5, Velocity F1 is 0.341±0.027, while α=0.2 keeps note execution intact and is adopted.At α=0.2, note F1 is 0.793±0.004 and is the steadiest across seeds.Source: E. Robustness to Design and Evaluation… · p. 7, Figure 9 · p. 7
  • The ranking of all four methods is unchanged across Velocity F1 tolerances τ=4–15, including the reported τ=8 setting.The residual’s largest relative margin over End-to-End occurs at the strictest tolerance.Source: E. Robustness to Design and Evaluation… · p. 7, Table IV · p. 7
  • Velocity F1 is highest with a look-ahead of nvel=2; longer horizons do not help, and note accuracy remains unaffected.Overall, CANTABILE remains effective across moderate design and evaluation variations.Source: E. Robustness to Design and Evaluation… · p. 7, Figure 10 · p. 8

V. LIMITATIONS

The paper’s expressive modeling is limited to velocity, and its results are confined to simulation. The conclusion summarizes CANTABILE’s approach and suggests it may extend to other pretrained dexterous controllers.

V. LIMITATIONS

  • The model covers velocity alone, leaving tempo and articulation open; dynamics-module generalization to unseen repertoire remains unresolved, and all results are in simulation.On unseen songs, the distilled residual’s velocity gains came at the cost of missed notes; real-piano deployment remains untested.Source: §V · p. 8

VI. CONCLUSION

  • CANTABILE combines a score-to-contact interface, coverage-aware objectives, and a finger-only residual; the paper suggests this recipe may extend beyond piano to other pretrained dexterous controllers.The proposed extension is framed as a possibility, not an evaluated result.Source: §VI · p. 8
Loading 2609.18213v1…