PaperlayerView exact revision on arXiv

Source-linked AI summary

Expressive Robotic Pianist: Mastering Complex Piano Repertoire with Graph-Mimic and Musical Dynamics

Yanhong Liang, Xianwei Liu, Chaojie Fu, Shaowen Cheng, Yanyan Yuan, Chengwei Zhuo, Xi Chen, Yongbin Jin, Wei Yang, Hongtao Wang

arXiv:2609.10844v1cs.RO

TL;DR

Robotic pianists still face difficulty reproducing human-like fingering and expressive dynamics. This paper combines graph-based hand-motion planning with acoustic dynamics control; on simpler pieces, its F1 was 0.92 versus previous approaches’ approximately 0.60 average (45).

  • Problem

    Robotic hands still lack musically aware strategies for matching the expressive artistry and dexterous movement required for piano performance.

    Source: Introduction · p. 2

  • Method

    Graph-Mimic uses a unified graph representation to learn pre-press fingering coordination and key-press postures, paired with a physics-inspired acoustic model for score-consistent dynamics.

    Source: Piano acoustics model for tuning… · p. 6, Discussion · p. 9

  • Results

    F1 was 0.92 on simpler pieces, versus previous approaches’ average of approximately 0.60 (45), while real-world trials across styles remained highly accurate.

    Source: Experimental validation across diverse… · p. 8, Discussion · p. 9

  • Takeaways & Limitations

    The system demonstrates execution of technically demanding piano repertoire, including Grade 7 works such as Croatian Rhapsody.

    Source: Discussion · p. 9

    The evaluation still lacks a holistic measure of expressive piano performance, since F1 assesses pitch–onset accuracy but not musical expressivity well.

    Source: Limitations and future work · p. 10

Abstract

from arXiv · show

Enabling robots to perform musical instruments with human-level expressivity represents a frontier in bridging the gap between mechanical precision and artistic interpretation. Despite advances in robotic dexterity, replicating the fluid finger transitions and nuanced dynamic control characteristic of human pianists remains a significant challenge. Through a reinforcement learning-based control framework, we demonstrate that a dexterous robotic hand can achieve high-fidelity performance across a diverse piano repertoire. Central to our approach is a graph-based optimization strategy that guides the robot to generate natural pre-press and key-press fingering strategies that closely resemble human movement patterns. To achieve expressive sound production, the control system is coupled with a physics-inspired acoustic model that modulates keypress velocity to accurately reproduce the dynamic variations specified in musical scores. Quantitative evaluations demonstrate that our expressive control model significantly outperforms baseline methods in both finger morphology similarity and dynamic velocity accuracy. In a perceptual test involving participants from diverse listener groups, performances generated by our system are significantly preferred over baseline robotic performances and are indistinguishable from human performances for non-professional audiences. Furthermore, extensive experiments across multiple musical styles confirm that our method maintains high note-level accuracy while achieving expressive performance. Our approach provides a robust pathway for robotic systems to move beyond mere mechanical accuracy, elevating robotic musicianship to a level of expressive performance comparable to human pianists.

Introduction

The paper introduces Graph-Mimic and Musical Dynamics to support expressive robotic piano performance. Across varied repertoire, general listeners rated the system comparably to skilled human pianists, with no statistically significant preference between them.

Introduction

  • Expressive piano performance requires fluid pre-press fingering, articulation-specific key touch, and precise dynamic control; the work demonstrates robotic performance of piano literature at Grade 7 standard.The introduction characterizes robotic artistry as difficult because of mechanical constraints and the lack of musically aware control strategies.Source: Introduction · p. 2

Related work

  • Prior robotic systems often emphasize correct notes and timing with fixed postures, while existing RL formulations underrepresent musical dynamics (29) and leave fingering coordination, articulation, and expressive control unresolved.The paper frames pre-press coordination as a kinematic planning problem, key-press behavior as contact dynamics, and loudness as requiring precise contact-velocity control.Source: Related work · pp. 2–3

Contributions

  • Graph-Mimic aligns human and robotic hand motion through spatial relationship graphs, using inter-fingertip vectors for pre-press dexterity and phalangeal vectors for key-press contact configuration.Its Action Frame Graph avoids strict joint-to-joint correspondence and targets articulation-related effective stiffness during key presses.Source: Contributions · p. 3
  • The simplified piano acoustics model links key angular velocity to note volume, letting the robot learn when and how fast to press keys without directly simulating string vibration.The energy-transfer approach is intended to avoid inaudible or excessively forceful strikes.Source: Contributions · pp. 3–4
  • In a varied repertoire, the InReal hand produced musically appropriate fingering and nuanced loudness modulation; general listeners showed no statistically significant preference between its performances and skilled human pianists'.The authors report expressivity comparable to human performance in their perceptual evaluation.Source: Contributions · p. 4

Results

Across piano repertoire, Graph-Mimic supported natural finger motions, while acoustic-model-based dynamics guidance improved alignment with score dynamics. Listener rankings and note-level accuracy showed strong performance, with results varying by listener group and execution setting.

Fingering techniques during pre-press phase

  • During pre-press fingering, Graph Distance and End-Effector Distance showed diagonal similarity-matrix alignment for thumb-under motions, unlike Joint Angle Error.Graph Distance encodes inter-fingertip relationships, supporting consecutive fingering, hand-span adjustment, thumb-under, and finger-over strategies; joint-angle differences are sensitive to morphology and can poorly represent task-space similarity.Source: Fingering techniques during pre-press… · pp. 4–5

Articulation-specific touch techniques during key-press phase

  • Graph-Mimic produced vertical fingertip contact for staccato and overlapping, stable-pad contact for legato, while guiding more natural finger trajectories than the baseline.The baseline often used abnormal PIP extension during key depression; without Graph-Mimic, thumb-under trials showed curled postures, wrist compensation, and missed keystrokes. The three-link model attributes style differences to configuration-dependent stiffness and torque under loading.Source: Articulation-specific touch techniques… · pp. 5–6

Piano acoustics model for tuning musical dynamics

  • Dynamic guidance achieved 85.7% velocity accuracy on wide-range pieces, while baseline accuracy fell to zero as MIDI Velocity variance increased.In Für Elise, the reference shifted from about 49 (mp) to about 80 (mf), and most generated velocities fell within the corresponding levels; the baseline instead produced excessive, uniformly loud tones.Source: Piano acoustics model for tuning… · pp. 6–7

The piano perceptual test and comparative analysis

  • Untrained listeners did not significantly distinguish the expressive robot from either human performance, and significantly preferred it over the baseline robot (p < 0.001).Across all three listener groups, the expressive robot ranked ahead of the baseline, though both ranked below human performances. Overall excerpt rankings differed significantly in every group (all p < 0.01); among trained pianists, humans ranked above both robots, while comparisons between the expressive robot and humans were nonsignificant for generally educated listeners.Source: The piano perceptual test and… · p. 7
  • The expressive performance’s Reference Absolute Difference of MIDI Velocity was 10.72 versus 15.50 for baseline, and velocity accuracy was 50% versus 37.5%.The expressive robot reproduced the human performance’s opening crescendo, which was absent from the baseline; baseline deviations exceeded 15 units in both reported metrics, while the expressive performance largely remained within the human dynamic range.Source: The piano perceptual test and… · pp. 7–8
  • Compared with baseline, the expressive robot showed more human-like pre-press and key-press postures, and a piano professor noted improved timing, articulation, coordination, and dynamic shaping.The baseline showed excessive little-finger curling and bent ring and little fingers, while the expressive robot maintained smoother, human-consistent transitions.Source: The piano perceptual test and… · p. 8

Experimental validation across diverse musical styles

  • Simulation achieved an average F1 of 0.96 across pieces, while real-world averages ranged from 0.80 to 0.95 across ten trials per piece.The simulation average was comparable with the 0.94 average reported in (23) and above (24), where most pieces exceeded 0.79. Technically demanding passages introduced errors but did not disrupt musical continuity.Source: Experimental validation across diverse… · p. 8
  • Human-robot accompaniment achieved average Time Gaps of -44ms for Croatian Rhapsody and -31ms for Für Elise, both below the 50 ms MIDI acquisition resolution.The performances were described as musically coherent, with excellent temporal synchronization. The system also reproduced mixed black-white key chords, overlapping presses, rapid sixteenth-note passages, three-finger chords, and octave spans.Source: Experimental validation across diverse… · p. 9

Discussion

Graph-Mimic combines human-like fingering and touch techniques with a simplified acoustic model to support expressive robotic piano performance. The paper also identifies limitations in dynamic control, arm motion, evaluation, sensing, and policy generality, and outlines future directions.

Discussion

  • On Twinkle, Twinkle, Little Star, the system achieved an F1 score of 0.92 versus prior approaches’ average F1 scores of approximately 0.60 (45), while non-experts showed no statistical difference from human pianists.The system performed Grade 7 repertoire, including Croatian Rhapsody, and coordinated reliably in human duets; expert evaluations found significantly greater expressivity than baseline controllers.Source: Discussion · p. 9
  • Piano performance offers a quantitative dexterous-manipulation testbed because it combines fingertip contact–separation, rapid musical sequences, and fine keystroke-velocity control.These demands jointly test hand degrees of freedom, response speed, and force control, and may help advance dexterous manipulation toward real-world deployment.Source: Discussion · p. 9
  • A unified graph encodes phalangeal and inter-fingertip vectors to learn pre-press fingering coordination and key-press contact postures, with potential to support natural, efficient, compliant manipulation.The proposed extension is to incorporate object feature points into the graph for general manipulation tasks.Source: Discussion · pp. 9–10

Limitations and future work

  • The four-level p/mp/mf/f loudness classification cannot capture the full ppp–fff dynamic continuum needed for nuanced expression.Future work could use pressure-sensitive control, high-bandwidth tactile transducers, and deep-learning-based haptic interpretation for more continuous dynamics.Source: Limitations and future work · p. 10
  • Without proximal arm kinematics, the system cannot represent weight transfer and joint energy flow required in demanding passages such as Liszt’s Transcendental Étude No. 4.The paper proposes full-arm motion capture and hybrid impedance-admittance control; it also names Brahms’s Variations on a Theme of Paganini, Op. 35 as a demanding example.Source: Limitations and future work · p. 10
  • F1 measures pitch–onset accuracy but not expressivity well, since expressive timing variations in human performances can lower scores.The study supplements F1 with Reference Absolute Difference, Mean Velocity Delta, and perceptual evaluation; holistic metrics spanning note accuracy, timing flexibility, and dynamic shaping remain open.Source: Limitations and future work · p. 10
  • The controller depends on simulation piano-state variables, including key angular velocities, so current deployment uses simulation observations.Future approaches could reconstruct velocities from real-time MIDI with the simplified piano model or distill a student policy that avoids privileged simulation state.Source: Limitations and future work · p. 10
  • The current work uses song-specific policies, leaving generalist piano-playing controllers for real-time and improvisational performance as future work.The authors describe their insights and control strategies as a foundation for developing those generalist controllers.Source: Limitations and future work · p. 10

Materials and Methods · Study Design

The study was designed to benchmark an expressive robotic hand controller against professional pianists and a baseline robotic controller using perceptual and objective technical evaluations.

Study Design

  • The study compared the expressive robotic controller with professional human pianists and a baseline robotic controller.Its experimental framework comprised two core dimensions.Source: Study Design · p. 10
  • The perceptual test included 125 participants: 24 formally trained pianists, 64 people with general piano education, and 37 untrained listeners.Participants ranked randomized audio excerpts to assess perceived musical quality.Source: Study Design · p. 10
  • Participants ranked randomized audio excerpts to determine perceived musical quality.The test covered formally trained pianists, people with general piano education, and untrained listeners.Source: Study Design · p. 10
  • Objective technical validation assessed mechanical precision using F1 score as its primary metric.This was the study’s second core evaluation dimension.Source: Study Design · pp. 10–11
  • Physical robot results were aggregated across 10 independent trials per musical piece to account for mechanical variance.Source: Study Design · p. 11
  • Human-performance and simulation reference datasets used their highest-scoring instances across multiple recordings for a rigorous benchmark.Source: Study Design · p. 11

Reinforcement Learning Setup

The reinforcement-learning controller combines key-press, energy, and fingering rewards with graph-based hand-movement similarity and acoustics-based musical-dynamics rewards.

Reinforcement Learning Setup

  • The setup adapts RoboPianist (21) rewards for correct key presses, energy use, and fingering, then adds Graph Distance and Musical Dynamics rewards.Full algorithm and network details are provided in text S6 and table S5; the observation, action, and reward setup is summarized in Table 1.Source: Reinforcement Learning Setup · p. 11

Validation of Graph Distance

  • Graph Distance compares human and robotic hand graphs built from landmarks and phalangeal and inter-fingertip vectors, normalized by wrist-to-middle-fingertip distance.Human landmarks are extracted from performance videos using MediaPipe (46); the robot’s corresponding feature points are defined in MuJoCo.Source: Validation of Graph Distance · p. 11
  • Fingering compensates for underestimated video depth, while Graph Distance weights metacarpal, proximal, middle, and distal segments 1, 2, 2, and 5.The reward decays smoothly outside a graph-distance tolerance of [0, 5], with margin 10 and reward value 0.1; fingertip direction also affects key location and tone.Source: Validation of Graph Distance · pp. 11–12

Piano Acoustics Model and Musical Dynamics Reward

  • The acoustics model relates key angular velocity to MIDI Velocity by modeling key energy transfer into string vibration and considering the string’s first-order vibration mode.MIDI Velocity represents loudness on a 0–127 scale.Source: Piano Acoustics Model and Musical… · pp. 12–13
  • The controller observes angular velocities for all 88 keys and rewards agreement between computed and reference MIDI Velocity, with tolerance [0, 3], margin 9, and margin reward 0.1.Source: Piano Acoustics Model and Musical… · pp. 13–14

Physical World Setup

The physical platform combines a digital piano, a UR5 robotic arm, and an InReal dexterous hand. The setup pairs expressive piano sound and real-time performance data with a hand designed for precise, rapid, and dynamic key striking.

Physical World Setup

  • The platform comprises a digital piano, robotic arm, and dexterous hand.Fig. 7A presents these three core components.Source: Physical World Setup · p. 14

Piano

  • The Yamaha P-48B has 88 weighted hammer-action keys, stereo-sampled tones with expressive dynamic range, and USB MIDI for real-time data acquisition and analysis.Its tactile response and sound loudness are described as similar to a grand piano and closely approximating a real piano, respectively.Source: Piano · p. 14

Robot Arm

  • The UR5 provides repeatable, precise positioning but can limit rapid octave transitions in virtuoso passages such as La Campanella.Its ±0.1 mm positional repeatability meets piano-performance precision needs; its typical maximum end-effector speed is 1 m/s, and RTDE enables low-latency control.Source: Robot Arm · p. 14

Dexterous Robotic Hand

  • The InReal hand combines high dexterity, rapid actuation, fingertip force, a narrow profile, and wide span for piano performance.It has 18 active DoFs, joint speeds exceeding 1000°/s and a maximum key-strike frequency of 10 Hz, peak fingertip force up to 18.5 N, 14 mm fingertips versus 22.5 mm white keys, and a 265 mm thumb-to-pinky span versus a 157.5 mm octave span.Source: Dexterous Robotic Hand · p. 15

Statistical Analysis

Preference rankings were analyzed with related-sample nonparametric tests: a Friedman test assessed differences among four excerpts, followed by adjusted pairwise tests when significant.

Statistical Analysis

  • Analyses used participant ranks coded numerically (1 = most preferred) and were conducted separately for trained pianists (n=24), general-education participants (n=64), and untrained listeners (n=37).Individual rank matrices and group-level mean ranks were computed in Python using pandas and SciPy.Source: Statistical Analysis · p. 15
  • The Friedman test evaluated overall rank differences among four excerpts: human1, human2, expressive robot, and baseline robot, with significance set at p < 0.05.Its p-value was obtained from a χ2 distribution with k−1 degrees of freedom; a significant result indicates at least two excerpts differed in rank.Source: Statistical Analysis · pp. 15–16
  • When the Friedman test was significant, Wilcoxon signed-rank tests compared excerpts pairwise, with Holm–Bonferroni-adjusted p-values used to control the family-wise error rate.Corrected p-values were reported in S6, with significance defined as p < 0.05.Source: Statistical Analysis · p. 16

Supplementary materials

The supplementary materials provide six text sections, five figures, six tables, references, thirteen movies with legends, and one data supplement.

Supplementary materials

  • Supplementary materials include Text S1–S6, Figs. S1–S5, Tables S1–S6, References (49,50), Movies S1–S13 and their legends, and Data S1.Source: Supplementary materials · p. 16

Supplementary Materials for Expressive Robotic Pianist: Mastering Complex Piano · Repertoire with Graph-Mimic and Musical Dynamics

The supplementary materials detail the robotic pianist’s mechanical, perceptual, and training aspects, including finger stiffness, coordination, piano-action energy transfer, performance assessments, ensemble timing, and reinforcement-learning implementation. They also provide video examples of expressive control and challenging techniques, along with Piano Turing Test stimuli and preference-ranking data.

Supplementary Text

  • In the three-link finger model, fingertip stiffness varies with configuration, and simulated total joint torque under vertical loading decreases with flexion, reaching a minimum near q1 = q2 = q3 ≈ 0.67 rad.The analysis models the MCP, PIP, and DIP joints as revolute joints and derives fingertip stiffness from the Jacobian.Source: Supplementary Text · pp. 30–31, Figure S1 · p. 38
  • Graph-Mimic preserved relaxed, extended finger postures during a thumb-under transition, while the baseline’s wrist compensation led to a missed key and disrupted performance continuity.The comparison attributes the difference to Graph-Mimic prioritizing finger articulation over wrist translation, helping resolve redundant hand degrees of freedom into reliable keystrokes and plausible movements.Source: Supplementary Text · p. 32, Figure S2 · p. 39
  • The fitted power-law relation between key angular velocity and energy transfer captures the measured nonlinear trend and informs the acoustic simulations’ link between key motion and hammer excitation.The study focuses on grand piano data and simplifies the energy model by neglecting dissipative correction terms for computational tractability.Source: Supplementary Text · p. 33, Figure S3 · p. 40
  • The two human references illustrated distinct approaches to phrasing: one showed inconsistent dynamics and finger evenness, while the other sustained natural tempo flow and motivic integration.The first reference had abrupt phrase releases and uneven alternating E5 and D♯5; the second used consistent emphasis in measures 9–12 and a less accented measure 8 resolution.Source: Supplementary Text · pp. 34–35
  • The expressive robotic performance improved timing, articulation, coordination, and dynamic shaping over the baseline, but some accents remained abrupt or overemphasized.The expert commentary notes better continuity at the measure 8 resolution and a gradual softening in measures 9–12, while the C5 accent at the end of measure 5 sounded mechanistic.Source: Supplementary Text · pp. 34–35
  • The ensemble time-gap metric measures robot–human onset differences for simultaneous notes, with positive values meaning the human plays later and negative values the reverse.It is applied to performances of Croatian Rhapsody and Für Elise, where the robot plays melody and the human provides chord accompaniment.Source: Supplementary Text · p. 36, Figure S4 · p. 41
  • The SAC implementation uses a multi-stage curriculum: 6 × 10^6 base-training steps followed by fine-tuning rounds of 3 × 10^6 steps to prioritize note accuracy and dynamic precision.The framework uses automatic entropy adjustment and actor–critic networks; implementation and training hyperparameters are listed in Table S5.Source: Supplementary Text · p. 37, Table S5 · p. 46

Supplementary Figures

  • Movie S6 shows velocity modulation reproducing dynamic contrasts in Für Elise in alignment with reference MIDI dynamics.Source: Supplementary Figures · p. 48
  • Movie S7 presents four audio-only Piano Turing Test excerpts—two human performances, one baseline robot, and the proposed expressive robot—for participants to rank by preference.Source: Supplementary Figures · p. 48
  • Movies S9 and S10 show human–robot ensemble performances of Croatian Rhapsody and Für Elise with seamless acoustic integration and timing coordination.Croatian Rhapsody is identified as Grade 7 repertoire.Source: Supplementary Figures · p. 48
  • Movies S11–S13 demonstrate mixed-key chords with overlapping presses, rapid sixteenth notes and three-finger chords, and sustained octave spans requiring hand reconfiguration and stability.The examples are from The Truth That You Leave, Sonata No.16, 1st Movement, K.545, and See You Again, respectively.Source: Supplementary Figures · p. 49
  • Data S1 provides preference rankings in three listener-group CSV files and documents anonymization and the shared-device exception for the last five untrained responses.The files are formally-trained.csv, generaleducated.csv, and untrained.csv; the proposed method is stimulus No. 3.Source: Supplementary Figures · p. 49

Related papers

Loading 2609.10844v1…