PaperlayerView exact revision on arXiv

Source-linked AI summary

KPI: A Promptable Kernel for Physical Interaction on Humanoids

Yikai Wang, Honghao Zhu, Xiao Hu, Hao Zhang, Zelin Wang, Yip Fun Yeung, Ding Zhao, Lingfeng Sun

arXiv:2609.36151v1cs.RO

TL;DR

General humanoid trackers follow motion references, but the reference alone leaves interaction behavior to a controller that may be preset or task-specific. KPI adds a promptable kernel that adapts an unmodified tracker’s arm controller from a per-direction contract during contact. In demonstrations, a vision-language agent generated references and contracts from instructions, and KPI completed the reported winch, door, and box trials.

  • Problem

    General humanoid stacks use preset arm gains, while hard interactions in task-specific policies commonly require trajectory and contact response to be optimized together in simulation.

    Source: §I · p. 1

  • Method

    KPI places a kernel between the trajectory source and an unmodified whole-body tracker, adapting selected arm parameters at contact rate from a per-direction contract and interaction state.

    Source: §I · p. 2, A. KPI: Contract-Driven Physical… · p. 3

  • Results

    KPI completed all five trials on each of the winch, door, and box tasks, while native joint PD and sensorless admittance completed zero and one of 15 trials, respectively.

    Source: §I · p. 2

  • Takeaways & Limitations

    A vision-language agent can write the reference and contract from one instruction without task-specific code, and KPI turned a winch until a second robot was lifted off the ground.

    Source: §I · p. 2, §V · p. 8

Abstract

from arXiv · show

Humanoids now walk, balance and reach with remarkable generality: one whole-body tracking policy follows references from a human, or from an end-to-end policy. That generality travels in the trajectory, and a trajectory alone carries limited information about the interaction it should produce: at contact, the executing controller determines how the robot behaves. Single-task policies usually reach hard interactions by optimising trajectory and controller together in simulation; general stacks usually assume a preset or hand-chosen controller. We present KPI, a promptable kernel for physical interaction between the trajectory source and an unmodified whole-body tracker. Instead of a controller fixed before the task, the trajectory source sends a contract: per direction, track, comply, or hold a force range. From tracking error and a wrench estimate, the kernel adapts the arms' stiffness, damping, reference and feedforward toward it at contact rate. We demonstrate KPI through an agentic framework: from one instruction, a vision-language agent writes both the reference trajectory and the contract, with no task-specific code. We demonstrate instruction-driven winch operation, door opening, and box transport, alongside scripted surface-interaction experiments. In the winch demonstration, the humanoid is able to turn a crank to hoist a second robot fully off the ground.

I. INTRODUCTION

General humanoid trackers carry motion references but lack task-specific interaction information, while learned and general-purpose approaches either co-optimize contact per task or use preset gains. KPI places a contract-driven interaction kernel between the commander and an unmodified tracker, and its evaluations show stronger task completion than the reported baselines.

II. RELATED WORK

  • General humanoid stacks use arm gains chosen before deployment, whereas task-specific policies optimize trajectory and contact response together in simulation [10], [11], [12], [13], [14].The paper also situates these approaches alongside trackers driven by teleoperation, retargeted recordings, or VLA models,, [3], [4],, [6], [7], [8], [9].Source: §I · p. 1, §II · p. 2

I. INTRODUCTION

  • KPI places a kernel between the commander and an unmodified whole-body tracker, adapting arm gains, reference, and feedforward at contact rate to meet a per-direction interaction contract.The commander specifies whether to track, comply, or hold a force range, rather than choosing gain values.Source: §I · p. 2
  • Across winch, door, and box tasks, KPI completes all five trials on each, versus zero for native joint PD and one for sensorless admittance.On the winch, KPI turns the handle until a second G1 is lifted off the ground.Source: §I · p. 2

II. RELATED WORK

  • The contract retains the structure of hybrid position/force task specifications but states force ranges and which parameters may change,,.Language-driven alternatives set gains directly, periodically, or by retrieval; KPI instead uses a contract and closes the loop at contact rate [33], [34],, [36].Source: §II · pp. 2–3
  • Prior humanoid compliance is trained into a tracker, fixed or operator-selected, or adapted by task-specific policies; KPI instead feeds estimated wrench information back into parameters during contact [15], [16], [17], [18],, [26],.KPI builds on MCC’s estimator and adds a path from estimated wrench to parameters during contact [18].Source: §II · pp. 2–3

III. METHOD

KPI makes interaction objectives explicit in a contract and adapts selected controller parameters during execution. Its update uses the interaction state to jointly satisfy directional tracking, compliance, or force-range requirements within bounded parameter choices.

III. METHOD

  • The commander supplies a nominal reference and an interaction contract, and KPI adapts controller parameters during execution while returning the interaction state.The same interface is available to a teleoperator or a VLA policy.Source: §III · p. 3

A. KPI: Contract-Driven Physical Interaction

  • At each adaptation cycle, KPI uses the nominal end-effector reference, current pose and velocity, and estimated external wrench to compute a contract-constrained parameter update.The selected parameters are restricted to contract-authorized changes within fixed controller bounds.Source: A. KPI: Contract-Driven Physical… · p. 3
  • The force prediction linearizes the impedance response at measured pose and velocity; a separate short-horizon surrogate predicts tracking error without identifying environmental stiffness.An estimated wrench anchors force prediction, and a regularizer holds parameters where the contract leaves them free.Source: A. KPI: Contract-Driven Physical… · p. 3
  • A contract specifies directions, tracking, compliant, or constrained behavior, force bounds, and which of gain, reference, or feedforward channels may change.The configured directions and channels are not an exhaustive vocabulary; parameters selected across directions are optimized jointly.Source: A. KPI: Contract-Driven Physical… · p. 4
  • Tracking objectives penalize predicted pose errors, while compliant behavior favors a fixed low-stiffness target and constrained behavior imposes force bounds without an added tracking or stiffness objective.A tracking cost can allow the execution reference to slow when the hand lags; parameters outside active specifications remain fixed.Source: A. KPI: Contract-Driven Physical… · p. 4
  • The update runs at 100 Hz, using closed-form solutions where available and bounded numerical optimization otherwise; unattainable force intervals are handled by clipping predicted violation within gain bounds.Winch contracts enable gain and reference adaptation, while board writing enables feedforward adaptation for board-normal force regulation.Source: A. KPI: Contract-Driven Physical… · p. 4

B. Realization on a Humanoid Robot

KPI realizes a tunable Cartesian impedance arm on the humanoid and splits damping across the motor-driver and host interfaces. Its sensorless wrench estimate and shared motion references support agentic execution and teleoperation.

B. Realization on a Humanoid Robot

  • Cartesian impedance realizes KPI’s tunable palm response on the G1, whose low-ratio quasi-direct-drive actuators facilitate backdriving,.The controller runs at 500 Hz with a wrench estimate refreshed at 200 Hz; translational stiffness defaults to 300 N/m, is bounded to [30, 1000] N/m, and uses 100 N/m as its compliant target.Source: B. Realization on a Humanoid Robot · p. 4
  • The damping matrix is split so joint-local diagonal terms use fresh driver velocity measurements, while cross-joint terms and remaining impedance torques are computed on the host.In the zero-delay limit, the combined paths recover Cartesian damping; residual delay still constrains the achievable response.Source: B. Realization on a Humanoid Robot · p. 4
  • The sensorless wrench estimator maps gravity-corrected external-torque residuals through the palm Jacobian using regularized least squares under a quasi-static approximation.Friction is not fully eliminated, but mean force-estimation error is expected below 5 N for the tasks considered.Source: B. Realization on a Humanoid Robot · pp. 4–5
  • The task coverage spans bimanual transport, articulated-object manipulation, and surface interaction.Figure 3 labels these three task categories.Source: Figure 3 · p. 5
  • SONIC commands the legs and waist while KPI commands the arms, with shared commanded motion supplying agentic references and retargeted motion supplying teleoperation references.The contract is written by the agent in agentic execution and stated by the operator under teleoperation.Source: B. Realization on a Humanoid Robot · p. 5

C. Agentic Execution

The agent turns instructions and observations into staged nominal motions and interaction contracts, with a verifier checking proposals against geometry and robot constraints. KPI adjusts the controller during execution, and stage outcomes guide verified revisions of the remaining plan.

C. Agentic Execution

  • From instruction and observations, the Analyzer requests task geometry, and the Detector maps RGB-localized features through depth into 3D grasp points and trajectory parameters.Reusable motion cores, such as linear or arc motions, define each stage’s nominal reference.Source: C. Agentic Execution · p. 5
  • The Analyzer pairs each stage’s motion with a contract, while the Verifier checks proposals against observations, kinematics, and collision constraints.The framework composes shared motion cores and contracts across tasks without task-specific code.Source: C. Agentic Execution · p. 5
  • During a stage, KPI uses the interaction state to adapt selected parameters so contract objectives and force constraints guide responses to changing geometry and loading.The VLM does not specify each parameter update.Source: C. Agentic Execution · p. 5
  • After each stage, the Verifier checks fresh observations against the intended outcome, and the Analyzer revises and re-verifies the remaining reference and contract while retaining completed stages.Completion is judged against the environmental goal, not the end of a trajectory.Source: C. Agentic Execution · p. 5

IV. EXPERIMENTS

The experiments evaluate KPI across three categories, using zero-shot agentic execution on winch operation, door passage, and box transport and placement. The robot and controller comparisons test KPI alongside shared tracking and control baselines.

IV. EXPERIMENTS

  • Zero-shot agentic evaluations cover winch operation, door passage, and box transport and placement.KPI also supports bimanual transport, articulated-object manipulation, and surface interaction.Source: §IV · p. 5
  • All conditions use the same SONIC whole-body tracker on a Unitree G1, with no wrist force sensors.Agentic execution uses three-point tracking; teleoperation uses SMPL tracking.Source: §IV · p. 6
  • The four comparisons are SONIC native joint PD, MCC* sensorless admittance, KPI-fixed impedance, and KPI with online adaptation.KPI-fixed uses KPI’s impedance realization without controller-parameter adaptation.Source: §IV · p. 6
  • Each agentic trial begins from one instruction without task-specific code or a manually supplied contract.The shared agent requests stiffnesses for MCC*; SONIC ignores the generated contract, and force-requiring contacts use a default 10 cm reference offset.Source: §IV · p. 6

A. Winch Operation

KPI completes all four requested winch revolutions across five trials, while SONIC and MCC* do not exceed half a revolution. Online adaptation addresses the changing resistance and constrained path that make fixed controller parameters problematic.

A. Winch Operation

  • The handle follows a constrained circular path whose resistance varies within each turn and rises as the other robot is lifted.Excessive force can impede rotation, push the operating robot backward, or cause loss of grasp.Source: A. Winch Operation · p. 6
  • 100% mean progress: KPI completes all four winch revolutions, while SONIC and MCC* achieve 20%.Progress is measured across grasping and one stage per completed revolution, over five trials per method.Source: A. Winch Operation · p. 6
  • As resistance slows the hand, KPI slows the conditioned reference and raises tangential stiffness to maintain tracking.Limiting reference lead also reduces misalignment between tracking force and the locally admissible tangent; force caps constrain loading in other directions.Source: A. Winch Operation · p. 6
  • With fixed parameters, KPI-fixed’s hand stalls as its advancing reference misaligns tracking force from the path tangent, then reverses.The hand trajectory comparison and failure case are shown in Fig. 5(b).Source: A. Winch Operation · p. 6, Figure 5 · p. 6
  • MCC*’s continued reference advance at a stalled hand creates upward tracking force that deflects the wrist.Across baseline trials, excessive contact forces commonly cause unintended body motion or loss of grasp; Fig. 5(c) shows the MCC* failure case.Source: A. Winch Operation · p. 6, Figure 5 · p. 6

B. Door Opening and Passage

KPI completes the door-passage task in all five trials, whereas SONIC never unlatches the door and MCC* completes passage once. The task combines incomplete visibility with a handle motion constrained by the door hinge.

B. Door Opening and Passage

  • The robot cannot initially observe the full door, so it cannot accurately estimate the opening trajectory.The VLM can infer the required motions despite incomplete visibility.Source: B. Door Opening and Passage · p. 6
  • With only part of the door visible, the agent combines backward stepping and a leftward reach, using compliance in all directions to open it.It assigns rotational tracking to turn the handle and release the latch.Source: B. Door Opening and Passage · p. 6
  • Across five trials, KPI completes all five door-passage stages; SONIC never unlatches, and MCC* completes passage once.Completion requires both feet and the torso to pass through the doorway.Source: B. Door Opening and Passage · p. 6
  • MCC*’s force-based reference modification has estimation errors that cause slight oscillations and reduced tracking accuracy, including a missed grasp.The caption depicts the task sequence from grasping through walking through the doorway.Source: B. Door Opening and Passage · p. 7, Figure 6 · p. 7
  • After unlatching, pulling while stepping backward conflicts with the hinge constraint, making compliance essential to accommodate the mismatch.MCC* responds more slowly to external forces than KPI in this implementation; repeated handle impacts and oscillations make pulling unreliable.Source: B. Door Opening and Passage · p. 7

C. Box Transport and Load Robustness

KPI achieves 100% mean progress in box transport, while SONIC and MCC* each achieve 20% and fail to lift the box. The full agentic sequence uses a light box, while separate lifting tests assess load robustness across three masses.

C. Box Transport and Load Robustness

  • The task requires maintaining a bimanual grasp through turns and squats, with inward forces that prevent slip without crushing the box or overheating the robot.Small hand–box orientation changes can break individual finger contacts.Source: C. Box Transport and Load Robustness · p. 7
  • The full agentic sequence uses only the Light 0.85 kg box; separate single-step lifting tests assess Light, Medium 1.85 kg, and Heavy 3 kg.The restriction limits overheating risk during prolonged agentic execution.Source: C. Box Transport and Load Robustness · p. 7
  • Across five trials per method, progress over five stages averages 100% for KPI on the box task, while SONIC and MCC∗ each average 20% and fail to lift the box.Source: C. Box Transport and Load Robustness · p. 7
  • SONIC and MCC* cannot lift because they fail to maintain stable multi-point contact between both hands and the box sidewalls.The Fig. 7 caption identifies SONIC’s multi-point-contact failure.Source: C. Box Transport and Load Robustness · p. 7, Figure 7 · p. 7
  • For Medium and Heavy loads, KPI-fixed’s selected stiffnesses of 500 and 600 N/m trigger motor errors, while KPI uses force ranges of 25–40 N and 35–50 N.KPI’s closed-loop adaptation yields lower contact forces than KPI-fixed in these tests.Source: C. Box Transport and Load Robustness · p. 7

D. Board Writing

In board writing, KPI and KPI-fixed both produce recognizable letters, while KPI brings estimated contact force closer to its target with little visible change in the writing. Teleoperation also extends KPI to drawer opening and box carrying across several whole-body motions.

D. Board Writing

  • KPI’s mean estimated contact force is closer to the target than KPI-fixed’s, with little visible effect on the written pattern.Source: D. Board Writing · p. 8
  • KPI and KPI-fixed both produce recognizable N and B shapes in board writing.The comparison uses the nominal NB reference.Source: D. Board Writing · p. 7, Figure 8 · p. 8

E. Additional Tasks and Whole-Body Motions

V. CONCLUSION

KPI adapts an arm controller online at contact rate from an interaction contract while leaving the whole-body tracker unmodified. Demonstrations include instruction-driven winch operation that lifts a second robot, door opening, and box carrying; the kernel does not choose objectives, and its wrench estimate has stated limits.

V. CONCLUSION

  • From one instruction and with no task-specific code, a vision-language agent operated a winch to lift a second robot, opened a door, and carried a box.A teleoperator also used the interface for drawer opening and box carrying while kneeling, standing, and running.Source: §V · p. 8
  • KPI adapts the arm controller online at contact rate according to the interaction contract, between the commander and an unmodified whole-body tracker.Source: §V · p. 8
  • The kernel does not choose objectives, and unanticipated jams appear only as a stalled stage.The wrench estimate is quasi-static, and its error floor rises with stiffness and gait; torque sensors or tactile palms could lower it and admit lighter contact.Source: §V · p. 8

S1. LOCAL MODEL AND PARAMETER UPDATE

The local model predicts immediate load changes from controller-parameter updates and uses a separate short-horizon tracking surrogate. Its predictions assume slowly varying contact, while constrained updates retain current stiffness when feasible and use a bounded fallback otherwise.

S1. LOCAL MODEL AND PARAMETER UPDATE

  • The local predictions assume slowly varying contact; impacts and strongly inertial interaction are outside their scope.Both predictions are state-local, and fresh measurements replace them at each update.Source: S1. LOCAL MODEL AND PARAMETER UPDATE · p. 10
  • Immediate load prediction holds measured pose and velocity fixed, and the measured wrench anchors the parameter-induced load estimate rather than being optimized.The wrench estimate is refreshed at the next update.Source: S1. LOCAL MODEL AND PARAMETER UPDATE · p. 10
  • A separate damping-dominated surrogate predicts tracking error over a 10 ms horizon while holding the current external load fixed.The surrogate depends on robot controller parameters and the horizon, not estimated environmental compliance.Source: S1. LOCAL MODEL AND PARAMETER UPDATE · p. 10
  • A constrained update adds no tracking or stiffness-preference objective, so its regularizer keeps stiffness closest to its current value when feasible.If the force-band boundary is unattainable, clipping selects the bounded endpoint with the smallest predicted violation; when gain has no local force authority, stiffness is retained.Source: S1. LOCAL MODEL AND PARAMETER UPDATE · p. 10

Related papers

Loading 2609.36151v1…