PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the End-Effector: Learning Whole-Arm Contact Manipulation from Human Demonstrations

Bo-Ruei Huang, Daniel Seita, Erdem Bıyık

arXiv:2610.17802v1cs.ROcs.CVcs.LG

TL;DR

Humans use whole-arm contact for secondary tasks, but learning robot behavior from bare-hand demonstrations is complicated by the human-to-robot embodiment gap. WACL combines robot-agnostic task representations with a surface-contact optimizer that grounds them on the robot. Across four real-world tasks, it achieves 78% success under end-effector constraints versus 33% for a strong link-pose baseline.

  • Problem

    Bare-hand demonstrations of whole-arm contact provide no robot actions and introduce a human-to-robot embodiment gap.

    Source: §1 · p. 2

  • Method

    WACL learns robot-agnostic keypoints, contact trajectories, and end-effector path constraints, then grounds them through a surface-contact optimizer.

    Source: §1 · p. 2

  • Results

    78% vs. 33% success under end-effector constraints, with WACL outperforming a strong link-pose baseline across four real-world tasks on a 7-DoF xArm.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    WACL handles a second simultaneous end-effector constraint without retraining and transfers zero-shot to a second 7-DoF arm by changing only the optimizer’s robot model.

    Source: §5 · p. 9

    WACL plans geometrically from contact position and normal without modeling force or object dynamics, so feasible plans can still slip or under- or over-press.

    Source: §6 · p. 9

Abstract

from arXiv · show

Robot manipulation has traditionally relied on end-effector-centric interactions. In contrast, humans routinely exploit whole-arm contacts to improve efficiency, robustness, and adaptability, especially when the hands are occupied or when precise grasping is unnecessary. Enabling similar capabilities in robots remains challenging: the robot must select where on its arm to make contact, generate effective non-prehensile interactions, and satisfy task-specific pose or safety constraints on the end-effector. We propose Whole-Arm Contact Learning (WACL), a hierarchical framework that describes each whole-arm contact task with three robot-agnostic representations and hands them to a quasi-static surface-contact optimizer: semantic keypoints that summarize the scene, a reference contact trajectory that specifies where and how the arm presses on the object, and safety-aware path constraints that a vision-language model generates from the task description to protect the end-effector's task. The robot's kinematic chain enters only in this optimizer, which grounds these representations onto the arm by jointly searching over which link makes contact, where on that link, and the joint configuration that realizes it. This decomposition lets WACL learn the contact trajectory from 10 bare-hand human demonstrations per task collected without teleoperation, while new end-effector constraints change only the optimizer's input rather than any trained component. Real-world experiments on a 7-DoF arm across four whole-arm tasks show that WACL sustains contact under end-effector constraints where a strong link-pose baseline fails (78% vs. 33% success) and generalizes to new constraints without retraining. Website: https://wacl-robot.github.io

1 Introduction

Humans use whole-arm contact to perform secondary tasks while keeping a primary hand task intact, but robot manipulation remains largely end-effector-centric. WACL combines robot-agnostic task representations with an optimizer that grounds them on the robot, and its constrained real-world success rate exceeds a strong link-pose baseline.

1 Introduction

  • Whole-arm manipulation lets people perform a secondary task while the hand maintains its primary task, but robot manipulation remains overwhelmingly end-effector-centric.In constrained or cluttered settings, gripper-only motion may be infeasible; learning whole-arm behavior is also challenged by cross-link observation differences and the human-to-robot embodiment gap.Source: §1 · pp. 1–2
  • WACL represents each task with semantic keypoints, a reference contact trajectory, and safety-aware end-effector path constraints, then grounds them in a single surface-contact optimizer.The optimizer jointly searches for the contact link, anchor location, and joint configuration; new end-effector constraints change its input rather than the trained policy.Source: §1 · p. 2
  • 78% vs. 33% success under end-effector constraints, with WACL outperforming a strong link-pose baseline across four real-world tasks on a 7-DoF xArm.WACL also handles a second simultaneous constraint without retraining and transfers zero-shot to a Franka FR3 by changing only the optimizer’s robot model.Source: §1 · p. 2
  • WACL learns contact trajectories from bare-hand, multi-view human demonstrations without requiring whole-arm teleoperation or robot demonstrations.Its optimizer uses a VLM-generated path constraint to preserve the end-effector task while selecting the arm contact.Source: §1 · p. 2

2 Related work

Prior work addresses whole-body coordination, contact-rich manipulation, and semantic or geometric planning, but does not jointly exploit arm-surface contact while preserving an ongoing end-effector task. WACL combines bare-hand demonstration learning with optimizer-level end-effector constraints and contact-link selection.

2 Related work

  • Prior whole-body manipulation focuses on balance, locomotion, and floating-base coordination rather than using limb surfaces for functional interaction.Contact-rich approaches use tactile feedback or compliant skins, which assume specialized hardware.Source: §2 · p. 2
  • Existing arm-surface learning and model-based planning methods do not hold a concurrent end-effector task while the arm makes contact.WACL learns from bare-hand demonstrations without robot demonstrations, object models, or tactile sensing, and enforces the end-effector task in the optimizer that chooses the contact link.Source: §2 · p. 2
  • VLM-based robotics work includes spatial planning, control-code synthesis, reward functions, and geometric constraints, while WACL applies generated constraints at the link level.The constraints must hold on one link while a different link makes contact, and can change without retraining the policy.Source: §2 · pp. 2–3

3 Method

WACL separates whole-arm manipulation into robot-agnostic scene, contact-trajectory, and safety-constraint representations, then grounds them in a robot-specific optimizer. Human demonstrations train the contact trajectory, while typed constraints and surface-contact search produce executable motion.

3.1 Problem formulation

  • Three robot-agnostic representations describe each task: scene keypoints, a world-frame contact-position and surface-normal trajectory, and typed link-level path constraints.The resulting optimization seeks a feasible joint trajectory, contact link, and anchor sequence that track contact while satisfying constraints.Source: §3.1 · pp. 3–4

3.2.1 Offline data pipeline

  • Bare-hand, multi-view demonstrations yield reference contact trajectories by fusing fingertip observations and pairing sampled positions with surface normals.The demonstrations are segmented at contact, and multi-view triangulation addresses monocular depth inaccuracy.Source: §3.2 · p. 4, §3.2.1 · p. 4

3.2.2 Keypoint distillation and policy training

  • A keypoint matcher supplies scene observations to a conditional diffusion policy trained to generate reference contact trajectories.The policy is trained on keypoint–trajectory pairs from the demonstration pipeline.Source: §3.2.2 · p. 4

3.3 Safety-aware path constraint generation

  • At inference, two VLM calls estimate a scene state and instantiate typed axis-or-position constraints from task requirements.Constraint violations contribute both a soft optimization cost and a hard per-waypoint tolerance test.Source: §3.3 · p. 5

3.4 Surface-contact trajectory optimization

  • The optimizer jointly selects a contact link, surface anchors, and joint configurations to track the reference trajectory while satisfying constraints and collision checks.It samples reachable anchors, solves per-waypoint inverse kinematics, and assembles trajectories with beam search before ranking candidates.Source: §3.4 · pp. 5–6

3.5 Inference and execution

  • Inference sequentially obtains keypoints, samples a contact trajectory, generates constraints, and optimizes a joint trajectory with a free-space approach.The approach and contact trajectory are concatenated and streamed to the robot.Source: §3.5 · p. 6

4 Experiments

The experiments compare WACL with a link-pose baseline on four real-world tasks, then test new constraints, another robot, and design choices. Under end-effector constraints, WACL more than doubles the baseline’s aggregate success rate.

4.2 Constrained whole-arm manipulation on a 7-DoF xArm

  • 78% vs. 33%: under end-effector constraints, WACL more than doubles KALM’s aggregate success across four tasks.The comparison uses the xArm 7 and ten rollouts per task and setting.Source: Table 1 · p. 7, §4.2 · p. 8

4.1 Experimental setup

  • The four tasks pair a secondary contact action with a primary end-effector task, evaluated with AXISCONE or POSITIONREGION constraints.WACL uses ten bare-hand demonstrations per task; KALM uses ten teleoperated demonstrations, and both are evaluated over ten rollouts per task and setting.Source: §4.1 · pp. 7–8

4.3 Generalization to new end-effector constraints and embodiments

  • On CLOSEFAUCET, WACL succeeds in 7/10 rollouts with either one or two simultaneous end-effector constraints.The two-constraint test adds a cup-upright requirement; constraints are regenerated at inference and added as optimizer inputs.Source: §4.3 · p. 8
  • WACL transfers zero-shot to the Franka FR3 after only its optimizer’s robot model is changed, reaching 75% unconstrained and 68% constrained success.The constrained gap is concentrated on OPENCOFFEE, where the FR3’s thicker links struggle to thread the handle.Source: §4.3 · p. 8

4.4 Ablation study

  • On OPENCOFFEE, pinning the surface anchor lowers success from 7/10 to 3/10 because fixed-anchor IK becomes infeasible mid-trajectory under the cup-upright constraint.A sliding anchor avoids this failure.Source: §4.4 · p. 9
  • On PRESSBUTTON, link search reaches 9/10 versus 6/10 with link 5 alone and 7/10 with link 6 alone; on OPENCOFFEE, a geometric arc reaches 4/10 versus 7/10 for the learned policy.The two links cover largely different placements, while the learned trajectory captures coupled motion that the arc does not; registered replay reaches 6/10 on OPENCOFFEE versus 7/10 for the policy.Source: §4.4 · p. 9

5 Conclusion

WACL performs whole-arm contact while preserving a primary end-effector task. Across four real-world tasks, it more than doubles a link-pose baseline’s constrained success, accommodates a second constraint without retraining, and transfers to another arm by changing only the optimizer’s robot model.

5 Conclusion

  • Across four real-world tasks, WACL more than doubles the constrained success rate of a strong link-pose baseline, achieving 78% vs. 33%.The experiments also show zero-shot transfer to a second 7-DoF arm after changing only the optimizer’s robot model.Source: §5 · p. 9
  • WACL absorbs a second simultaneous end-effector constraint without retraining.The conclusion describes the framework as performing whole-arm contact without disturbing a primary end-effector task.Source: §5 · p. 9
  • Decoupling task requirements from how an arm achieves them lets the skills carry over to new constraints and a second arm.Source: §5 · p. 9

6 Limitations

WACL plans contact geometrically and runs open-loop over a short horizon, with constraints limited to two primitive types. The paper identifies physical contact modeling, multi-contact representation, and closed-loop replanning as areas for future work.

6 Limitations

  • Geometric contact plans track position and surface normal but model neither force nor object dynamics, so feasible plans can still slip or under- or over-press.Source: §6 · p. 9
  • The contact representation is one point on one link; simultaneous contacts would have spacing fixed by arm link lengths and would no longer be robot-agnostic.Source: §6 · p. 9
  • End-effector requirements are limited to AXISCONE and POSITIONREGION, so requirements outside those primitive types cannot be enforced.Source: §6 · p. 9
  • Keypoints, trajectory, and constraints come from one RGB-D frame and run open-loop for at most 8 waypoints; one inference with one candidate plan takes 34.6 s.The optimizer’s share of runtime is small, but its anchor search grows with the number of contact-capable links.Source: §6 · p. 9

A.1 Demonstration data pipeline

The appendix details how demonstrations, keypoint policies, and generated safety constraints feed a robot-specific search for executable whole-arm contact trajectories. The optimizer selects contact links and anchors while enforcing path constraints and maintaining continuous motion.

A.1 Demonstration data pipeline

  • Ten bare-hand demonstrations per task are captured with three calibrated RGB-D cameras at 10 Hz, with contact locations deliberately varied.The cameras record synchronized streams; multi-view fingertip fusion weights views by geometric consistency between MANO meshes and SAM3 hand masks.Source: §A.1 · p. 14

A.2 Keypoint-conditioned policy

  • The keypoint-conditioned policy uses the KALM architecture with eight keypoints to train on observation and reference-trajectory pairs.Training uses 100,000 iterations; when multiple views are available, the selected trajectory comes from the view with the highest mean keypoint feature similarity.Source: §A.2 · p. 14

A.3 Constraint primitives

  • Path constraints use AXISCONE and POSITIONREGION penalties, with a headroom gate rejecting trajectories that exceed a constraint’s budget.The gate normalizes consumed budget across angular and positional constraints; experiments use cutoff 1.0, so the stated bound is the safety specification.Source: §A.3 · pp. 14–15

A.4 Safety-aware constraint generation

  • Two VLM calls map task descriptions and scene observations to segmented entities, scene geometry, and a minimal set of typed safety constraints.The constraint call derives axes from grasp geometry, adds margins to regions, and covers risks such as spilling, contamination, fragility, or lost task utility.Source: §A.4 · p. 15, Figure 4 · p. 16, Figure 5 · p. 17
  • The constraint-instantiation prompt accepts new primitive types, although reported experiments use only AXISCONE and POSITIONREGION.The optimizer uses cuRobo’s GPU-batched inverse kinematics and trajectory kernels, with the surface-contact cost, constraint primitives, and beam-search assembly added for WACL.Source: §A.4 · p. 15, §A.5 · p. 15

A.5 Surface-contact trajectory optimizer

  • Surface-contact IK uses 16 random seeds per waypoint-anchor pair and leaves roll about the contact normal free for satisfying path constraints.Contact-normal precision is adjusted by task; accepted solutions must meet position tolerance and pass the active path-constraint gate.Source: §A.5 · p. 18
  • The beam search expands candidate anchors at each contact waypoint, preferring stationary anchors and rejecting extensions that exceed the joint-displacement cap.It uses beam width 20 and expands each surviving beam by 32 anchors per step; sliding is favored only when constraints make a stationary extension infeasible.Source: §A.5 · p. 18
  • The optimizer uses beam search rather than continuous optimization because earlier alternatives required many multi-start solves and produced non-physical intermediate contact points.The search’s trade-off is greedy waypoint assembly, with smoothness enforced only by the step cost and joint-displacement cap.Source: §A.5 · p. 18

A.7 Runtime breakdown

A representative inference takes 34.63 seconds, with constraint generation setting the critical-path barrier. The appendix also reports an offline constraint-generation audit and paired tests of the xArm results.

A.7 Runtime breakdown

  • 34.63 s is the total runtime for a representative inference, with constraint generation setting the 28.31 s barrier and hosted-VLM round-trips dominating wall-clock time.The three constraint-generation stages take 10.88 s, 2.76 s, and 14.68 s; optimizer stages after the barrier add 6.32 s.Source: §A.7 · p. 19, Figure 6 · p. 19

B.1 Real-robot setup

  • The main experiments use a 7-DoF xArm 7; cross-embodiment experiments use a 7-DoF Franka FR3 with a Robotiq gripper.The xArm setup uses two fixed RGB-D cameras and the wrist camera; both setups use the same planning computer.Source: §B.1 · p. 19, Figure 7 · p. 20

B.2 Constraint-generation audit

  • 94 of 100 constraint generations were fully correct, while the six incorrect outputs were over-constrained and rejected by the planner’s hard gate.The audit used 20 seeds per task and for the two-constraint variant; no output was looser than deployed.Source: Table 4 · p. 20, §B.2 · p. 20

B.3 Confidence intervals and tests

  • Under constraints, WACL succeeded where KALM failed on 19 placements and the reverse held on 1 (exact McNemar p = 4.0×10−5).Without constraints, the corresponding counts were 11 and 4 (p = 0.12); per-task constrained tests were not corrected for four tasks, so only the aggregate claim is made.Source: §B.3 · p. 21, Figure 8 · p. 21

B.4 Failure-case analysis

The appendix analyzes planner and execution failures, paired offline replay, and transfer to a different arm. It identifies contact-stage planning and robot morphology as important constraints on performance.

B.4 Failure-case analysis

  • All 24 failed contact candidates fell into four causes: 11 collision-gate rejections, 8 infeasible sliding transfers, 4 first-waypoint IK failures, and 1 constraint-margin rejection.The optimizer completed plans for 69/74 attempts and 124/148 candidates; all 11 collision-gate rejections occurred on CLOSECABINET.Source: §B.4 · p. 21
  • Feasible plans still failed at execution through keypoint mislocalization on reflective or transparent surfaces and uncorrected contact slip from one-time sensing.The slip was most visible on OPENCOFFEE, where the handle must be lifted through a precise arc.Source: §B.4 · p. 21

B.5 Paired offline planner replay

  • Offline replay success measures planning only: every waypoint must satisfy constrained IK and collision checks, while KALM can reject pose targets inside the constraint cone.Nothing in this replay measure is executed.Source: §B.5 · pp. 21–22

B.6 Cross-embodiment transfer

  • On the Franka FR3, changing only the robot model led the optimizer to select a different contact strategy than on the xArm for several tasks.The FR3 differs in link geometry, contact-capable links, and end-effector frame.Source: §B.6 · p. 22
  • The constrained gap on the FR3 is attributed mainly to its larger, rounder links, which offer smaller usable contact patches and present more collision-prone volume.This is especially apparent on OPENCOFFEE, where a link must seat on a slim handle and follow a precise lifting arc.Source: §B.6 · p. 22

B.7 Ablation and control details

These ablations and controls examine the effects of anchor sliding and link choice, and compare learned contact trajectories with geometric and replayed controls. They also report paired offline planner-replay outcomes.

Sliding anchor

  • On OPENCOFFEE, 5 of 7 no-sliding failures occurred when a fixed anchor became IK-infeasible mid-trajectory under the cup-upright constraint.Allowing the anchor to migrate trades some sliding for joint-space slack.Source: §B.7 · p. 22

Paired offline planner replay

  • In paired replay, switching the end-effector constraint off helps both planners; it closes the gap on CLOSEFAUCET, while KALM still fails on most OPENCOFFEE scenes.The replay counts complete plans over the same 30 logged scenes per row; nothing is executed.Source: Table 5 · p. 22

Link search

  • Across 10 PRESSBUTTON placements, both pinned links completed four, link 5 alone completed two, link 6 alone three, and neither completed one; multi-link search achieved 9/10.Every pinned plan was feasible, and all seven pinned failures occurred during execution.Source: §B.7 · p. 22

Geometric primitive

  • The geometric primitive uses six parameters for PRESSBUTTON and seven for OPENCOFFEE, while the learned policy needs none; the OPENCOFFEE arc failed by pushing or slipping off the handle in 5 of 6 failures.Its trajectory parameters are read from one demonstration and registered to each scene by ICP.Source: §B.7 · p. 22

Registered replay

  • All four registered-replay failures occurred during execution, and every replay produced an executable plan.The OPENCOFFEE medoid trajectory was registered by ICP, with no re-capture needed in ten rollouts.Source: §B.7 · pp. 22–23

C Additional limitations

The evaluation leaves several boundaries untested: robot structures beyond two fixed-base 7-DoF arms, new object instances, and the isolated effect of contact-normal roll. The stated safety constraints protect the end-effector task, not nearby people.

Scope of the evaluation

  • Robot-agnosticity is demonstrated only on two fixed-base 7-DoF arms; other kinematic structures, mobile bases, and whole-body robots remain untested.The rollouts vary object placement and robot home configuration, not object instance, so generalization to new instances is not measured.Source: C Additional limitations · p. 23

Constraints and parameters

  • The safety-aware constraints protect the end-effector’s own task and make no guarantee about people near the robot.The alignment deadzone and trajectory horizon are set per task.Source: C Additional limitations · p. 23

Demonstration data

Related papers

Loading 2610.17802v1…