PaperlayerView exact revision on arXiv

Source-linked AI summary

PanTact: Teaching Whole-Body Dexterity with a Wearable Contact Interface

Xiaomeng Xu, Han Zhang, Zhian Ruan, Xiaofeng Guo, Jinkun Cao, Ruihan Yang, Carmelo Sferrazza, Shuran Song, Guanya Shi

arXiv:2610.12558v1cs.RO

TL;DR

Humanoid loco-manipulation often limits manipulation to the hands, despite robots having other potential contact surfaces. PanTact learns whole-body dexterity from robot-free human demonstrations by transferring contact goals across embodiments, and its Unitree G1 deployment demonstrates body support, pushing, sweeping, and load-sharing.

  • Problem

    Loco-manipulation research often restricts manipulation to the hands, while existing demonstration pipelines do not explicitly carry body-contact information through policy prediction and robot execution.

    Source: §1 · pp. 1–2

  • Method

    PanTact uses wearable contact patches to record human body contacts, predicts contact goals alongside end-effector trajectories, and applies contact-aware whole-body control.

    Source: §1 · p. 2

  • Results

    On a Unitree G1 without additional body contact sensors, PanTact demonstrates body support, pushing, sweeping, and load-sharing; Upper-Body IK achieves an 85% success rate on grasp-and-embrace rollouts.

    Source: §1 · p. 2, §6.1 · p. 7

  • Takeaways & Limitations

    Contact location and timing provide task-level interaction information across embodiments, while the robot can choose a suitable body configuration to realize active contacts.

    Source: §1 · p. 2

Abstract

from arXiv · show

Humanoid robots have many degrees of freedom and contact surfaces, yet most loco-manipulation research restricts manipulation to the hands and avoids other body contacts with the environment. To extend manipulation beyond the hands, we present PanTact, a system that learns humanoid whole-body dexterity, the capability to manipulate with any suitable body surface, directly from robot-free human demonstrations. Our key idea is to use contact as manipulation-centric information that transfers across embodiments. Contact locations and timing specify the interactions needed to accomplish a task, while the robot chooses how to coordinate its body to realize them within its own physical constraints. Specifically, PanTact captures, learns, and executes whole-body contact via: 1) a wearable, robot-free data collection interface that augments egocentric vision and bimanual UMI grippers with whole-body modular contact patches to record body contact locations and states; 2) a visuomotor policy that mitigates the egocentric observation gap and predicts whole-body contact goals; and 3) contact-aware whole-body control formulations that realize active contact goals using suitable robot surfaces. We deploy PanTact on a Unitree G1 humanoid without additional body contact sensors, demonstrating whole-body dexterity involving body support, pushing, sweeping, and load-sharing. Website: https://pantact.github.io

1 INTRODUCTION

Humanoid whole-body dexterity uses suitable body surfaces for manipulation, but loco-manipulation research often limits manipulation to the hands. PanTact transfers contact information from robot-free human demonstrations into a policy and controller that realize whole-body contact on a humanoid.

1 INTRODUCTION

  • Most loco-manipulation research limits manipulation to hands, leaving other humanoid contact surfaces underused.The paper defines whole-body dexterity as accomplishing manipulation tasks using any suitable body surface.Source: §1 · pp. 1–2
  • Existing demonstration pipelines capture end-effector or joint trajectories but do not explicitly preserve body-contact information through policy prediction and robot execution.The identified challenges are capturing contact location and timing, preserving contact across different human and robot proportions, and executing contact goals while maintaining motion tracking and balance.Source: §1 · p. 2
  • PanTact treats contact location and timing as task-level information that can transfer across embodiments while leaving the robot to select its body configuration.Tracking active contact goals without assigning them to specific robot links lets the controller choose a configuration suited to the robot while maintaining balance.Source: §1 · p. 2
  • The system combines robot-free contact-patch data collection, a policy predicting end-effector trajectories and contact goals, and three contact-aware whole-body controller formulations.The controller formulations are upper-body contact-aware IK, whole-body IK with a dense tracker, and direct sparse contact tracking.Source: §1 · p. 2
  • PanTact deploys on a Unitree G1 without additional body contact sensors, demonstrating body support, pushing, sweeping, and load-sharing.The demonstrations include grasping and body support to move differently sized objects simultaneously, leg-based non-prehensile manipulation alongside locomotion, and sequential contact switching.Source: §1 · p. 2

2 RELATED WORK

Prior work addresses whole-body sensing, robot-free data collection, and demonstration-based loco-manipulation, but the paper identifies gaps in transferring and executing body-contact information. PanTact combines wearable human contact sensing, policy contact prediction, and contact-aware control.

Whole-body sensing for robots

  • Prior whole-body sensing is primarily integrated into robots and paired with teleoperated demonstrations, whereas PanTact uses wearable human contact measurements to supervise robot contact-goal actions.The robot realizes those actions without additional body sensors.Source: §2 · p. 3

Data collection interfaces for robot learning

  • Robot-free and humanoid-oriented interfaces capture end-effector, keypoint, or joint motion, but the cited interfaces do not measure body-contact signals.PanTact adds contact sensing to preserve contact interactions during human-to-robot transfer.Source: §2 · p. 3

Learning loco-manipulation from demonstrations

  • Demonstration-based loco-manipulation commonly passes motion targets from policy to controller; cited contact-conditioned tracking assigns contacts to predefined body parts and lacks high-level policy contact prediction.PanTact integrates human contact sensing, policy contact-goal prediction, and contact-aware whole-body control.Source: §2 · p. 3

3 PANTACT DATA COLLECTION INTERFACE

PanTact’s wearable interface combines egocentric and wrist vision and bimanual grippers with modular contact patches to record human motion and body-contact signals. The recorded contact states and locations supervise policy predictions and indicate when the controller should track a contact.

3 PANTACT DATA COLLECTION INTERFACE

  • The interface records where and when body contact occurs alongside vision and motion, using egocentric-augmented bimanual UMIs and modular contact patches.Head-mounted stereo and wrist cameras record at 60 fps; head and wrist trackers record at 250 Hz, and gripper widths at 1 kHz, with streams aligned to 60 Hz.Source: §3 · p. 3
  • Identical sensing modules assemble into wearable patches of varying shapes and sizes for different demonstrators and body regions.The modules carry force-sensitive resistors, and the custom PCB streams up to 16 channels at 1 kHz.Source: §3 · p. 3
  • Sensor, connector, and strap modules snap together into patches of arbitrary shape and extent, including arm and knee patches.The figure describes modular contact patches and shows examples for the arm and knee.Source: Figure 2 · p. 3
  • Adaptive per-channel baselines binarize contact, while patch tracking maps sensor centers into 3D positions for policy supervision.The stored N × 4 contact array gives each sensor’s binary state and 3D position; the state indicates when the controller should track the location.Source: §3 · p. 4

4 CONTACT-PREDICTIVE WHOLE-BODY MANIPULATION POLICY

The policy predicts end-effector trajectories and whole-body contact goals from observations designed to reduce human-to-robot viewpoint and appearance mismatch. Its contact goals distinguish active contacts, which the controller should realize, from inactive contacts that need not reproduce free-space motion.

4 CONTACT-PREDICTIVE WHOLE-BODY MANIPULATION POLICY

  • The policy predicts world-frame end-effector and contact goals alongside a masked egocentric observation to mitigate the human-to-robot embodiment gap.Its observations combine a 3D egocentric representation, wrist images, and proprioception.Source: §4 · p. 4, §4.1 · p. 4

4.1 EMBODIMENT-AGNOSTIC OBSERVATION SPACE

  • Geometrically masking body regions reduces appearance and viewpoint mismatch without requiring an appearance model of either embodiment.The mask approximates body regions using the intersection of body-facing half-spaces defined by gripper frames and contact patches.Source: §4.1 · p. 4

4.2 CONTACT-PREDICTIVE ACTION SPACE

  • Each action step includes head look-at and end-effector targets alongside per-sensor contact states and positions.Only active contact goals request contact at their predicted positions; inactive goals do not require reproducing the corresponding sensor’s free-space trajectory.Source: §4.2 · p. 5

5 CONTACT-AWARE WHOLE-BODY CONTROL

PanTact’s controller tracks policy-predicted end-effector trajectories and active contact goals, letting suitable robot surfaces realize contacts without reproducing free-space body motion. It provides three control formulations with different capabilities and trade-offs.

5.1 GENERAL FORMULATION

  • Active contact locations are tracked only when their predicted state is active, and any suitable robot surface can realize each goal under robot-specific constraints.The controller converts end-effector trajectories and contact goals into whole-body motions.Source: §5.1 · p. 5
  • Virtual targets offset active contact points into the object so residual tracking error produces a pressing effort.The offset uses the pressing direction, nominal per-site force, design stiffness, and a maximum bound.Source: §5.1 · p. 5

5.2 WHOLE-BODY CONTROL STRATEGIES

  • Upper-Body IK jointly tracks end-effector and contact goals without training or simulation, but is limited to upper-body tasks with fixed legs.It assigns contact tasks to the three nearest candidate robot bodies and integrates joint velocities into arm targets.Source: §5.2 · p. 5
  • WBIK+SONIC extends contact tracking to lower-body motion and locomotion, but dense posture references can compete with contact accuracy and may be dynamically infeasible.The IK solver does not model SONIC’s execution limits.Source: §5.2 · p. 6
  • The Sparse Contact Tracker directly follows sparse end-effector and contact goals, choosing the remaining body motion rather than tracking a dense intermediate reference.It is trained with PPO in simulation using hindsight-labeled rollouts and domain randomization, at the cost of simulation training and dependence on the training distribution.Source: §5.2 · p. 6
  • Hindsight contact labeling derives achievable contact goals by sampling points on contactable body surfaces along expert-tracker rollouts, providing supervision without human annotation.Training samples active in-contact and pre-contact slots; the reward combines motion tracking, contact tracking, and regularization.Source: §5.2 · p. 6

6 EVALUATION

The evaluation tests contact transfer, upper-body non-prehensile manipulation, and lower-body manipulation with locomotion across three hardware tasks. Methods use the same demonstrations and matched test configurations; controller comparisons replay demonstrations in simulation.

6 EVALUATION

  • The evaluation probes cross-embodiment contact transfer and upper-body support or sweeping, with all methods trained on the same demonstrations and tested from matched configurations.The tested capabilities also include lower-body loco-manipulation and contact sequencing across body regions.Source: §6 · pp. 6–7, Figure 4 · p. 7

6.1 GRASP-AND-EMBRACE

  • 85% success with Upper-Body IK and 80% with Sparse Contact Tracker: the robot grasped a small object while supporting and moving a bulky object with its arms.Humanoid UMI consistently failed to support the bulky object because its controller tracked only end-effector commands, leaving arm posture unspecified.Source: §6.1 · p. 7, Figure 4 · p. 7

6.2 SWEEP-AND-STOW

  • 90.63% end-to-end success: WBIK+Dense Tracker swept objects into a bin and closed the cabinet, adapting to wrist, forearm, or gripper-side sweeps and knee or foot closures.Across 16 rollouts, occasional partial closures occurred when the controller prioritized balance over reaching the contact target.Source: §6.2 · p. 8, Figure 5 · p. 8

6.3 CARRY-AND-PUSH

  • 90% success: WBIK+Dense Tracker carried a box while walking and pushing a floor box with the lower body.Failures involved dropping the carried box and occurred more frequently with the Sparse Tracker and Humanoid UMI++.Source: §6.3 · p. 8, Figure 6 · p. 9

6.4 CONTROLLER COMPARISONS

  • Controller comparisons replay demonstrations in simulation using three whole-body control formulations, both with and without contact goals.The supplied description states the comparison setup but not its outcome.Source: Figure 7 · p. 9

7 CONCLUSION

PanTact learns humanoid whole-body dexterity from robot-free human demonstrations by transferring contact information from human to robot. Its system combines wearable contact sensing, contact-goal prediction, and whole-body controllers that realize active contacts.

7 CONCLUSION

  • PanTact learns humanoid whole-body dexterity from robot-free human demonstrations.Source: §7 · p. 9
  • Modular contact patches provide wearable, reconfigurable, low-cost sensing of body contact.Source: §7 · p. 10
  • An embodiment-agnostic policy predicts contact goals alongside end-effector trajectories, and three controller formulations realize active goals using suitable robot surfaces.Source: §7 · p. 10

A.1 RECORDED DATA

Each demonstration is recorded as a synchronized sequence combining egocentric sensing, end-effector state, and whole-body contact measurements.

A.1 RECORDED DATA

  • Each demonstration is a synchronized sequence of observations and measurements across time.The sequence length is Tdemo.Source: §A.1 · p. 13
  • The recorded visual and motion data include RGB images, egocentric 3D pointmaps, head pose, and left and right end-effector poses and gripper widths.Each end-effector pose is represented in SE(3).Source: §A.1 · p. 13
  • For each sensor, the sequence records binary contact, contact intensity, and contact location.The recording specifies the number of sensors and sequence length.Source: §A.1 · p. 13

A.2 POST-PROCESSING

Post-processing converts sensor readings into contact states and prepares synchronized visual and sensor sequences for training.

A.2 POST-PROCESSING

  • Contact is set when the baseline minus raw ADC count exceeds the larger of 0.35 times the baseline or a fixed noise floor.The resulting contact state is binary.Source: §A.2 · p. 13
  • Stereo depth is computed offline from 960 × 600 camera images, back-projected into a pointmap, and combined with the body mask.Depth uses the neural model provided with the ZED camera.Source: §A.2 · p. 13
  • Sensor streams share a monotonic clock, are aligned to the 60 Hz camera timeline, and are sampled at 20 Hz for training.Each training example uses an observation history h = 2 and action horizon H = 16.Source: §A.2 · p. 13

B POLICY DETAILS

The policy uses a diffusion architecture to predict actions from visual observations. The section also points to the paper’s control-strategy implementation details.

B POLICY DETAILS

  • The policy uses diffusion with a DiT denoising head and predicts actions using 16 DDIM inference steps.It is trained with ϵ-prediction and uses EMA weights for evaluation.Source: B POLICY DETAILS · p. 13
  • A pretrained DINO-v3 vision transformer is fine-tuned on egocentric and wrist images, while masked pointmap features receive sinusoidal 3D-coordinate encodings.The conditioned denoising transformer uses color-jitter image augmentation.Source: B POLICY DETAILS · p. 13

C CONTROLLER DETAILS

C.1 IK AND REFERENCE GENERATION

The control details combine inverse-kinematics objectives, sparse contact tracking, and deployment procedures to execute whole-body goals.

C.1 IK AND REFERENCE GENERATION

  • IK weights prioritize end-effector position and orientation and contact tasks, while whole-body IK adds pelvis and center-of-mass objectives.The contact-task weight is 250; whole-body IK uses horizontal center-of-mass weight 250 to encourage upright stability.Source: §C.1 · p. 13
  • In WBIK + SONIC, stepping begins after a 0.12 m pelvis offset from the support midpoint persists for 0.2 s.The swing lasts 0.4 s with a 0.06 m peak lift, and stride length is clamped to 0.10–0.25 m.Source: §C.1 · p. 14

C.2 SPARSE CONTACT TRACKER

  • Sparse trackers use eight upper-body contact slots or 16 whole-body slots, with four knee slots added for whole-body tracking.Valid slots request tight in-contact tracking or a softer pre-contact approach.Source: §C.2 · p. 14
  • The sparse tracker reward uses global-frame motion errors and end-effector tool-center-point errors, with narrow kernels for precise gripper tracking.Table 2 lists the reward terms and weights.Source: §C.2 · p. 14, Table 2 · p. 14
  • Training randomizes friction, restitution, base center of mass, joint biases, PD gains, link masses, and pelvis velocity perturbations.Static friction ranges from 0.3 to 1.6 and dynamic friction from 0.3 to 1.2.Source: §C.2 · p. 14
  • PPO training runs for 30,000 iterations in 4096 parallel environments, with adaptive actor learning rates and a KL target of 0.01.The critic learning rate is 10^-4.Source: §C.2 · p. 14

C.3 REAL-WORLD DEPLOYMENT

  • Asynchronous inference and low-level control discard expired action steps, replace overlapping targets with newer predictions, and resample commands at 50 Hz.Each predicted action chunk is timestamped at its source observation.Source: §C.3 · p. 14
  • Actions are expressed in the left-gripper frame and transformed to the world frame using pose estimates from Super Odometry.The estimates use the G1’s built-in head-mounted LiDAR-inertial unit.Source: §C.3 · p. 14
Loading 2610.12558v1…