PaperlayerView exact revision on arXiv

Source-linked AI summary

Whole-Body Dexterity

Conor Mc Gartoll, An Dang, Pranay Thangeda, Arjun Gupta, Carolina Higuera, Nima Fazeli, Mustafa Mukadam, Manikantan Nambi

arXiv:2610.22184v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Humanoid manipulation has largely focused on end-effectors or sustained grasps, leaving general object movement across body regions unaddressed. WBD trains grasp-based repositioning experts and distills their trajectories into a shared controller. The distilled policy deploys on a standing and walking G1, holding objects through leaning and walking during a shelf task.

  • Problem

    Prior humanoid work does not treat establishing and releasing contact with any body link to move an object across diverse body regions as a general capability.

    Source: §I · p. 2

  • Method

    WBD trains mode-constrained re-grasping experts from a whole-body grasp corpus and distills their trajectories into a shared controller conditioned on proprioception and object geometry.

    Source: §I · p. 2

  • Results

    The distilled policy deploys on a standing and walking G1, holding objects through leaning and walking during the shelf task.

    Source: §V · p. 8

  • Takeaways & Limitations

    The shared controller refines a coarse command stream into stable control and supports teleoperation.

    Source: §V · p. 8

Abstract

from arXiv · show

Current humanoid dexterous manipulation remains largely confined to end-effectors and simple grasps, falling short of humans' ability to engage the entire body by bracing and sliding bulky objects against the arms and torso. We introduce Whole-Body Dexterity (WBD), a general framework that uses the robot's body and appendages as contact surfaces to grasp and manipulate objects. Our sim-to-real pipeline first generates a dataset of feasible whole-body grasps, then trains reinforcement-learning experts on a general any-to-any repositioning objective by sampling initial and goal grasps from the dataset, and distills these experts into a single policy that uses proprioception and object geometry. We show that the distilled policy transfers zero-shot to a real Unitree G1, where it repositions a basketball and a cardboard box to regions around the torso and completes a long-horizon task that stows a box under one arm, carries it from a shelf to a table, and places it. The controller operates interactively, steered by coarse motion references from a human teleoperator. Across real-world whole-body manipulation tasks, the distilled controller completes 90% of trials, compared with 18% for direct retargeting and 4% for a generalist whole-body controller.

I. INTRODUCTION

Humanoid whole-body dexterous manipulation uses the body as a changing set of object supports, unlike hand-only manipulation. WBD learns to reposition objects between grasp modes across body regions and distills the learned behavior into a controller.

I. INTRODUCTION

  • WBD defines whole-body dexterous manipulation as repositioning objects by continually changing supporting body contacts.People brace boxes against the chest, cradle baskets across both arms, and tuck objects under one arm.Source: §I · p. 1
  • The pipeline generates whole-body grasps, trains sphere experts, warm-starts experts for other objects, and distills their trajectories into a general controller.The controller is intended for simulation and real-world tasks.Source: Figure 2 · p. 2
  • Prior humanoid work does not address general contact establishment and release with any body link to move objects across diverse body regions.Related work focuses on hand manipulation, end-effector dexterity, or sustained arm grasps on bulky objects, [2],,,, [6],,,,.Source: §I · pp. 1–2
  • Whole-body repositioning is harder than finger gaiting because humanoids have fewer independent supports and may need torso bracing during handoffs.Changing grasps can also require rapid reconfiguration and rolling or sliding contact.Source: §I · p. 2
  • WBD-EX learns mode-constrained re-grasping between left-arm, both-arm, and right-arm holds, then distills expert trajectories into a controller using object geometry.Transitions allow brief contact loss but require stable target-mode holds, supporting handoffs and repositioning behind the back or above the shoulders.Source: §I · p. 2

II. RELATED WORK

Prior work studies whole-body contact, loco-manipulation, and hand re-grasping, but the paper focuses on general object repositioning across humanoid body regions. It extends contact sequencing to the whole humanoid and situates teleoperation as a complementary but feedback-limited approach.

II. RELATED WORK

  • Prior bulky-object methods use arm and torso contacts, while soft-torso reinforcement learning re-grasps only in front of the torso and excludes standing or walking.Other work uses upper-body tactile sensing to hold boxes against the arms and chest on a real humanoid, while related methods address bulky objects,,, or torso re-grasping [23].Source: §II · p. 2
  • Whole-body dexterity has also described a snake-like arm manipulating with its entire body surface.Humanoid proprioception can provide a coarse whole-body tactile channel through joint-encoder readings under compliant contact [19].Source: §II · p. 2
  • Classical contact planning often assumes planar, quasi-static, or box-shaped settings, while this work focuses on general object manipulation across body regions.Other learning approaches expose whole-body contact as a commandable interface [27] or imitate human-object interactions in simulation, [29].Source: §II · pp. 2–3
  • Without force or tactile feedback, teleoperators may struggle to detect pinned or slipping objects, limiting prior teleoperation largely to hand-only pick-and-place.Prior systems transfer human motion through retargeting and shadowing,,, or whole-body target tracking,,,.Source: §II · p. 3
  • Prior dexterous-hand controllers use re-grasping and finger gaiting, including motion-command steering and goal-conditioned manipulation, [12], [45], [46].A constrained sampler over feasible contact-rich states has also guided goal-conditioned manipulation reinforcement learning [47].Source: §II · p. 3
  • This paper lifts the re-grasping principle from dexterous hands and arms to the whole humanoid.The related framework treats limbs and torso as counterparts to a hand’s fingers and palm.Source: §II · p. 3

III. METHODOLOGY

The methodology builds whole-body grasp data, trains object-specific repositioning experts, and distills them into deployable controllers. It treats links and link subsections as contact surfaces and accounts for sliding and rolling across the torso.

III. METHODOLOGY

  • The method defines contact surfaces as any robot link or link subsection, extending beyond end-effectors and front-of-torso contact.Whole-body re-grasping often involves sliding and rolling across the torso mesh.Source: §III · p. 3
  • The pipeline generates a large whole-body grasp dataset, trains per-object experts to reposition objects around the body, and distills them into controllers without privileged object state.Torso mesh subregions support diverse grasps, and expert training encourages sliding and rolling contact across the torso.Source: §III · p. 3
  • The final controller enables real-world whole-body object repositioning after distillation of the per-object experts.The described pipeline has three stages: grasp generation, expert training, and controller distillation.Source: §III · p. 3

A. Whole-Body Grasp Generation

Whole-body grasp generation specifies which appendages and contact regions hold the object, then optimizes and filters candidate grasps for stable contact. On the G1, torso subdivision broadens contact coverage, while WBD-EX training uses arm-based grasp modes.

A. Whole-Body Grasp Generation

  • A grasp mode abstracts a configuration type to the appendages contacting the object and serves as the level at which goals are commanded.The system model, object mesh, and selected contact links are the procedure’s inputs, so it is not limited to humanoids or a particular object class.Source: A. Whole-Body Grasp Generation · p. 3
  • The grasp-generation procedure takes the kinematic system, object mesh, and chosen contact links as inputs, making it applicable beyond humanoids and particular object classes.Configuration types specify which bodies or appendages hold the object and which links or subregions contact it.Source: A. Whole-Body Grasp Generation · p. 3
  • Subdividing the G1 torso into separate contact targets spreads grasps across its surface, but WBD-EX training excludes leg grasps and uses left-arm, both-arm, and right-arm modes.In a single-arm mode the other arm must release at the goal; in both-arm mode, both arms must contact the object.Source: A. Whole-Body Grasp Generation · p. 4
  • Grasp candidates are optimized by minimizing a differentiable force-closure residual over the object’s signed distance field while keeping links out of the object and one another.Candidates start from randomized object placements and are optimized in GPU batches,,,.Source: A. Whole-Body Grasp Generation · p. 4
  • Candidates must retain contact under 4× gravity and swept six-axis wrench disturbances, then drift less than 10 cm and rotate less than 45° under standard gravity.Accepted grasps are also filtered by torso-surface distance and a size-dependent height threshold.Source: A. Whole-Body Grasp Generation · p. 4

B. WBD-EX Training

WBD-EX learns to reposition objects between dataset-derived grasp goals using a goal-conditioned objective. Goals specify object position and grasp mode, allowing varied contact paths while requiring stable target holds and arm-to-arm handoffs.

B. WBD-EX Training

  • The goal-conditioned policy is trained with PPO to maximize expected discounted return over states, goals, actions, and transitions.The reward combines progress, settling, transition success, holding, arm neutrality, failure penalties, and regularizers.Source: B. WBD-EX Training · pp. 4–5
  • Goals sample initial and target grasps from a graph of object-position voxels and grasp modes, with compatible transitions requiring both-arm holds between left- and right-arm grasps.Commands advance toward randomly sampled far goals through legal-mode nodes; 20% of commands instead ask the policy to hold its current position and mode.Source: B. WBD-EX Training · p. 4
  • A goal-distance curriculum increases the maximum commanded transition from 2 cm to 8 cm while retaining shorter transitions.Goal distance is sampled from 2 cm to the current maximum.Source: B. WBD-EX Training · p. 4
  • A goal requires the object to settle at its target position in the required grasp mode, with the inactive arm clear for single-arm goals.Required arm contact may be lost briefly within a 0.48 s grace window, permitting dynamic reconfiguration.Source: B. WBD-EX Training · p. 4
  • Domain randomization varies object mass, center of mass, and friction and perturbs objects; sphere experts are also used to warm-start other primitives at lower amortized GPU cost.Warm-started experts achieve comparable or higher regional route completion than from-scratch experts.Source: B. WBD-EX Training · p. 5

C. WBD Controller Distillation

WBD distills WBD-EX repositioning trajectories into a controller that refines coarse commands. The controller combines conditional diffusion with inverse dynamics and conditions motion generation on proprioceptive history and object geometry.

C. WBD Controller Distillation

  • WBD-EX trajectories are filtered for validity, then balanced across objects and motion types with cross-body mirroring.Three equally sampled speed ranges use fixed goal distances of 2–4, 4–6, and 6–8 cm per trajectory.Source: C. WBD Controller Distillation · p. 5
  • A conditional diffusion model uses proprioceptive history and a five-dimensional object-geometry descriptor to generate keypoint motions, which inverse dynamics converts into joint-target increments.For real objects, the descriptor is computed from measured dimensions.Source: C. WBD Controller Distillation · p. 5
  • At inference, gradient guidance steers candidate motions toward coarse commands, and the selected motion is converted into refined joint commands without privileged per-step object state.A tunable guidance weight balances command tracking against the learned motion prior.Source: C. WBD Controller Distillation · p. 5

IV. EXPERIMENTS AND RESULTS

The experiments evaluate grasp and expert-data generation, then test the distilled controller in simulation and on a real Unitree G1. A shelf-to-table task assesses whether the manipulation skills compose over a longer sequence.

IV. EXPERIMENTS AND RESULTS

  • The evaluation covers grasp and WBD-EX data generation, simulated and real-humanoid on-body repositioning, and a longer shelf pick, stow, carry, and place sequence.The long-horizon task tests whether the skills compose into a manipulation sequence.Source: §IV · p. 5
  • The distilled WBD controller is evaluated for on-body repositioning in both simulation and on a real Unitree G1.The passage describes the evaluation scope, not its outcomes.Source: §IV · p. 5
  • Grasp and WBD-EX data generation are reported before controller evaluations and the long-horizon task.The passage does not provide numerical results for these evaluations.Source: §IV · p. 5

A. Experimental Setup

The setup uses a diverse grasp dataset and object-specific experts to train a shared controller, then evaluates command refinement on training and withheld object sizes. The simulation figure distinguishes training-object means from held-out sizes and compares WBD refinement with noisy and clean experts.

A. Experimental Setup

  • The dataset contains 2.51 million simulation-stable grasps, with mean per-object torso-position coverage of 90.1% and pooled coverage of 98.4%.The coverage figures use different distance tolerances: each object’s minimum half-extent for the mean and the mean minimum half-extent for pooled coverage.Source: A. Experimental Setup · p. 5
  • One expert is trained per object and their trajectories are distilled into a shared controller; sphere experts have the highest overall route completion, while other primitive experts succeed at lower rates.Less proficient experts yield valid demonstrations less efficiently under the trajectory quality filter.Source: A. Experimental Setup · p. 5
  • Figure 4 reports simulated object repositioning under noisy commands across transfers among six body regions, comparing WBD refinement with noisy and clean experts for training-object means and held-out sizes.WBD refinement is averaged over three training seeds. Bands show pointwise 95% paired bootstrap confidence intervals over 256 matched trials per object, and trials are capped at 600 s.Source: Figure 4 · p. 6
  • The recorded dataset contains 873 million frames, and training samples objects equally; in Fig. 4, trials are capped at 600 s and use 256 matched trials per object.Source: A. Experimental Setup · p. 6, Figure 4 · p. 6

B. Simulation Experiments

The simulation tests whether WBD can refine noisy commands while maintaining object support and route grasps across body regions. Despite command noise, WBD largely recovers clean experts’ mean goal-reaching performance.

Simulation Experiments

  • The command-refinement test trains on 13 of 15 primitives, withholds a 180 mm sphere and cube, and degrades expert streams using noise on all 14 arm commands.The tested command noise levels are α ∈ {2.5, 4}, across guidance weights from 1 to 350.Source: B. Simulation Experiments · p. 6
  • Regional trials span six body regions and adjacent-region sequences, with underarm and shoulder goals requiring opposite-arm release and success measured by completed voxel-mode goals.Fig. 5 illustrates a visible start grasp and a sampled far-goal grasp in another region.Source: B. Simulation Experiments · pp. 6–7, Figure 5 · p. 6
  • WBD largely recovers clean experts’ mean goal-reaching performance across training objects at both noise levels, while mean time to failure rises 70% on training objects and 158% and 71% on held-out sphere and cube.The time-to-failure gains are averaged across two noise levels and five guidance values.Source: B. Simulation Experiments · p. 7

C. Real-World Experiments

In real-world on-body repositioning and shelf tasks, WBD substantially outperforms direct GMR and SONIC. The trials also show that underarm stowing and maintaining support during transport remain important challenges.

Real-World Experiments

  • A single skilled teleoperator guided the trials with Xsens motion capture, while WBD refined arm targets to maintain object support as the arms and torso reconfigured.For the long-horizon task, the operator switched guidance weight between 10 for repositioning and 100 for pick-and-place.Source: C. Real-World Experiments · p. 7
  • 45 of 50 trials (90%) were completed by WBD, versus 9 of 50 (18%) for GMR and 2 of 50 (4%) for SONIC across three real-world tasks.The comparison counts complete on-body round trips for both objects and full shelf sequences.Source: C. Real-World Experiments · p. 7, Table I · p. 8, Table II · p. 8
  • Underarm stowing remained difficult and required a complete single-arm handoff; complete box round trips succeeded in 1/10 GMR and 0/10 SONIC trials, versus 10/10 for WBD.Source: C. Real-World Experiments · p. 8
  • All methods completed shelf pickup, but WBD maintained support through carrying and un-stowing after successful stows, while a few trials still ended in dropped objects.The shelf sequence also requires stowing the box under one arm, carrying it while walking, and placing it on a table.Source: C. Real-World Experiments · pp. 7–8

V. CONCLUSION

The section summarizes WBD’s mode-typed grasp and controller approach, its deployment on a standing and walking G1, and the evaluation protocol for regional route completion; it also identifies leg-contact grasps as future work.

V. CONCLUSION

  • WBD enumerates mode-typed whole-body grasps, trains reinforcement-learning experts with common rewards, and distills them into a shared controller for stable control from coarse commands.The controller supports teleoperation; although WBD-EX is trained without a lower-body controller, it deploys on a standing and walking G1. Future work would add leg contacts and enable behaviors such as bracing a box on a knee.Source: §V · p. 8

APPENDIX I WHOLE-BODY GRASP GENERATION DETAILS

  • The grasp generator adapts energy-based synthesis to prescribed contacts on arms, legs, and torso, with the torso serving as a fixed support surface and kinematic reference.It samples allowed contact links and points anew at each iteration, while checking penetration over all collision spheres.Source: Appendix I · p. 10, B. Geometry and Contact Sampling · p. 10

C. WBD Grasp Dataset

  • For WBD-EX, grasp generation specializes to arm-and-torso contacts, with fixed leg and waist joints and three contact configurations spanning right-arm and bimanual grasps.The dataset uses 626 candidate points across 34 links, samples five contact links per iteration, and tags torso points across six regions.Source: C. WBD Grasp Dataset · p. 11

D. Pre-Grasp and Post-Grasp Targets

  • Each solved grasp yields a retracted pre-grasp and pressed post-grasp target, and candidates enter the dataset only after two-stage dynamic simulation validation.The second stage retains settled poses only when object drift is below 0.10 m and rotation is below 45°; those settled poses are used at WBD-EX reset.Source: D. Pre-Grasp and Post-Grasp Targets · p. 11, F. Simulation Validation · p. 12

B. Position Voxelization

  • Across objects, the goal graph contains 1.0–5.5 × 10^3 occupied voxels and 1.7–9.1 × 10^3 (voxel, mode) nodes, spanning roughly ±0.21 m laterally and 0.03–0.45 m vertically.Elongated prisms reach the high end of the ranges because their settled centers span a wider area around the body.Source: B. Position Voxelization · p. 12

C. Goal Graph

  • The goal graph groups grasps by occupied position voxel and mode, permits mode changes within a voxel as zero-distance transitions, and samples intermediate and farther goals from legal nodes.Goal completion requires the object to be within 0.02 m of its target, settled, and in the required contact mode; single-arm goals also require the other arm to release contact.Source: B. Position Voxelization · p. 12, C. Goal Graph · p. 12, C. Goal Completion Criteria · p. 13

A. Observations and Actions

APPENDIX IV WBD-EX REGIONAL ROUTE COMPLETION

  • WBD-EX evaluation uses fourteen fixed routes through six named torso regions, with required grasp modes varying by region.The routes specify ordered region sequences, and evaluations apply the goal-completion and inactive-arm release requirements.Source: A. Body Regions · p. 14, B. Routes · p. 14

B. Routes

  • Regional route completion requires reaching every goal of a prescribed route in order; partial progress scores zero.Routes begin with an initial region and list subsequent regions as goals; fourteen routes are grouped by region.Source: B. Routes · p. 14, Table VII · p. 14
  • Evaluation reports route completion across fourteen routes and by visited body region for one expert per geometry class, with one training seed per object.The overall measure averages equally across routes; each route uses 2,048 trials, and results are reported at 50,000 training iterations.Source: Figure 9 · p. 15, Figure 10 · p. 15

C. Trials and Budget

  • Each checkpoint receives 28,672 trials across fourteen routes, with a 1,500-step limit per trial and termination at the first failure or completion.Evaluation preserves object-mass, friction, and center-of-mass randomization, freezes the goal-distance curriculum at its final value, and disables perturbations.Source: C. Trials and Budget · p. 15

D. Expert Evaluation Across Object Primitives

WBD-EX experts were assessed across fourteen routes, with performance varying by object geometry and the most demanding loops exposing especially low completion for some large objects. The section also sets up a matched-budget comparison of warm-started and from-scratch experts.

D. Expert Evaluation Across Object Primitives

  • Completion is highest for spheres and lowest for slender prisms and cylinders, while four-goal-loop completion is near zero for the 200 mm cube and two larger prisms.All experts still supply trajectories for distillation; improving success across shapes remains future work.Source: Table VIII · p. 15, D. Expert Evaluation Across Object… · p. 16

APPENDIX V WARM-START VERSUS TRAINING FROM SCRATCH

  • The warm-start study compares target-object experts initialized from a pre-trained sphere expert against same-object training from scratch.Source: Appendix V · p. 16

A. Setup and Matched Budgets

  • At matched 50,000 total iterations, warm-started experts receive 15,000 fewer target-object iterations; both conditions use the same recipe and are evaluated on all fourteen routes.Source: A. Setup and Matched Budgets · p. 16

B. Results by Body Region

Warm-started experts improve regional performance with object-dependent gains, while escapability-based filtering yields modest improvements and more usable training data. The appendix details the controller’s training pipeline and shows WBD refinement outperforming command smoothing at both tested noise levels.

B. Results by Body Region

  • The prism’s aggregate completion rose from 23.30% to 50.12%; cube 200’s four back routes rose from 0.3% to 12.5% with warm-starting.All six warm-started experts had higher mean regional route completion, though some regional scores were lower; the sphere’s loop completion was lower despite a higher overall mean.Source: B. Results by Body Region · p. 16, Figure 11 · p. 17

C. Training Cost

  • Warm-starting reduced aggregate training iterations by 20% across six objects, from 300,000 to 240,000, while all six final means were higher.A shared 15,000-iteration sphere pre-training cost is amortized across target experts.Source: C. Training Cost · p. 16

A. Escapability Scoring

  • Escapability measures the fraction of single-goal transitions from a grasp that reach the commanded position and grasp mode under the expert’s completion requirements.The study scored 2,510,864 grasps across fifteen objects using 128 sampled targets per grasp; slender objects had nearly flat distributions, while larger spheres had more grasps in the top band.Source: A. Escapability Scoring · p. 17, Figure 12 · p. 18

B. Removal Threshold Selection

  • Escapability-filtered datasets improved per-object mean completion by 1.2–13.3 percentage points over the full-dataset baseline at adopted cuts.Cuts preserved every occupied voxel–grasp-mode pair and at least 98.8% of average per-joint span at caps up to 70%.Source: B. Removal Threshold Selection · p. 17, Figure 13 · p. 18
  • Removal tolerance tracks the escapability distribution: flat-distribution objects gained little from small cuts, while spheres gained 5 to 13 points with 20–60% removal.Completion peaked and then declined at larger caps for the slender cylinder and two largest spheres; the adopted cap was the smallest within one point of the best mean, subject to stated coverage constraints.Source: B. Removal Threshold Selection · pp. 17–18

C. Trajectory Collection Yield

  • Across both route samplers, filtering raised usable trajectory yield from approximately 29.0% to 32.5% and usable frames per attempt from approximately 938 to 1,053.Accepted trajectories averaged approximately 3,200 frames under both conditions, while early failures fell from approximately 26.7% to 22.9%.Source: C. Trajectory Collection Yield · p. 19

B. Trajectory Collection; C. Sampling and Augmentation

  • The shared controller is distilled from frozen object-specific experts’ continuous goal-transition trajectories, with sampling balanced across route types, object geometry, goal distances, and body-region coverage.Training samples come from contiguous trajectory segments after failed-tail and stalled-interval removal; trajectories and derived samples are split by initial grasp for validation, and training data use mirroring and randomized dynamics.Source: B. Trajectory Collection · p. 19, C. Sampling and Augmentation · pp. 19–20

E. Command-Smoothing Comparison

  • WBD refinement improves both goals per trial and time to failure over EMA smoothing at both tested noise levels.EMA yields similar goal counts and shorter time to failure than the noisy expert; the comparison weights 13 training objects equally, with WBD values averaged over three seeds.Source: E. Command-Smoothing Comparison · p. 20, Table X · p. 20
Loading 2610.22184v1…