PaperlayerView exact revision on arXiv

Source-linked AI summary

COWBOY: A Scalable Sim-to-Real Framework for Learning Contextual Whole-Body Manipulation

Jiahui Yang, Jason Jingzhou Liu, Ray Muxin Liu, Andrew E. Wang, Kenneth Shaw, Deepak Pathak, Ruslan Salakhutdinov

arXiv:2610.28948v1cs.RO

TL;DR

Mobile manipulators must adapt to object and scene geometry while coordinating their base, arm, and camera from partial observations. COWBOY combines environment-aware reaching with reusable local RL experts and distills their behavior into a vision policy. It transfers zero-shot to varied real-world scenes, achieving 91% average grasp success across 24 scenes.

  • Problem

    Mobile manipulation policies need to generalize across spatial contexts as well as objects, despite partial scene observations and the difficulty of training RL across diverse environments.

    Source: §1 · p. 2

  • Method

    COWBOY uses a vectorized environment-aware WBC for reaching, local RL experts for object interaction, and distills composed teachers into a single vision-based whole-body policy.

    Source: §1 · p. 2

  • Results

    91% average grasp success across 24 real-world scenes, with zero-shot transfer to diverse scenes and objects unseen in simulation training.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The policy adapts base motion, camera viewpoint, arm trajectories, and grasp strategies to object and scene geometry without explicit conditioning on grasp type, base placement, or approach direction.

    Source: §5 · p. 8

Abstract

from arXiv · show

Mobile manipulators must adapt not only to objects, but also to the spatial context in which they appear: where to position the base, how to manipulate, and where to look while navigating surrounding scene geometry. To address this, we present COWBOY, a scalable sim-to-real framework for learning such contextual whole-body manipulation. On whole-body dexterous grasping, COWBOY transfers zero-shot to hundreds of objects and diverse in-the-wild scenes, spanning cluttered tabletops, shelves, drawers, and other unstructured settings. Rather than learning scene adaptation directly with RL in simulation, a vectorized environment-aware whole-body controller (WBC) handles reaching in thousands of scenes by coordinating base-arm motion, collision avoidance, and active vision. Then, local RL experts are trained in local scenes for object interaction and can be reused across thousands of full-scene layouts. We distill these composed WBC-RL teachers into a single vision policy that controls the mobile base, arm, dexterous hand, and actuated camera from point-cloud observations. The resulting policy exhibits contextual whole-body behavior, automatically adapting base placement, camera viewpoint, arm trajectory, and grasp strategy to observed scene and object geometry. Videos are available at https://cowboy-manip.com

1 Introduction

Mobile manipulators must adapt their whole body to object location and scene geometry, despite partial observability. COWBOY addresses the challenge by separating scene-aware reaching from local object interaction and distilling their composed behavior into a vision policy.

1 Introduction

  • Policies must generalize across spatial contexts as well as objects, coordinating the base, arm, and camera from partial scene observations.The paper calls this scene-conditioned capability contextual whole-body manipulation.Source: §1 · pp. 1–2
  • Directly training RL across diverse scenes is challenging because it requires global scene understanding, effective rewards, and coordination across high-dimensional actions.Recent sim-to-real systems have shown object-set generalization but typically train in one fixed setting and do not generalize across diverse environments [1] [2].Source: §1 · p. 2
  • A vectorized environment-aware WBC coordinates base-arm reaching, obstacle avoidance, and camera visibility across thousands of scenes, while local RL policies learn object interactions.The local policies are reused across thousands of full-scene layouts, then distilled with WBC behavior into one vision-based whole-body policy.Source: §1 · p. 2
  • COWBOY achieves zero-shot sim-to-real transfer for whole-body dexterous grasping across diverse in-the-wild scenes and hundreds of geometrically diverse objects.The policy coordinates approach direction, base placement, camera viewpoint, grasp strategy, and collision-avoiding motion from the surrounding scene.Source: §1 · p. 2

2 Related Works

Prior work advances mobile manipulation through demonstrations, RL, sim-to-real dexterity, and whole-body control, but the cited approaches leave challenges in scaling scene variation and whole-body planning. COWBOY targets these gaps with a single end-to-end policy for mobile manipulation.

2.1 Robot Learning for Mobile Manipulation and Whole-Body Control

  • Demonstration-driven systems learn coordinated base, arm, and end-effector behavior, but scaling demonstrations is difficult because trajectories must encode scene-dependent placement, visibility, and obstacle avoidance.Work includes whole-body teleoperation, articulated-object interaction, active perception, imitation, and household data collection [4] [5] [6] [7] [8].Source: §2.1 · p. 2
  • RL whole-body control spans imitation and keypoint tracking, residual learning, motion priors, teleoperation, and visual sim-to-real loco-manipulation [9] [10] [11] [12] [13] [14] [15] [16].Spin studies wheeled mobile manipulation combining perception, interaction, and navigation [17].Source: §2.1 · p. 2
  • Prior simulation-trained whole-body controllers show promise, but evaluate generalization under more constrained object–environment variation than COWBOY; related methods also condition base control on targets or learn modular navigation and grasping [18, 19, 20, 21, 22].The cited robot platform combines a holonomic base, arm, dexterous hand, and active-vision camera in a 32-DoF system.Source: Figure 2 · p. 3

2.2 Sim-to-Real for Manipulation

  • Sim-to-real dexterity work demonstrates transfer for in-hand reorientation, while newer modalities and tasks expand policy complexity but remain largely evaluated in fixed environments [26] [27] [28] [29] [30] [35] [36] [37].The cited directions include depth, point clouds, RGB, affordance-guided grasps, bimanual manipulation, and humanoid manipulation.Source: §2.2 · p. 3
  • ManipGen uses local policies but relies on a separate inference-time motion planner [33], whereas COWBOY distills local policies and whole-body motion into one end-to-end policy without a whole-body planner at inference.The paper motivates this distinction with unstructured scenes and partial onboard observations.Source: §2.2 · p. 3

2.3 Whole-Body Control for Mobile Manipulation

  • Optimization-based WBC and MPC handle richer constraints but require online optimization that is expensive to batch, while fabrics provide a fixed reactive computation graph for parallel simulation.Fabrics compose reaching, posture regulation, collision avoidance, and gaze control through metric-weighted task-space policies; the authors do not claim universal optimality over optimization-based WBC.Source: §2.3 · p. 3

3 COWBOY: Scaling Sim-to-Real for Contextual Whole-Body Manipulation

COWBOY decomposes contextual whole-body manipulation into environment-aware reaching and local contact-rich interaction, then distills their composed teachers into a vision policy. Its controller, reusable local experts, and full-scene distillation support scalable training across diverse scenes.

3 COWBOY: Scaling Sim-to-Real for Contextual Whole-Body Manipulation

  • COWBOY combines a vectorized environment-aware WBC, local RL experts, and full-scene distillation to train across thousands of simulated object–environment configurations.The resulting policy is deployed zero-shot in diverse real-world scenes and adapts base motion, camera viewpoint, and grasp strategy from observed geometry.Source: §3 · p. 4

3.1 Vectorized Environment-Aware Whole-Body Controller

  • The WBC takes an end-effector target pose and look-at point, coordinating base and arm reaching, collision avoidance, and camera gaze without an explicit base goal.Fabrics compose task-space behaviors through metric-weighted pullbacks and provide a fixed computation graph suited to GPU-parallelized simulation.Source: §3.1 · p. 4
  • Coarse-to-fine reaching lets the base handle distant motion, then prioritizes arm motion near the target for precise reaching.The base attractor vanishes when the target enters arm workspace, while the end-effector attractor activates near the local manipulation workspace; the full base-to-end-effector chain remains actuated.Source: §3.1 · p. 5

3.2 Contact-Rich Local RL Policies

  • Local RL policies learn contact-rich object interaction after WBC brings the end effector near the target, reducing the need for RL to learn scene-dependent reaching.Three grasp experts cover top-down, side, and constrained side grasps, with randomized objects and wall configurations encoding spatial constraints.Source: §3.2 · p. 5

3.3 Local-to-Full-Scenes Multi-Teacher Distillation

  • The distilled policy relies on onboard visual observations rather than accurate full-scene geometry and privileged object-state information.Training distills three teacher families across 8192 full-scene environments and 250 objects.Source: §3.3 · p. 6
  • The same local expert can be reused across full scenes when the local interaction region preserves its geometric constraints, while surrounding geometry and clutter are randomized.Expert families correspond to tabletop/drawer/box-like, elongated-object, and shelf-like interaction regions.Source: §3.2 · p. 5, §3.3 · p. 6
  • Full-scene teachers compose the WBC approach phase with local expert interaction, then online DAgger trains one student from their supervision.The student receives point clouds from an actuated-neck depth camera and base-mounted LiDAR and outputs base velocities plus arm, hand, and neck joint targets.Source: §3.3 · p. 6
  • COWBOY achieves 91% grasp success across 24 in-the-wild scenes and 100 objects.The figure reports per-scene success over 25 objects with three trials each.Source: Figure 5 · p. 7

4 Experiments

The experiments evaluate real-world grasping across varied scenes and objects, alongside simulated held-out settings and design ablations. Real-world perception tracks targets using segmentation with frequent policy updates.

Real-world evaluation

  • The real-world evaluation spans 24 scenes, using 20 fixed objects and five randomly sampled objects per scene, each tested in three orientations.Success requires lifting an object 10 cm without collisions.Source: §4 · p. 7

Real-world perception pipeline

  • Target positions update autoregressively at 60 Hz, with SAM3 segmentation corrections at 3 Hz when detections are available.The perception pipeline uses the segmentation mask and point-cloud centroid to estimate target position.Source: §4 · p. 7

Simulation and ablations

  • Ablations test policy observations and WBC variants on held-out simulation environments and objects.The evaluation includes 8,192 environments and 150 objects; WBC variants are assessed on grasp success, collisions, manipulability, and joint-limit margin.Source: §4 · p. 7

Real-world evaluation

  • Figure 6 shows two evaluation object sets, both unseen during training: 100 random objects and 20 fixed objects.The random set is on the left and the fixed set on the right.Source: Figure 6 · p. 7, §4 · p. 7

5 Results

COWBOY transfers to diverse real-world scenes and unseen objects, producing scene-conditioned grasping behaviors. Simulated evaluations and ablations assess its comparisons, observation design, WBC, and teacher-to-student transfer.

Real-world generalization

  • 91% average grasp success across 24 real-world scenes, including scenes not explicitly modeled during simulation training.Performance is lower in settings requiring side grasps, where attempts risk knocking objects over.Source: §5 · p. 8
  • COWBOY succeeds on everyday objects unseen in simulation, including objects with diverse geometries and irregular or deformable shapes.The results support zero-shot transfer to diverse real-world settings through scaling object–environment configurations and distilling WBC and local experts.Source: §5 · p. 8

Contextual whole-body manipulation

  • The policy adapts base motion, camera viewpoint, arm trajectories, and grasp strategy to object and scene geometry without explicit grasp-type or approach-direction conditioning.For example, in a sofa scene it grasps an elongated bottle from the side and a regularly shaped plushie from the top.Source: §5 · p. 8
  • The policy changes whole-body motion to avoid obstacles, including compensating with arm reach when base motion is blocked.These behaviors arise from distilling environment-aware WBC and local experts across diverse simulated scenes and objects.Source: §5 · p. 8

Comparison with vision-based policies

  • COWBOY outperforms alternative vision-based policies on all four scene categories in held-out simulation environments.Point-cloud end-to-end RL achieves 0% with obstacles and 96% without them; the comparison with learning baselines is summarized in Table 1.Source: §5 · p. 8, Table 1 · p. 8

Teacher performance transfer

  • The student closely matches teacher performance in tabletops, drawers, and boxes, while performance drops more noticeably in shelves.The shelf drop is attributed likely to heavier occlusion and the need for more precise obstacle-aware side-extraction motion; the student maintains low collision rates across scene types.Source: Table 2 · p. 8, §5 · p. 9

Policy observation design

  • Active vision expands the observable workspace, LiDAR supplies global scene context, and rendered partial observations support sim-to-real transfer.Training with ground-truth full point clouds performs well in simulation but fails in the real world.Source: §5 · p. 9, Table 3 · p. 9

WBC design

  • The full fabric WBC achieves the highest teacher grasp success and lowest collision rate while preserving strong manipulability and joint-limit margins.Table 4 reports these teacher metrics; without collision avoidance, fabric WBC still has frequent collisions.Source: §5 · p. 9, Table 4 · p. 9

6 Conclusion and Limitations

COWBOY combines environment-aware WBC, local RL experts, and full-scene distillation to train contextual whole-body manipulation at scale. Its grasping policy transfers zero-shot to diverse in-the-wild scenes, but scene context alone does not specify every functional grasp.

6 Conclusion and Limitations

  • COWBOY combines vectorized environment-aware WBC, local RL interaction experts, and full-scene distillation.The framework trains across thousands of simulated object–environment configurations.Source: §6 · p. 9
  • The learned policy transfers zero-shot to diverse in-the-wild scenes and adapts base motion, camera viewpoint, and grasp strategy to observed geometry.The conclusion characterizes these behaviors as contextual whole-body manipulation.Source: §6 · p. 9
  • Functional grasping may require conditioning beyond scene context because an object can admit multiple valid grasps for different downstream uses.The stated limitation concerns the grasp strategy inferred from scene and object context.Source: §6 · p. 9

A Whole-Body Controller Implementation Trade-offs

Geometric fabrics offer a reactive, fixed-computation-graph alternative suited to batched control, rather than a universally superior replacement for optimization-based whole-body control. The WBC combines task-space behaviors to coordinate reaching, base motion, posture, joint-limit and collision avoidance, and camera gaze.

A Whole-Body Controller Implementation Trade-offs

  • Geometric fabrics trade off against optimization-based WBC by enabling reactive collision-aware control with a fixed batched computation graph, not by being universally superior.Per-instance constrained IK and whole-body optimization solves, and horizon-based MPC optimization, are less naturally suited to thousands of parallel environments.Source: A Whole-Body Controller Implementation… · pp. 15–16

B.1 Motion Policies and Fabric Composition

  • A fabric represents a task-space behavior with a force and state-dependent metric, then pulls task terms back and combines them in the robot configuration space.Energization modifies acceleration along the velocity direction to make the dynamics compatible with a chosen energy function.Source: §B.1 · pp. 16–17

C.1 Fabrics WBC Configuration Space and Interface

  • The WBC accepts an end-effector target pose and look-at point, then outputs base-velocity and joint-position commands while accounting for local geometry.The mobile base uses a planar translation-and-yaw parameterization; the manipulation arm and camera arm are included, but the hand is controlled by a learned policy.Source: C Vectorized Environment-Aware Fabric… · p. 17, §C.1 · p. 17

C.2 End-Effector Attractor (EEA)

  • The end-effector attractor is gated off for distant targets, while its full-configuration Jacobian can also induce base motion as the target becomes reachable.This avoids stretching the arm toward unreachable targets.Source: §C.2 · p. 18

C.3 Base Attractor (BA)

  • The base attractor derives a planar base target from the end-effector goal and fades when the target enters the arm workspace, avoiding unnecessary base movement near the goal.Its target yaw faces the end-effector target.Source: §C.3 · p. 18

C.4 Posture Regularization (PR)

  • A weak posture regularizer biases the Franka arm toward a nominal configuration to resolve redundancy while allowing reaching, collision avoidance, and gaze terms to dominate.Its metric is much smaller than those of the end-effector and base attractors.Source: §C.4 · p. 19

C.5 Joint-Limit Repulsion (JLR)

  • Joint-limit repulsion is weak away from boundaries and becomes dominant near them, steering the arms away from configurations with poor joint-limit margin.It activates when a joint is near a limit or moving toward one.Source: §C.5 · p. 19

C.6 Body Repulsion (BR); C.7 Gaze Control Attractor (GCA); C.8 Fabric Energization

  • Body repulsion uses robot collision spheres and nearby geometry to induce whole-body motion for self-collision and scene-obstacle avoidance, while gaze control turns the camera toward a look-at point.The composed fabric acceleration is energized to stabilize WBC dynamics while preserving the behavior geometry of its constituent terms.Source: §C.6 · pp. 19–20, §C.7 · p. 20, §C.8 · p. 20

D Local RL Expert Training Details

The authors train local RL experts for contact-rich object interaction, leveraging the WBC for scene-aware reaching. Shared action and observation structures let each expert specialize in one interaction type and be reused in full-scene layouts.

D Local RL Expert Training Details

  • Local RL experts learn contact-rich object interaction while the WBC handles scene-aware reaching.Experts train in compact local scenes with the end effector initialized near the object.Source: D Local RL Expert Training Details · p. 21
  • Each expert specializes in one local interaction type, reducing its task to object interaction rather than scene-aware reaching.The local scenes keep the end effector near the object.Source: D Local RL Expert Training Details · p. 21
  • Shared action and observation structures allow the local experts to be reused across many full-scene layouts during distillation.Source: D Local RL Expert Training Details · p. 21

D.1 Environment Design

Local RL experts learn contact-rich object interaction in constrained local regions, and composed WBC–expert teachers are distilled across diverse full scenes into one policy. The student receives partial sensor point clouds and other observations and outputs unified full-body actions.

D.1 Environment Design

  • Local expert environments pair a target object and support surface with a geometric interaction region, and sample object sizes and resets to vary training conditions.The expert families cover top-down, side-grasp, and constrained side-grasp interactions.Source: §D.1 · pp. 21–22

D.2 Action Space Formulation; D.3 Observation Space Formulation

  • Experts share end-effector-frame actions and observations, with local scene geometry encoded to support reuse across different base placements.The actor observation has 61 dimensions; an asymmetric critic also receives privileged robot and task information.Source: §D.2 · p. 22, §D.3 · pp. 22–23

D.4 Reward Terms Formulation

  • Dense expert rewards encourage approach, contact, lifting, and movement toward an expert-specific goal; collision penalties are omitted because the WBC generates collision-free actions.The stated lift condition differs from task success, which requires lifting at least 0.10 m and moving the object to its goal.Source: §D.4 · pp. 23–24

D.5 Optimization Algorithm; D.6 Randomization

  • Experts are trained with asymmetric actor-critic PPO in 8192 Isaac Gym environments on one NVIDIA A6000 GPU, taking approximately 36 hours to converge.Training also uses domain randomization and randomized object wrenches.Source: §D.5 · p. 24, §D.6 · p. 25

E Full-Scene Multi-Teacher Distillation Details; E.1 Full-Scene Environment Design; E.2 Teacher Policies

  • Full-scene distillation combines WBC reaching with the corresponding local expert's contact-rich manipulation across diverse scene layouts.Each rollout switches from WBC reaching to the local expert near the object; expert-family-specific switching targets reflect the approach geometry.Source: E Full-Scene Multi-Teacher… · p. 25, §E.1 · p. 26, §E.2 · p. 26

E.3 Student Action Space

  • The student outputs one 32-dimensional action spanning the mobile base, Franka arm, LEAP hand, and ARX camera neck, trained to regress encoded teacher actions.Source: §E.3 · p. 27

E.4 Student Observation Space

  • The student observes partial point clouds from depth camera and LiDAR alongside robot state and a target-object position estimate.Cylindrical base crops provide broader scene context, while spherical object crops preserve denser local geometry; the final cloud contains 3072 points.Source: §E.4 · pp. 27–28
  • The student receives object-centered point-cloud crops transformed into the Franka-base frame, removing global translation and yaw variation while preserving local scene geometry.The target estimate can be initialized manually or from SAM3, which may be rerun at 3 Hz to recover from tracking failures.Source: §E.4 · p. 28

E.5 Student Policy Model Architecture and Distillation Objective

COWBOY distills scene-aware teachers into a point-cloud-conditioned student, with autoregressive object localization and simulated partial sensor observations supporting training across diverse scenes.

E.5 Student Policy Model Architecture and Distillation Objective

  • The student transformer maps point-cloud tokens, robot-state tokens, and an object-position estimate to full-body actions, with an auxiliary query updating the target estimate from geometry.The auxiliary query cannot attend to action queries; the predicted target estimate feeds the next observation and object crop, while training perturbs estimates to reduce reliance on perfect anchors.Source: §E.5 · p. 29
  • Multi-teacher online DAgger shares student weights across expert-specific full-scene environments, with teacher forcing for 10k steps and a linear decay over the next 10k.Training used 8 NVIDIA A6000 GPUs, 512 environments per rank, and approximately 60 hours.Source: §E.5 · p. 30

E.6 Object Localization Pipeline

  • The policy autoregressively updates its target-object position during execution, using ground-truth object positions to supervise the auxiliary head in simulation.SAM3 supplies the real-world initial estimate; it runs at 3 Hz and can overwrite the policy's 60 Hz updates when it returns a valid detection, while tracking continues if SAM3 fails.Source: §E.6 · p. 30

E.7 Randomization

  • Student distillation starts with domain-randomization magnitudes at their maximum and adds object teleportation without resetting the robot to train recovery from off-distribution states.Teleportation applies to a periodically resampled cached subset of environments with proportion ρtele = 0.2.Source: §E.7 · p. 30

F Vectorized Depth Camera and LiDAR Simulation

  • Point-cloud-based sensor rendering trades fidelity for speed and GPU memory, rendering depth and LiDAR observations across 1024 environments in 0.07 s and 0.25 s, respectively.The renderer converts batched full-scene point clouds into partial onboard observations rather than rendering directly from ground-truth meshes.Source: F Vectorized Depth Camera and LiDAR… · p. 31

F.1 Vectorized Depth Camera Renderer

  • The depth renderer projects full-scene points into camera coordinates, keeps the nearest point per pixel, and backprojects valid depths into a world-frame point cloud.A local min-depth operation inflates occluders to reduce background leakage through holes caused by sampled geometry.Source: §F.1 · p. 31

F.2 Vectorized LiDAR Renderer

  • The LiDAR renderer bins points by angular direction and keeps the closest range per bin to form a viewpoint-dependent first-return scan.A local-neighborhood range check suppresses likely leak-through returns while preserving empty bins; the fixed-size output is produced with batched operations.Source: §F.2 · p. 32

G.1 Policy Observation Ablation Details

The observation ablation retrains the student with three sensing variants while leaving the rest of the distillation pipeline unchanged, and evaluates them in simulation and the real world.

Ablation protocol

  • All three observation variants are retrained with the rest of the distillation pipeline unchanged and tested in simulation and the real world.Source: §G.1 · p. 32

No active vision

  • The no-active-vision baseline fixes the camera at its default pose, testing whether an actively controlled viewpoint is needed to maintain visibility during approach and interaction.The teacher also omits the camera gaze-attractor term in the fabric WBC.Source: §G.1 · pp. 32–33

No LiDAR input

  • The no-LiDAR baseline retains active vision but removes LiDAR input, testing its contribution to global context for base motion and collision avoidance.The stated cases of interest include structures outside the depth camera's field of view or minimum sensing range.Source: §G.1 · p. 33

No simulated rendering

  • The no-simulated-rendering baseline trains on complete ground-truth point clouds instead of viewpoint-dependent partial observations, testing the role of rendered sensor inputs in sim-to-real transfer.The real robot receives incomplete onboard point clouds.Source: §G.1 · p. 33

G.2 WBC Ablation Details

The WBC ablation replaces the controller used during both reaching and interaction, evaluating grasp success, collisions, and whole-body motion quality on held-out scenes and objects.

Evaluation setup

  • The evaluation uses 8,000 simulation environments and 150 held-out objects, reporting grasp success, collision rate, arm manipulability, and joint-limit margin.Higher joint-limit margin means the arm stays farther from its limits; higher manipulability indicates configurations better suited for local manipulation.Source: §G.2 · p. 33

Weighted IK

  • Weighted IK penalizes base motion more than arm motion, encouraging the arm to reach whenever possible.Source: §G.2 · p. 33

Fabrics WBC without collision avoidance

  • The fabrics WBC without collision avoidance retains end-effector and base attractors plus posture regularization, but removes self- and environment-collision avoidance.Source: §G.2 · p. 33

Full fabrics WBC

  • The full COWBOY WBC combines whole-body reaching, posture regularization, joint-limit repulsion, active gaze control, and self- and environment-collision avoidance.Source: §G.2 · p. 33

Motion-quality metrics

  • Arm manipulability is computed with Yoshikawa's measure using the Franka arm-only end-effector Jacobian, averaged over rollout timesteps and environments.Source: §G.2 · p. 33
  • Joint-limit margin is the distance from each Franka joint to its closest limit, averaged across arm joints, timesteps, and environments.Source: §G.2 · p. 33
Loading 2610.28948v1…