Source-linked AI summary
Contact-Anchored Policies: Contact Conditioning Creates Strong Robot Utility Models
Zichen Jeff Cui, Omar Rayyan, Haritheja Etukuru, Bowen Tan, Zavier Andrianarivo, Zicheng Teng, Yihang Zhou, Krish Mehta, Nicholas Wojno, Kevin Yuanbo Wu, Manan H Anjaria, Ziyuan Wu, Manrong Mao, Guangxun Zhang, Binit Shah, Yejin Kim, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah
TL;DR
Robot policies often use language to generalize, although language can be too abstract for precise manipulation. CAP instead conditions policies on physical contact and uses EgoGym for rapid simulation-supported iteration. The resulting policies generalize zero-shot across novel scenes, objects, and embodiments on three manipulation skills using 23 hours of demonstrations.
Problem
Language conditioning is imprecise for the spatial awareness required in robotics and increases model-size requirements.
Method
CAP replaces natural-language conditioning with physical contact anchors and develops modular policies through simulation-in-the-loop iteration with EgoGym.
Results
CAP generalizes zero-shot to novel scenes, objects, and robot embodiments across Pick, Open, and Close using 23 hours of human demonstrations.
Takeaways & Limitations
Contact conditioning and simulation-supported development provide a framework for studying general manipulation policies with modest data, model size, and compute.
Takeaways & Limitations
Minor differences between simulation and reality can make it difficult to improve real-world performance by optimizing a simulation metric.
Abstract
from arXiv · showhide
The prevalent paradigm in robot learning attempts to generalize across environments, embodiments, and tasks with language prompts at runtime. A fundamental tension limits this approach: language is often too abstract to guide the concrete physical understanding required for robust manipulation. In this work, we introduce Contact-Anchored Policies (CAP), which replace language conditioning with points of physical contact in space. Simultaneously, we structure CAP as a library of modular utility models rather than a monolithic generalist policy. This factorization allows us to implement a real-to-sim iteration cycle: we build EgoGym, a lightweight simulation benchmark, to rapidly identify failure modes and refine our models and datasets prior to real-world deployment. We show that by conditioning on contact and iterating via simulation, CAP generalizes to novel environments and embodiments out of the box on three fundamental manipulation skills while using only 23 hours of demonstration data, and outperforms large, state-of-the-art VLAs in zero-shot evaluations by 56%. All model checkpoints, codebase, hardware, simulation, and datasets will be open-sourced. Project page: https://cap-policy.github.io/
1. Introduction
The paper argues that language is imprecise and costly for guiding precise physical behavior, and proposes contact-conditioned policies with simulation-supported iteration for general manipulation.
- Language provides imprecise spatial guidance for robotics and contributes to increasingly large model requirements.
- Contact-Anchored Policies replace natural-language task conditioning with physical contacts while jointly modeling observations and actions.CAP targets picking, opening, and closing behaviors.
- CAP trains general policies for picking, opening, and closing from 23 hours of human demonstrations and supports deployment across multiple robot embodiments.
- EgoGym provides a lightweight simulation benchmark for repeated modeling and dataset-curation iterations focused on zero-shot environment and object generalization.It trades photorealism for speed and uses shifted simulation success as a metric for general behavior.
2. Background
The background situates CAP within behavior cloning, autoregressive action modeling, and the broader movement toward robot utility models that generalize across environments, objects, or tasks.
- Behavior cloning learns a robotic policy by minimizing a supervised loss over human demonstration trajectories mapping observations to actions.
- Figure 2 depicts CAP training and inference as concatenating contact and visual tokens before action prediction.Training derives contacts by hindsight relabeling; inference derives them from a user click or VLM conditioned on a command.
- VQ-BeT uses a discrete action representation followed by an autoregressive transformer that predicts tokenized actions from observation sequences.The paper selects it because autoregressive conditioning is more straightforward and produces smaller, faster models than diffusion alternatives.
- Robot utility models target generalization across novel environments, objects, or tasks, while many large proprietary models emphasize task-level generalization.
3. Contact-Anchored Policies
CAP combines handheld RGB-D demonstrations, hindsight-labeled 3D contact anchors, conditional action prediction, and a lightweight simulation loop for developing policies across three manipulation tasks.
- Data Collection: A low-cost 3D-printed gripper supports both handheld data collection and robot mounting to reduce the embodiment gap.
- Data Collection: The dataset contains 20,365 demonstrations totaling 23.1 hours across 424 environments for Pick, Open, and Close.The task subsets contain 14,606 Pick, 3,690 Open, and 2,069 Close demonstrations.
- Policy Inputs and Actions: The observations combine resized RGB-D images and camera odometry, while actions comprise delta end-effector pose and gripper aperture.Horizontal-flip augmentation supports left-right symmetries such as cabinet doors.
- Contact Anchors: Contact anchors are 3D coordinates where the policy is expected to interact with an object.
- Hindsight Contact Labeling: Hindsight labeling detects contact, defines the anchor between the gripper fingers, and propagates it backward using recorded camera odometry.For later timesteps in tasks such as Pick or Open, the anchor is frozen and repeated after contact.
- Policy Model: CAP formulates conditional imitation learning over visual-input and contact-anchor histories and implements it with VQ-BeT.RGB and contact embeddings are projected to 256 dimensions and concatenated.
- Inference: At inference, a manually selected or VLM-selected RGB-D pixel is deprojected with depth and camera intrinsics into an initial contact anchor.The anchor is transformed into the world frame during execution and frozen after the gripper closes.
- Simulation-in-the-loop Development: EgoGym is a lightweight MuJoCo simulation suite that trades visual realism for scene diversity and execution speed during CAP development.Procedural scenes support checkpoint evaluation and failure-mode discovery across Pick, Open, and Close.
4. Evaluating CAP
CAP is evaluated zero-shot across unseen environments, objects, robot embodiments, and autonomous contact prompting, with strong task performance and broad transfer. Additional studies examine retries, baselines, compositional behavior, simulation-guided refinement, and contact-anchor importance.
- Zero-shot environment and object generalization: 83% in Pick, 81% in Open, and 96% in Close single-try performance on unseen environments with oracle contact prompts.Evaluations use zero-shot trials on unseen scenes, objects, doors, and drawers without additional fine-tuning.
- Autonomous contact prompting: 81% in Pick, 80% in Open, and 97% in Close single-try performance with VLM-generated contact anchors, comparable to human-oracle prompting.The same evaluation procedure is used, supporting autonomous contact-prompt generation.
- Automatic retries: 90% in Pick, 91% in Open, and 98% in Close performance with VLM verification and up to 10 automatic retries per trial.Most remaining failures arise from verifier false positives that classify failed tasks as successful.
- Cross-embodiment and external evaluation: The same CAP checkpoint transfers across Stretch 3, Franka FR3, XArm 6, and UR3e with comparable success rates after embodiment-specific gripper and inverse-kinematic adaptations.UR3e performs worst among the arms because of its particularly short forward reach; external evaluations broadly align with internal results.
- Baseline comparisons: CAP outperforms comparable baselines by 23% to 56% in the reported evaluations, including AnyGrasp, π0.5-DROID, and stretch-open.The cited comparison reports AnyGrasp at 47%, π0.5-DROID at 25%, and stretch-open at 58%.
- Contact-anchor ablation: Contact conditioning improves manipulation: CAP reaches 96% on Close, whereas its RGB-only ablation reaches 58%.The ablation tests Close because the open-door or open-drawer objective is visually apparent without ambiguity.
5. Related Works
General robot policies target novel scenes, objects, tasks, and robots, but current approaches require substantial data and compute. The paper motivates contact-based conditioning and fast, diverse simulation for evaluating generalization.
- General policies aim to operate across novel scenes, objects, tasks, and robots.
- Existing multi-task generalist models typically use 1,000 to 10,000 hours of data, while some single-task general policies use as few as 1,000 demonstrations per task.
- Policy conditioning has progressed from future states and images toward language as a way to communicate user intent or goals.
- Simulation is attractive because training and test losses poorly predict policy success, while statistically significant real-world evaluation requires onerous schedules.
- Minor sim-to-real differences can make optimizing a simulation metric fail to produce real-world improvement, motivating factored single-task simulation with many procedurally generated scenes.
6. Conclusion
The paper presents CAP as a contact-conditioned framework for general manipulation policies, developed with simulation in the loop and intended for resource-constrained research. It reports strong zero-shot performance on single tasks while identifying extensions for future work.
- CAP conditions general policies on physical contact and achieves superior zero-shot performance on single tasks with modest data, model size, and compute.
- The framework uses simulation in the loop and can chain CAPs through tool calling for long-horizon manipulations.
- CAP is positioned as a framework for researchers with limited resources to study general behavior in robotics.
- Future extensions include multiple contact anchors, bimanual tasks, and studying how CAP weights its two input modalities.
A.1.1. Visual gripper state estimation by SAM2
The visual gripper-state pipeline uses SAM2 segmentation to estimate gripper aperture from resized video observations.
- Video observations are resized to 256 × 256 before gripper-state processing.
- SAM2 uses positive gripper points and a negative outside point to generate masks across frames.
- Gripper aperture is estimated from the horizontal distance between left and right gripper centers and mapped from 0 fully closed to 1 fully open.
A.1.2. Data Filtering and Augmentation
The data pipeline filters minimally changing frames and augments Open and Close demonstrations through horizontal trajectory mirroring.
- Frames are retained when cumulative motion exceeds 0.3 cm translation, 0.1 radians rotation, or 0.05 gripper-aperture change.
- Open and Close demonstrations retain the original trajectory and add a horizontally mirrored copy.
A.2. Policy training details
CAP training uses configurable EgoGym environments with shared visual and proprioceptive observations, task-specific rewards, and simulation success metrics alongside loss. Evaluation covers unseen objects and environments across Pick, Open, and Close, with defined trial protocols and failure categories.
- Evaluation protocol: All evaluation objects and environments are unseen during training, with 10 trials per object or articulated object from 10 initial robot positions.Pick evaluation uses 25 objects, while Open and Close each use five doors and five drawers.
- EgoGym environments: EgoGym provides tabletop Pick, articulated-object Open, and articulated-object Close environments with configurable embodiments, action representations, and sampled object sets.The environments can use CAP or DROID embodiments and relative or absolute actions.
- Observations and rewards: Shared visual and proprioceptive observations are supplemented by target-object poses for Pick and handle poses for Open and Close.DROID additionally provides joint positions.
- Observations and rewards: Dense rewards measure vertical lift for Pick, opening fraction for Open, and residual distance to closed for Close.These task-specific rewards provide continuous signals aligned with each manipulation objective.
- Failure analysis: Pick failures are assigned by an ordered decision tree distinguishing insufficient lift, object contact without grasping, wrong-object contact, and empty grasps.Success requires maximum lift > 0.03m; intermediate lifting is categorized separately when maximum lift ≥0.005m.
- Training evaluation: EgoGym periodically reports simulation success rates alongside loss because loss alone is not a reliable indicator of policy performance.Example training-run plots cover Pick and Open.