PaperlayerView exact revision on arXiv

Source-linked AI summary

RT-Affordance: Affordances are Versatile Intermediate Representations for Robot Manipulation

Soroush Nasiriany, Sean Kirmani, Tianli Ding, Laura Smith, Yuke Zhu, Danny Driess, Dorsa Sadigh, Ted Xiao

arXiv:2411.02704v1cs.ROcs.AIcs.CLcs.CVcs.LG

TL;DR

Existing policy representations can leave manipulation guidance underspecified or provide over-specified context, while expanding diverse robot datasets is resource-intensive. RT-Affordance predicts key-stage robot poses as affordances and conditions a policy on them; it reports stronger task success than language conditioning and robustness to distribution shifts.

  • Problem

    Language can lack guidance on how to perform a task, while goal images and spatial plans can be over-specified or omit manipulation details; scaling diverse robot datasets is resource-intensive.

    Source: §I · p. 1

  • Method

    RT-Affordance predicts end-effector affordances from task language and uses them with language to condition a manipulation policy, training across robot trajectories, web data, and cheap in-domain affordance images.

    Source: §I · pp. 1–2, A. Affordance-conditioned policies · p. 3

  • Results

    Across real-world tasks, RT-Affordance achieved 69% overall success rate versus 15% for language-conditioned policies, with out-of-distribution performance within 10% of in-distribution evaluations.

    Source: §I · p. 2

  • Takeaways & Limitations

    The reported approach learns novel tasks using web data and cheap-to-collect affordance images without additional robot demonstrations.

    Source: §I · p. 2

    The policy did not generalize to completely novel motions or skills.

    Source: §V · p. 6

Abstract

from arXiv · show

We explore how intermediate policy representations can facilitate generalization by providing guidance on how to perform manipulation tasks. Existing representations such as language, goal images, and trajectory sketches have been shown to be helpful, but these representations either do not provide enough context or provide over-specified context that yields less robust policies. We propose conditioning policies on affordances, which capture the pose of the robot at key stages of the task. Affordances offer expressive yet lightweight abstractions, are easy for users to specify, and facilitate efficient learning by transferring knowledge from large internet datasets. Our method, RT-Affordance, is a hierarchical model that first proposes an affordance plan given the task language, and then conditions the policy on this affordance plan to perform manipulation. Our model can flexibly bridge heterogeneous sources of supervision including large web datasets and robot trajectories. We additionally train our model on cheap-to-collect in-domain affordance images, allowing us to learn new tasks without collecting any additional costly robot trajectories. We show on a diverse set of novel tasks how RT-Affordance exceeds the performance of existing methods by over 50%, and we empirically demonstrate that affordances are robust to novel settings. Videos available at https://snasiriany.me/rt-affordance

I. INTRODUCTION

RT-Affordance uses robot end-effector poses at key task stages as lightweight guidance, addressing the limited task guidance of language and the over-specification of goal images. Across real-world tasks, it reports a 69% overall success rate compared with 15% for language-conditioned policies.

I. INTRODUCTION

  • Expanding robot datasets to improve coverage is resource-intensive, while affordance reasoning offers a proposed bridge between web data and robot control.The paper connects affordance reasoning to semantic and spatial reasoning used in VQA and object detection.Source: §I · p. 1
  • Language descriptions often lack guidance on how to manipulate, while goal images are high-dimensional and spatial plans may omit gripper pose information.The introduction gives picking up a clothes hanger as an example where a plan may not specify the gripper pose.Source: §I · p. 1
  • RT-Affordance represents the robot end-effector pose at key task stages as visual affordances and uses a hierarchical model to predict them from task language before steering the policy.The model can train on robot trajectories and web-scale spatially labeled datasets, supplemented by cheap in-domain affordance images.Source: §I · pp. 1–2
  • 69% overall success rate versus 15% for language-conditioned policies; out-of-distribution performance was within 10% of in-distribution evaluations.The paper also reports learning novel tasks with web data and cheap-to-collect affordance images without additional robot demonstrations.Source: §I · p. 2

II. RELATED WORK

Related work spans affordance and grasp prediction, learning from non-action internet data, and intermediate interfaces for policy conditioning. RT-Affordance is distinguished by its affordance representation and its approach to connecting these sources with robot control.

Affordances for robot manipulation

  • Unlike prior grasp-prediction approaches, RT-Affordance does not rely on large-scale grasp-pose datasets, 3D point clouds, or simulation-based geometric planning.Related methods include data-driven grasp prediction and systems combining VLMs with affordance or grasp prediction and downstream policies [16] [24].Source: §II · p. 2

Learning pre-trained representations from non-action data

  • RoboPoint predicts spatial affordance points using simulated 3D scenes [52], whereas RT-Affordance uses a more descriptive representation without large-scale simulated scene generation.The section also places RT-Affordance among methods adapting internet data and human videos for affordance prediction [5].Source: §II · p. 2

Intermediate representations for policy conditioning

  • Prior policy interfaces range from task vectors and language to trajectories, goal images, sketches, and videos; RT-Affordance uses visual or textual affordances as an interface.The paper characterizes this interface as balancing flexibility, expressivity, and data efficiency.Source: §II · p. 2

III. RT-A: AFFORDANCE-BASED POLICY LEARNING

RT-Affordance is a hierarchical policy designed to provide a compact, expressive interface for manipulation, bridge knowledge from external datasets, and support learning from cheap in-domain data.

III. RT-A: AFFORDANCE-BASED POLICY LEARNING

  • The model predicts an affordance plan from task language and the initial task image, overlays the plan on the image, and conditions the policy on the overlaid images.Training combines web datasets, robot trajectories, and a modest number of cheap-to-collect affordance-labeled images.Source: Figure 3 · p. 3
  • RT-Affordance first proposes an affordance plan with an affordance generator, then generates actions with an affordance-conditioned policy.The stated goals are a compact, expressive interface, bridging external-dataset knowledge, and learning novel tasks through cheap in-domain data collection.Source: §III · pp. 2–3

A. Affordance-conditioned policies

The affordance-conditioned policy uses task language, the current image, and a plan of end-effector poses at critical trajectory stages. Affordances can be supplied as text or projected visually onto images.

A. Affordance-conditioned policies

  • An affordance plan is a sequence of end-effector poses at key trajectory timesteps, such as object contact or bottleneck states.The implementation extracts timesteps when the gripper changes between open and closed, or at the trajectory end.Source: A. Affordance-conditioned policies · p. 3
  • The policy conditions on both language and affordance plans because the plan supplies precise spatial information while language retains task context.The plans encode both end-effector position and orientation.Source: A. Affordance-conditioned policies · p. 3
  • Affordances may be tokenized as text or visually overlaid on images; the policy is trained with behavioral cloning and co-trained on web datasets.Visual projection outlines the robot hand at the specified poses, uses distinct colors to indicate temporal order, and assumes known camera intrinsics and extrinsics; text tokens are an alternative when those are unavailable.Source: A. Affordance-conditioned policies · p. 3

B. Learning to predict affordances

RT-Affordance learns to predict compact affordance plans from task language and an initial image, then uses them to guide manipulation. Plans can also be supplied by users, and execution can replan as conditions change.

B. Learning to predict affordances

  • Users can provide plans by marking them on the image without moving the robot or changing the scene; the plans are lower-dimensional than goal images or trajectory sketches.The predictor can instead generate plans automatically, avoiding user input at test time.Source: B. Learning to predict affordances · p. 3
  • The affordance predictor maps a task instruction and initial scene image to an affordance plan, trained on robot-trajectory labels and co-trained web data.When more data is needed, images with task labels can be annotated offline, manually or with tools such as VLMs and grasp planners, instead of collecting teleoperated demonstrations.Source: B. Learning to predict affordances · pp. 3–4

C. Model Inference

  • At inference, a human or predictor supplies a plan that is projected onto the image and passed with the language instruction to the policy.Plans may be updated at fixed or adaptive intervals to handle novel scenarios during execution.Source: C. Model Inference · p. 4

IV. EXPERIMENTS

The experiments examine whether affordance-conditioned policies work across manipulation tasks, support learning novel tasks without costly teleoperation, and generalize to novel settings. The implementation combines robot trajectories, web data, and inexpensive in-domain affordance images.

IV. EXPERIMENTS

  • The experiments ask whether affordance-conditioned policies work across diverse tasks, enable learning novel tasks without costly teleoperation, and generalize to novel objects, camera views, and backgrounds.The implementation uses the same web datasets as RT-2 for co-training.Source: §IV · p. 4, A. Experiment implementation · p. 4

A. Experiment implementation

  • All methods use the same number of robot trajectories and web datasets, with PaLM-E 2's 1-billion-parameter variant as the underlying model unless otherwise noted.The experiments use an RT-1 manipulator, Cartesian end-effector control, and a single head-mounted camera.Source: A. Experiment implementation · p. 4
  • The affordance predictor is trained on web data, hindsight labels from robot trajectories, and approximately 750 manually annotated in-domain images.The images are collected in about one hour and annotated in an additional two hours; the paper describes this process as faster and more scalable than collecting robot trajectories.Source: A. Experiment implementation · p. 4
  • Table I reports success rates under 30% for language- and goal-conditioned VLA baselines, versus 76% with oracle affordances and 68% with predicted affordances.The table caption identifies this comparison as a grasping evaluation.Source: Table I · p. 4

B. Learning to grasp novel objects efficiently

On unseen-object grasping, the predicted-affordance model achieves 68% average success, near the 76% oracle-affordance result. The experiments also report strong performance beyond grasping and graceful degradation in out-of-distribution settings.

B. Learning to grasp novel objects efficiently

  • The grasping benchmark tests unseen household-object categories with complex shapes among distractors, using five rollouts per category.The objects include dustpans, kettles, pots, boxes, and headphones.Source: B. Learning to grasp novel objects… · p. 4
  • RT-2 averages 28% success and GC-RT-2 averages 24% on the novel-object grasping benchmark.The reported failure modes include difficulty grasping at the appropriate location and insufficient end-effector rotation for pregrasp orientation.Source: B. Learning to grasp novel objects… · pp. 4–5

C. Beyond object picking

  • On tasks beyond grasping—including placing objects into various receptacles and manipulating articulated objects—RT-Affordance achieves 70% success, significantly outperforming RT-2, which achieves only 3% success on these tasks.Source: Table II · p. 5

B. Learning to grasp novel objects efficiently

  • With manually supplied pregrasp and goal poses, the affordance-conditioned policy reaches 76% average success on object categories absent from the robot trajectory datasets.Failures are attributed to small imperfections in following the supplied poses.Source: B. Learning to grasp novel objects… · p. 5
  • The full hierarchical RT-A variant predicts affordance plans before conditioning the policy on them and achieves 68% average performance, close to the policy conditioned on oracle affordances.Predicted affordances were reasonably placed around the pot handle, though some trials involved freezing or insufficient lifting.Source: B. Learning to grasp novel objects… · p. 5
  • Affordance prediction degrades gracefully in out-of-distribution settings relative to in-distribution evaluations.The evaluation assesses predicted plans on hundreds of test images, assuming the policy can follow them perfectly.Source: Figure 4 · p. 5

C. Beyond object picking

RT-Affordance extends beyond grasping to receptacle placement and articulated manipulation, achieving 70% success compared with RT-2's 3% on these tasks.

C. Beyond object picking

  • 70% success with RT-Affordance exceeds RT-2's 3% on tasks including placing objects into receptacles and closing a cubby.The tasks are described as unseen in the robot trajectory datasets and require precise spatial reasoning and execution.Source: C. Beyond object picking · p. 5

D. Robustness to out of distribution factors

This section evaluates affordance-prediction robustness on grasp tasks across in-distribution and out-of-distribution settings. Performance degrades by different amounts across OOD factors, with novel object instances posing the largest challenge.

D. Robustness to out of distribution factors

  • Figure 5 shows successful and incorrect predictions across in-distribution and OOD settings, marked green and red, respectively.Source: Figure 5 · p. 6
  • The grasp evaluation compares the training distribution with novel object instances, shifted camera views, and novel object textures.Source: D. Robustness to out of distribution… · p. 6
  • 77% of in-distribution trials were classified as successful; OOD performance fell no more than 10%, with 77% on novel camera views and a 3% average fall on novel backgrounds.The evaluation was an offline human assessment of hundreds of test images, assuming the policy could follow each predicted affordance perfectly.Source: D. Robustness to out of distribution… · p. 6
  • Novel object instances caused the largest performance drop, especially for kettles and boxes.Source: D. Robustness to out of distribution… · p. 6

E. Ablation study

The ablation study finds that both augmented affordance images and web datasets contribute to affordance-prediction performance, with the authors assigning different hypothesized roles to each.

E. Ablation study

  • Removing either the 750 augmented affordance images or web datasets reduced performance; the authors hypothesize web data supports robustness and augmented data supports downstream-task performance.Source: E. Ablation study · p. 6

V. CONCLUSION

RT-Affordance uses affordances as a hierarchical intermediate representation for manipulation, and the paper reports novel-task performance without additional human demonstrations. The authors also identify a boundary: their policy did not generalize to completely novel motions or skills.

V. CONCLUSION

  • The authors report that affordance-conditioned policies perform a wide range of novel tasks without additional human demonstrations.Source: §V · p. 6
  • RT-Affordance is a hierarchical method using affordances as an intermediate representation, providing precise spatial guidance that is easy for humans to specify.Source: §V · p. 6
  • Cheap-to-collect images can be used to learn affordance-prediction models during deployment, and the authors report that these models are robust.Source: §V · p. 6
  • The policy did not generalize to completely novel motions or skills; exploring this boundary is left for future work.Source: §V · p. 6
  • The authors propose exploring combinations of policy representations to share knowledge across representations.Source: §V · p. 6

Related papers

Loading 2411.02704v1…