PaperlayerView exact revision on arXiv

Source-linked AI summary

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, Andy Zeng

arXiv:2204.01691v2cs.ROcs.CLcs.LG

TL;DR

Robots need physical grounding to use language models for complex, high-level instructions. SayCan combines language-model task knowledge with skill affordances, completing temporally extended tasks across 101 real-world kitchen tasks with 84% planning and 74% execution success.

  • Problem

    Language models lack physical-world grounding, limiting robots’ ability to extract and apply their knowledge to feasible actions in specific environments.

  • Method

    SayCan combines language-model selection of useful next actions with reinforcement-learned affordance functions that estimate which pretrained skills are executable.

  • Results

    84% planning and 74% execution success were achieved across 101 tasks, while affordance grounding reduced errors by half compared with FLAN.

  • Takeaways & Limitations

    Grounding language-model actions in robot affordances enabled a mobile manipulator to execute long-horizon, abstract natural-language instructions in a real kitchen.

  • Takeaways & Limitations

    The system’s primary bottleneck is the range and capabilities of its underlying skills, and it inherits limitations and biases from language-model training data.

Abstract

from arXiv · show

Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's "hands and eyes," while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.

1 Introduction

The paper addresses how robots can use LLM knowledge for physically grounded, high-level instructions despite LLMs lacking physical-world grounding. SayCan combines LLM action selection with learned affordance functions that identify executable actions in the world.

  • Motivation: LLMs contain broad knowledge about everyday tasks but lack physical-world grounding and feedback about the consequences of their generations.This gap can produce nonsensical, unreasonable, or unsafe interpretations for particular physical situations.
  • Approach: The paper studies how embodied agents can extract LLM knowledge to follow high-level textual instructions using learned atomic skills for low-level visuomotor control.The robot’s skills provide executable behaviors, while the LLM contributes task-level knowledge.
  • Approach: SayCan combines LLM task-grounding with learned affordance functions that determine which actions are useful for a goal and possible to execute.The LLM supplies the “Say” component, while the affordances supply the “Can” component.
  • Approach: Reinforcement learning trains language-conditioned value functions that represent the affordances of actions possible in the physical world.These value functions ground the LLM’s proposed plans in what the robot can execute.

2 Preliminaries

The preliminaries define large language models as autoregressive models whose scaling enables broad task generalization, and describe language-conditioned value functions for judging skill feasibility. In sparse-reward settings, these learned values correspond to affordance functions indicating whether skills can succeed from a state.

  • Large Language Models: LLMs model text probabilities autoregressively, predicting each successive string from its preceding context.The probability is factorized with the chain rule as p(W) = Π_n p(w_j|w_<j).
  • Large Language Models: Attention-based architectures enabled LLMs to scale to billions of parameters and terabytes of text, improving cross-task generalization.Examples include Transformers, BERT, T5, GPT-3, Gopher, LAMDA, FLAN, and PaLM.
  • Large Language Models: This work uses LLM semantic knowledge to identify useful tasks for solving high-level instructions.The language model supplies semantic knowledge while downstream methods determine task usefulness.
  • Value functions and RL: TD reinforcement learning learns value functions that predict whether a language-commanded skill is feasible in the current state.The formulation uses an MDP M = (S, A, P, R, γ), with state and action spaces, transitions, rewards, and discount factor γ.
  • Value functions and RL: Conditioning TD-learned values on language commands produces feasibility judgments that function as affordances under undiscounted sparse rewards.With terminal reward 1.0 for success and 0.0 otherwise, the learned value specifies whether a skill can be executed successfully from the given state.

3 SayCan: Do As I Can, Not As I Say

SayCan grounds language-model plans in a robot’s available skills by combining language-based task relevance with state-dependent skill affordances. It uses constrained candidate scoring and iterative skill selection to produce interpretable, executable action sequences for long, abstract, or ambiguous instructions.

  • Problem Statement: The system maps a long, abstract, or ambiguous instruction to a sequence of short robot skills, each paired with a language description and state-dependent completion probability.For skill π, the affordance function p(cπ|s, ℓπ) estimates successful completion from state s.
  • Connecting Large Language Models to Robots: Language-model knowledge alone may produce reasonable but physically unsuitable plans, such as suggesting that a robot purchase an apple rather than manipulate its environment.The method addresses the gap between high-level semantic knowledge and low-level robotic execution.
  • Planning: The language model scores fixed skill-label candidates, selects the highest-probability valid next step, and iteratively appends selected skills to the instruction.Planning is structured as a user–robot dialog in which the model produces an explicit action sequence.
  • Interpretability: Constraining the model to a fixed pattern preserves likelihood information across possible responses and makes the resulting plans interpretable.The prompt defines the skills available to the low-level policy and includes plan examples and user–robot dialog.
  • SayCan: SayCan forms an affordance space from value functions that estimate each skill’s probability of completion in the current state.This state grounding captures which available skills are feasible for the robot’s present observation.

4 Implementing SayCan in a Robotic System

SayCan represents executable skills with policies, value functions, and language descriptions, then combines language-model usefulness with affordance estimates to choose and execute actions iteratively. Its robotic implementation uses language-conditioned behavioral-cloning or reinforcement-learning skills on a mobile manipulator across diverse manipulation and navigation tasks.

  • Language-Conditioned Robotic Control Policies: Each skill pairs a policy, value function, and short language description, with policies trained through image-based behavioral cloning or reinforcement learning.The implementation follows BC-Z for behavioral cloning and MT-Opt for reinforcement learning.
  • Algorithm 1 SayCan: The system evaluates candidate skills, selects the highest-scoring one, executes it in the environment, updates the state, and repeats until the language output is “done”.The algorithm conditions scoring on the high-level instruction and previously selected skill descriptions.
  • Algorithm 1 SayCan: SayCan multiplies the LLM probability that a skill is useful with its affordance probability of successful execution, selecting an action that is both useful and possible.After each execution, the selected skill is appended to the response and the process repeats until termination.
  • Language-Conditioned Robotic Control Policies: A frozen sentence-encoder language model converts skill descriptions into embeddings that condition both the policy and value function.The embeddings specify which skill should be performed while the language model parameters remain fixed during training.
  • Robotic System and Skills: The mobile-manipulator system studies 551 manipulation and navigation skills spanning seven skill families and 17 objects.The skills include picking, placing, rearranging, opening and closing drawers, navigation, and placing objects in specific configurations.

5 Experimental Evaluation

PaLM-SayCan was evaluated on 101 instructions in real and mock office kitchens using plan and execution success metrics. It achieved strong performance, demonstrated long-horizon and interpretable grounded decision making, and extended to new skills, reasoning with negation, and multilingual queries.

  • Experimental Setup: The evaluation covered 101 instructions from 7 families, spanning single primitives to sequences of more than 10 actions and varying language complexity.Instructions were inspired by crowdsourcing, in-person kitchen users, ALFRED, and BEHAVIOR.
  • Performance: PaLM-SayCan achieved 84% planning success and 74% execution success in the mock kitchen, with real-kitchen performance reductions of 3% and 14%, respectively.The results indicate that the system and underlying policies generalize reasonably well from the mock environment to the full kitchen.
  • Long-Horizon Tasks: Long-horizon rollouts required contextual interpretation, multi-step planning, and navigation and manipulation across a substantial portion of the kitchen.Examples included interpreting “recover from a workout” as requesting healthy items such as water and an apple.
  • Interpretability and Grounding: Affordance-grounded decision making selected the correct skill sequence 84% of the time and executed it successfully 74% of the time, reducing errors by half compared with FLAN.Visualizing language completions alongside affordance estimates made the considered options and feasibility judgments interpretable, including correct sequence ordering.
  • New Skills: Adding drawer-manipulation skills yielded a 100% planning rate and 33% execution rate over 21 queries, with execution limited by chained manipulation-policy failures.Other instructions showed no loss in performance.
  • Reasoning and Multilingual Queries: Chain-of-thought prompting enabled handling of negation, while multilingual queries in Chinese, French, and Spanish caused almost no planning-success drop versus English.Vanilla SayCan’s limitation on negation was inherited from the underlying language models.

6 Open Source Environment

The authors open-source SayCan in a Google Colab notebook featuring a tabletop UR5 environment with randomly generated colored blocks and bowls. The implementation uses CLIPort for low-level pick-and-place actions and ViLD object detection for affordances.

  • Implementation: SayCan is open-sourced as a Google Colab implementation at say-can.github.io/#open-source.The environment is shown in Figure 8.
  • Affordances: Because CLIPort lacks a value function, affordances are implemented with a ViLD object detector.ViLD supplies the affordance mechanism for this policy.
  • Environment: The open-source environment uses a tabletop, UR5 robot, randomly generated colored blocks and bowls, and a CLIPort-based policy.CLIPort outputs pick-and-place locations.

7 Related Work

Related work grounds language models through environment inputs, action outputs, and prompt engineering, while language-conditioned behavior has commonly targeted low-level robotic instructions. Task-and-motion planning addresses sequencing tasks under embodiment constraints, with classical symbolic and optimization methods increasingly supplemented by machine learning.

  • Grounding Language Models: Recent language-grounding work trains models to accept environment inputs, directly output actions, or use prompt engineering to connect language with environments.
  • Learning Language-Conditioned Behavior: Prior research connects language and behavior through imitation learning and reinforcement learning, often for low-level instructions such as pick-and-place and manipulation primitives.
  • Task Planning and Motion Planning: Task-and-motion planning sequences tasks toward high-level goals while ensuring feasibility for a specific embodiment.
  • Task Planning and Motion Planning: Classical task-and-motion planning uses symbolic planning or optimization requiring explicit primitives and constraints, whereas machine learning enables abstract specifications, general primitives, relaxed constraints, and hierarchical long-horizon solutions.

8 Conclusions, Limitations and Future Work … C RL and BC Policies

The paper concludes that SayCan grounds LLM knowledge in robotic affordances through pretrained skills and value functions, while identifying LLM biases, skill limitations, and open directions for grounding and robot programming.

  • 8 Conclusions, Limitations and Future Work: SayCan grounds LLM knowledge for embodied tasks by conditioning natural-language action selection on pretrained skills and reinforcement-learned value functions.The value functions represent affordances—what is possible in the world—and constrain actions to be feasible and contextually appropriate.
  • 8 Conclusions, Limitations and Future Work: SayCan inherits LLM limitations and biases, while the range and capabilities of its underlying skills remain the system’s primary bottleneck.The method permits natural-language interaction, but performance is constrained by available skills.
  • 8 Conclusions, Limitations and Future Work: Future work could use grounded robotic experience to improve LLM factuality and common-sense reasoning, while exploring additional sources for scoring affordances.The passage specifically raises improving reasoning about real-world environments and physics.
  • 8 Conclusions, Limitations and Future Work: Natural language offers contextual and semantic cues and abstraction for robot programming, but requires supervision and may be less descriptive than alternatives such as hindsight goal images.The paper therefore questions whether natural language is the right programming ontology for robots.
  • 8 Conclusions, Limitations and Future Work: Further research could combine robot planning with language, use language models to pretrain policies, and integrate language with interaction in other ways.These are presented as extensions of SayCan’s factorization of language understanding and robotics.
  • A Version Control: Version v1 →v2 added PaLM results, new-capability studies, language-model-size ablations, an open-source simulated tabletop environment, and readability improvements.The new-capability studies covered drawer manipulation, chain-of-thought prompting, and multilingual instructions.
  • B.1 By Type: Contributors covered distributed robot-learning infrastructure, manipulation policies, data generation and curation, SayCan implementation, project management, evaluations, simulation scaling, writing, and an open-source environment.The passage lists contributors by type across these activities.
  • B.2 By Person: Person-specific contributions included deployment, logging, labeling, real-robot operations, offline RL, algorithm leadership, skill integration, experimentation, data collection, and research direction; corresponding emails are provided.The contribution appendix attributes these activities to named authors and gives {ichter,xiafei,karolhausman}@google.com as corresponding emails.

C.1 RL and BC Policy Architecture

The RL and BC policies adapt established visual-control architectures to language-conditioned robotic actions. RL incorporates action-state context for asynchronous control, whereas BC uses language-based FiLM conditioning without previous-action or gripper-height inputs.

  • RL policy architecture: RL models adapt an MT-Opt-like architecture for natural-language inputs by combining visual observations, language, robot action, and non-image state.The camera image is processed convolutionally, while the language instruction is embedded by the LLM and concatenated with action and state inputs such as gripper height.
  • RL policy architecture: RL inference supports asynchronous control by running while the robot executes the previous action and receiving the remaining execution amount.The conditioning input is processed through fully connected layers, tiled spatially, and added to the convolutional volume before further convolutional processing.
  • BC policy architecture: BC models adapt a BC-Z-like architecture, using a universal sentence encoder to FiLM-condition a ResNet-18-based visual policy.Unlike RL, BC omits previous action and gripper height, then applies multiple fully connected layers to final visual features to predict arm position, orientation, gripper, and termination actions.

C.2 RL and BC Policy Training

The paper trains language-conditioned policies through both reinforcement learning in simulation and behavior cloning from large-scale robot demonstrations. RL uses distributed prioritized replay, while BC combines teleoperated and filtered autonomous episodes with separate continuous- and discrete-action losses.

  • RL training: RL training uses simulated versions of the robot skills and environment, with RetinaGAN-transformed images to reduce the simulation-to-real gap.The language-conditioned RL policy is learned with MT-Opt in the Everyday Robots simulator.
  • RL training: The RL model trains on 16 TPUv3 chips for about 100 hours, using 3000 CPU workers for episode collection and another 3000 for target Q-values.Sparse 0-or-1 episode rewards produce a log-loss Q-function update, with prioritized experience replay.
  • RL training: Replay priority is defined as 1 + 10 · |p −0.5|, where p is the replay buffer’s average episode success rate.The priority was tuned to encourage episodes to be sampled proportionally to their priority.
  • BC training: BC training uses 68000 teleoperated demonstrations collected over 11 months with a fleet of 10 robots, augmented by 276000 autonomous episodes later success-filtered for training.Operators control the end effector through VR-tracked hand motion and the robot base through a joystick.
  • BC training: Language-conditioned BC policies use an MSE loss for continuous action components and cross-entropy for discrete components, with equal action-component weighting and image augmentations.The architecture builds on BC-Z and applies random brightness, contrast, and cropping augmentations.

C.3 RL and BC Policy Evaluations · D SayCan Details and Parameters · D.1 SayCan Details

The paper evaluates RL and BC manipulation policies through structured mock-kitchen trials before transferring stronger policies to the real kitchen, while SayCan combines language-model skill probabilities with robotic value functions for iterative planning. Across development, the evaluated skill set and policy capabilities expanded through repeated real-world evaluations and data collection.

  • C.3 RL and BC Policy Evaluations: Manipulation policies are evaluated separately by skill in randomized Mock Office Kitchen episodes using sampled object configurations.Evaluated skills include picking up, knocking over, placing upright, opening or closing drawers, and moving objects close to one another.
  • C.3 RL and BC Policy Evaluations: Each episode ends after 50 actions or when the policy samples a terminate action, with human-supervised robots and binary success or failure recording.Policies outperforming prior models are graduated to the same evaluation protocol in the real kitchen before integration into SayCan.
  • C.3 RL and BC Policy Evaluations: Higher success rates on mock stations usually correspond to higher success rates in the real kitchen despite the domain shift.The comparison spans mock stations and the actual kitchen counter and drawers.
  • D.1 SayCan Details: SayCan combines prompt-engineered language-model scoring with robotic affordances computed through value functions.The scoring approach, value-function computations, Algorithm 1, and Figure 13 together specify the system details.
  • D.1 SayCan Details: Given a high-level instruction and contextual examples, the language model outputs the probability of selecting each skill.The prompt-engineered context is used to query the scoring language model on the next skill choice.
  • D SayCan Details and Parameters: For iterative planning, SayCan appends the selected skill to the natural-language query and queries the language model again.This repeated querying process produces the next-step selection sequence.

D.2 Policies and Affordance Functions · D.3 LLM Prompt · E Experiments

SayCan grounds language-model skill selection by combining instruction relevance with embodiment-specific affordances, while prompt structure and simulation improve reliable planning. Its policies and affordances can mix learned, scripted, planning-based, and distance-based components across skill types.

  • D.2 Policies and Affordance Functions: The framework supports heterogeneous components: a multi-task language-conditioned policy for picking, a scripted gripper-state affordance for placing, and planning-based navigation aware of locations and distance.This flexibility allows policies and affordances from different methods to be mixed and matched.
  • D.2 Policies and Affordance Functions: SayCan combines language-model probabilities that a skill is useful with value-function probabilities that it can be executed successfully, selecting skills that are both possible and useful.The process repeats by appending each selected skill to the robot response and querying the model again.
  • D.2 Policies and Affordance Functions: Affordance handling calibrates value functions into probabilities and prevents skills from being selected after completion; navigation affordances use distance, while place is assigned paffordance place = 1.0.Terminate receives a small affordance value so planning can finish when no feasible skill remains.
  • D.2 Policies and Affordance Functions: Pick uses learned behavioral-cloning actions with reinforcement-learning value functions, go-to skills use classical planning to known object coordinates, and place uses Cartesian-space motion planning.The place policy is used consistently across objects because the available place-upright skill only applies to objects with a canonical upright direction.
  • D.3 LLM Prompt: The LLM prompt uses prompt engineering and a strict response structure, while a language-based simulator generates query-consistent affordances and distractors to test robustness without direct embodiment access.The simulator also verifies the resulting skill sequence.
  • D.3 LLM Prompt: Explicit numbered steps and separate lines improve prompt performance, whereas examples that overuse particular objects can bias scoring toward those objects.The guidance favors formatting each action independently rather than combining actions with phrases such as “and then.”
  • D.3 LLM Prompt: The prompt examples encode multi-step plans using ordered find, pick up, navigate, place, and done actions for tasks involving object transport, disposal, and sequential placement.Examples include bringing snacks, moving objects between locations, throwing items away, and placing multiple objects.

E.1 Tasks … E.5 Multilingual Queries

The evaluation spans task families, language-model-size ablations, drawer-manipulation skills, chain-of-thought prompting, and multilingual queries. Drawer tasks achieved perfect planning but limited execution, while the other sections define evaluation settings and prompting or model variations.

  • E.1 Tasks: SayCan evaluates single primitives, noun and verb substitutions, structured-language planning, varied embodiments, crowd-sourced requests, and long-horizon instructions.These task families test primitive recovery, synonym handling, explicit-sequence ablation, affordance reasoning, natural requests, and temporally extended reasoning.
  • E.3 Adding Skills: Drawer Manipulation: Drawer manipulation adds skills for opening, closing, inserting, and removing objects, using learned policies, behavioral-cloning actions, and heuristic value functions.Drawer tasks are considered possible when the robot is next to the drawer, and drawer-specific prompts teach skill chaining.
  • E.3 Adding Skills: Drawer Manipulation: 100% planning success and 33% execution success were achieved on drawer tasks.The main failures involved opening the drawer insufficiently for object placement or failing to close it completely.
  • E.4 Chain of Thought Reasoning: The chain-of-thought prompt generates rollouts by describing the robot, scene objects, locations, and explanations before listing action sequences.Examples cover single-skill requests, category-based retrieval, multiple sequential tasks, and object disposal.
  • E.4 Chain of Thought Reasoning: The rollout examples show SayCan resolving underspecified requests, sequencing multiple tasks, and translating goals such as disposal into find, grasp, move, and release actions.The examples include bringing snacks, selecting non-fruit food, moving rice chips and tea sequentially, and throwing away a redbull.
  • E.5 Multilingual Queries: Multilingual queries are evaluated with a plan-success-rate table whose queries 4–12 translate the first three queries into Chinese, French, and Spanish.The supplied passage identifies the table and translation structure but does not provide the numerical success rates.

E.6 Additional Results

Additional results show that SayCan supports interpretable, temporally extended execution, while failures arise from both language-model and affordance-model errors. The examples include successful multi-step plans and failures to recognize objects as pickable.

  • Long-horizon execution: The additional results include long-horizon execution sequences demonstrated in Figures 15 and 17.The figures direct readers to rollout videos on the project website.
  • Decision making: SayCan’s top combined score selects the correct skill, enabling interpretable decision making and successful temporally extended execution.Figure 14 visualizes the decision-making process.
  • Failure cases: In one failure, the affordance model failed to identify either bag of chips as pickable, although the language model approached the counter twice.The resulting behavior did not complete the intended pickup.
  • Failure cases: In another failure, the affordance model failed to identify the sponge as pickable.The associated plan was to find, put down, and bring the sponge.
  • Failure cases: 84% was the planning success rate; among errors, 65% resulted from LLM errors and 35% from affordance errors.These percentages are reported for the failure cases in Figure 16.
Loading 2204.01691v2…