Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

1 Introduction

• Because LLMs do not observe physical consequences, they may interpret instructions in ways that are infeasible or unsafe for a robot’s abilities and current environment.For example, a model might suggest using a vacuum cleaner that is unavailable or unusable by the robot.Source: §1 · p. 2
• SayCan weights the LLM’s estimate that a skill advances the instruction by its learned affordance for succeeding in the current state, producing an interpretable language plan.Skill descriptions constrain the LLM to the robot’s capabilities, while affordances ground its choices in the scene.Source: §1 · p. 2
• Across 101 real-world kitchen tasks, SayCan nearly doubles performance over non-grounded baselines and shows that enhancing the underlying LLM can improve robot performance.The mobile robot performed the language instructions zero-shot; the experiments also validated execution of temporally extended, abstract instructions.Source: §1 · p. 2

Large Language Models

• LLMs model text probabilities by predicting each successive string from prior strings, and Attention-based architectures [2] enabled scaling and cross-task generalization.The paper uses LLM semantic knowledge to determine useful tasks for solving high-level instructions.Source: §2 · p. 2

Value functions and RL

• TD reinforcement learning models an MDP and learns Q-values representing discounted rewards from a state and action under policy π.The preliminaries define the MDP through state and action spaces, transitions, rewards, and a discount factor; Q-values are learned from state-action data by approximate dynamic programming.Source: §2 · pp. 2–3
• Language-conditioned TD value functions assess whether a skill is feasible from the current state, and in the undiscounted sparse-reward case they express affordances.When success earns 1.0 at episode end and failure earns 0.0, the learned value function corresponds to an affordance function indicating whether a skill is possible in a state.Source: §2 · p. 3

SayCan: Do As I Can, Not As I Say

• Each skill has a short language description and an affordance function estimating its probability of successful completion from the current state.The affordance function is the skill’s value function when successful completion receives reward 1 and failure receives reward 0.Source: §3 · pp. 3–4
• Rather than relying only on generated plans, SayCan scores candidate skill descriptions with the LLM and chooses the most probable next skill for the instruction.Prompt engineering [11] can elicit a suitable response structure, but does not ensure that outputs are admissible skills or easy to parse.Source: §3 · pp. 3–4
• Even a sequence composed of available skills may be inappropriate for the current situation, such as seeking an apple when none is visible or one is already held.State-dependent affordances address this limitation by changing which skills are likely to succeed in the current environment.Source: §3 · p. 4
• SayCan selects the skill maximizing the product of its instruction relevance, p(ℓπ|i), and state-dependent success probability, p(cπ|s, ℓπ).The method treats these as task-grounding and world-grounding, respectively, to estimate whether a skill will make progress toward completing the instruction.Source: §3 · pp. 3–4
• After selecting a skill, SayCan executes its policy, adds its description to the instruction, and repeats until the LLM chooses a termination token such as “done”.The dialog-style sequence makes the plan interpretable while exposing likelihoods across candidate responses.Source: §3 · p. 4

Language-Conditioned Robotic Control Policies

• Skills use policies trained with behavioral cloning or reinforcement learning, while TD-backup value functions provide affordance estimates; BC policies currently succeed more often, but RL values are crucial for grounding capabilities in the scene.Multi-task BC and RL condition shared policies and models on skill descriptions to amortize training across skills.Source: §4 · p. 4
• SayCan selects each skill by combining the LLM’s probability that it is useful for the instruction with a value function’s probability that it can succeed, then repeats until termination.The chosen skill is executed in the environment, and the result informs the next planning step.Source: Figure 3 · p. 5, §4 · p. 5
• A frozen, pretrained sentence encoder embeds skill descriptions for the policies and value functions, allowing the planner and skill-conditioning encoder to use different language models.The planner can be suited to reasoning over many skills, while the encoder expresses individual skills more granularly.Source: §4 · p. 5

Training the Low-Level Skills

• Skill success receives a sparse reward of 1.0 or 0.0, assigned by human raters who label an episode positive when at least two of three agree it succeeded.Raters judge a video of the robot performing the skill alongside its instruction.Source: §4 · p. 5
• The system trains language-conditioned BC policies in the real world and RL policies in the Everyday Robots simulator, using RetinaGAN sim-to-real transfer and online data collection to improve RL performance.Simulation demonstrations bootstrap initial successes for the RL policies.Source: §4 · pp. 5–6

Robotic System and Skills

• The team proposed 551 manipulation and navigation skills across seven families and 17 objects, then used those most amenable to composition and planning and performing well at the current data-collection stage.Skill families include picking, placing, rearranging, drawer operation, navigation, and object configuration.Source: §4 · p. 6

5 Experimental Evaluation

• Evaluation covered 101 instructions across seven families, using a mobile manipulator in real and mock office kitchens.Instructions varied in time horizon, language complexity, and embodiment; three human raters assessed plan validity and execution success, with two agreeing for a success.Source: §5 · pp. 6–7, Table 1 · p. 6

5.1 Results

• PaLM-SayCan handled long-horizon tasks and produced interpretable skill choices, but struggled with early termination, negation, and ambiguous references.Examples show it using context to bring water and an apple for post-workout recovery, tracking previously delivered items, and choosing a sponge to clean a spill; 65% of errors were attributed to LLM failures and 35% to affordance failures.Source: §5.1 · pp. 7–8, Figure 5 · p. 7, Figure 6 · p. 8
• BC NL achieved 0% across tasks, BC USE achieved 60% on single primitives and 0% otherwise, while No VF and Generative reached 67% and 74% planning success versus PaLM-SayCan’s 84%.These comparisons show the limitations of the tested language-policy alternatives and of removing or changing affordance grounding.Source: §5.1 · p. 8, Table 2 · p. 9
• PaLM-SayCan achieved 84% planning and 74% execution success in the mock kitchen, and 81% planning and 60% execution success in the real kitchen.The real-kitchen results represent reductions of 3% in planning and 14% in execution relative to the mock kitchen.Source: §5.1 · p. 7, Table 2 · p. 9
• PaLM-SayCan achieved 84% planning and 74% execution success, compared with FLAN’s 70% planning and 61% execution success.The paper reports that PaLM-SayCan reduced errors by half compared with FLAN.Source: Table 3 · p. 9, §5.1 · p. 9

5.2 Case Studies of New Capabilities of PaLM-SayCan

• Adding drawer skills enabled 100% planning success and 33% execution success over 21 queries, with no loss in performance on other instructions.The execution failures were due to the chained manipulation policy.Source: §5.2 · pp. 9–10
• Chain-of-thought prompting [24] improved SayCan’s ability to handle negations and tasks requiring reasoning.The approach first generates an explanation and then uses it in the prompt when scoring candidate options; successful rollouts are shown in Table 4.Source: §5.2 · p. 10, Table 4 · p. 10
• PaLM-SayCan handled Chinese, French, and Spanish queries with almost no drop in planning success relative to English.The paper also notes that SayCan receives environmental feedback through value functions at the current decision step; Huang et al. [25] build on it with closed-loop planning using feedback such as success detectors, scene descriptors, or human input.Source: §5.2 · pp. 10–11

6 Open Source Environment

• The Colab implementation provides a tabletop with a UR5 robot and randomly generated colored blocks and bowls.Figure 8 shows the tabletop environment and CLIPort-based policy.Source: §6 · p. 11, Figure 8 · p. 11
• CLIPort outputs pick-and-place locations, while ViLD supplies affordances because CLIPort lacks a value function.The low-level policy is trained to output a pick and place location.Source: §6 · p. 11
• GPT-3 outputs steps in the structured form “pick up the object and place it in location”.This leverages the language model's ability to output code structures.Source: §6 · p. 11

Grounding Language Models

• SayCan grounds an LLM in the given environment through previously trained value functions, enabling general, long-horizon behaviors without additional training.Source: §7 · p. 11
• Prior language-grounding work trains models on environment inputs [30] [31] [32] [34], maps them directly to actions [35] [36] [37], or uses prompting [24].Huang et al. [23] also use prompting to extract temporally extended plans, but lack a mechanism to ensure grounding; these approaches are trained without physical-environment interaction.Source: §7 · p. 11

Learning Language-Conditioned Behavior

• Language-conditioned behavior has used imitation [51] [20] [13] [26] [37] and reinforcement learning [52] [53] [49] [54] [56] [21] [41], usually for low-level instructions, while SayCan grounds richer LLM knowledge for planning.Some prior work addresses long-horizon compound tasks in simulation [57] [58] [54]; SayCan instead extracts and leverages LLM knowledge through the robot’s affordances.Source: §7 · p. 11

Task Planning and Motion Planning

• Classical task and motion planning uses symbolic methods or optimization, which require explicit primitives and constraints; SayCan combines LLM semantics with general low-level policies for real-world abstract tasks.Learned approaches have explored abstract specifications, general primitives, relaxed constraints, and hierarchical solutions [71] [73] [74] [12] [54].Source: §7 · pp. 11–12

8 Conclusions, Limitations and Future Work

• SayCan combines skill value functions that represent real-world affordances with language-model knowledge to select feasible, contextually appropriate actions for real-robot tasks.The authors report evaluating it on long-horizon natural-language instructions in a real kitchen and note that improving the underlying language model can improve robot performance.Source: §8 · p. 12
• SayCan inherits LLM limitations and biases, while its skill repertoire and robustness constrain performance and high-value skill failures are difficult to handle.The authors suggest expanding and improving skills and using language-model prompting to correct failures.Source: §8 · p. 12
• Future work could use robotic grounding to improve LLM factuality and commonsense reasoning, and explore other grounding sources, including non-robotic contexts.The method currently uses generic value functions to score affordances.Source: §8 · p. 12
• The authors question whether natural language is the right ontology for robot programming: it conveys contextual cues and abstraction but requires supervision and may poorly describe some tasks.They contrast it with alternatives such as hindsight goal images.Source: §8 · p. 12
• Further research could combine robot planning with language, use language models to pre-train policies, or pursue other combinations of language and interaction.These are presented as possible extensions of SayCan’s approach to connecting language understanding and robotics.Source: §8 · p. 12

A Version Control

• The version-control note says v2 added PaLM results, studies of new capabilities and language-model size, a simulated tabletop release, and readability improvements.The listed capability studies covered drawer manipulation, chain-of-thought prompting, and multilingual instructions.Source: A Version Control · p. 18

B.1 By Type

• By-type acknowledgments cover distributed learning infrastructure, manipulation policies, SayCan design, evaluations, project management, paper writing, and the open-source environment.They also list data generation and curation among the contribution types.Source: §B.1 · p. 18

B.2 By Person

• By-person acknowledgments identify Ichter as SayCan algorithm lead, Hausman as project co-lead and SayCan developer, and Xia as on-robot lead, evaluator, and paper author.The listed roles illustrate the individual contributions described in the by-person section.Source: §B.2 · pp. 18–19

C.1 RL and BC Policy Architecture

• RL policies adapt MT-Opt [14], combining convolutional images with LLM-embedded language and state/action inputs to predict sigmoid-gated Q-values in [0, 1].For asynchronous control, inference runs while the previous action is still executing, and the model receives the time remaining for that action.Source: §C.1 · pp. 19–20
• BC policies adapt BC-Z [13], using universal sentence encoder [15] embeddings to FiLM-condition a Resnet-18 architecture that outputs arm position, orientation, gripper, and termination actions.Unlike the RL model, BC omits previous action and gripper height inputs because they were not necessary to learn the policy.Source: §C.1 · p. 20

RL training

• RL policies use MT-Opt [14] in an Everyday Robots simulator, with RetinaGAN [16] image transfer, simulation demonstrations for initial successes, and continued online data collection.Training also uses brightness and contrast augmentation and random cropping; padded-and-cropped images allow spatial shifts while preserving the field of view.Source: §C.2 · p. 20
• RL training uses 16 TPUv3 chips for about 100 hours and 3,000 CPU workers each for episode collection and target-Q computation; sparse 0/1 rewards use log loss and prioritized replay.Replay sampling is proportional to 1 + 10 · |p − 0.5|, where p is the replay buffer’s average episode success rate; priorities were tuned toward 50% success.Source: §C.2 · pp. 20–21

BC training

• BC uses 68,000 teleoperated demonstrations plus 12,000 successful episodes selected from 276,000 learned-policy episodes; unsafe, undesirable, or infeasible episodes are excluded.Operators use VR controllers to map hand motion to the robot’s end-effector pose and a joystick to move its base.Source: §C.2 · p. 21
• Real-world BC builds on BC-Z [13], using MSE for continuous actions and cross-entropy for discrete actions, evenly weighted, with about 27 hours of training on 16 TPUv3 chips.Inputs are downsampled to 256 x 320 for faster iteration, while affordance value functions use full-size images because half-size inputs performed worse for learning Q(s, a, ℓπ).Source: §C.2 · p. 21

C.3 RL and BC Policy Evaluations

• The mock-kitchen protocol evaluates skills separately, sampling 18–48 skills per skill category from three predetermined objects with randomized positions and one or two distractors.Source: §C.3 · p. 21
• Episodes end after 50 actions or a terminate action; a human operator supervises multiple robots, resets scenes as needed, and records success or failure.Source: §C.3 · p. 21
• Policies that outperform prior models per skill advance to the same real-kitchen evaluation protocol before integration into SayCan.Source: §C.3 · p. 21
• Higher success rates on mock stations usually corresponded to higher success rates in the real kitchen despite the domain shift.Source: §C.3 · p. 22
• Figure 11 tracks per-skill and average success rates as the project expanded from one instruction in April 2021 to hundreds at publication across 366 real-world evaluations.The figure caption attributes growth in performance and skills handled to continuous data collection and improving policy-training algorithms.Source: Figure 11 · p. 22, §C.3 · p. 22

D.1 SayCan Details

• SayCan combines prompt-engineered language-model scoring with value-function-based robotic affordances to connect high-level instructions to the robot’s available skills.The paper presents the components together in Algorithm 1 and Figure 13, alongside value-function computations at different states in Figure 2.Source: §D.1 · p. 22
• For iterative planning, the language model outputs probabilities for candidate skills, then receives the selected skill appended to its query before scoring again.Source: Figure 12 · p. 22

D.2 Policies and Affordance Functions

• SayCan mixes learned pick policies, scripted gripper-state place policies, and planning-based navigation, assuming object locations are known.The place policy follows a Cartesian motion-planned path to a supporting surface before releasing the object.Source: §D.2 · pp. 22–23
• SayCan combines the LLM’s estimate that a skill is useful with its value function’s estimate of successful execution, then repeats selection until termination.The selected skill is both possible and useful according to the two probabilities.Source: Figure 13 · p. 23
• SayCan can add skills by exposing them to LLM scoring and examples, but this evaluation omits skills unsuited to long-horizon tasks or too inconsistent for planning.Examples include “move object near object” and “knock object over” as unsuitable, and drawer opening as inconsistent.Source: §D.2 · p. 23
• Affordances are calibrated execution values, distance-based for navigation, set to 1.0 for place, and capped for completed skills; termination receives a small value.The system also prevents repeating a skill that has already been completed and rewarded.Source: §D.2 · pp. 23–24

D.3 LLM Prompt

• Prompt performance improved with numbered, line-separated steps and naturally phrased skill and object names; PaLM’s dialog format also generalized to imperative deployment instructions.Examples overly focused on objects used in planning could bias scoring toward those objects; user-generated instructions were taken as given.Source: §D.3 · p. 24
• 88% of solutions were recovered correctly with the final 17-example prompt, versus 82% after four examples in the language-based simulator.Without examples, planning succeeded on 54% of instructions without the done condition and 10% with it.Source: §D.3 · p. 24, Table 5 · p. 25

E.1 Tasks

• Task families test whether SayCan recovers a single primitive, handles noun or verb synonyms, and responds to crowd-sourced natural requests and long-horizon instructions.Crowd-sourced commands came from Mechanical Turk descriptions and people in an office kitchen.Source: §E.1 · pp. 25–26
• Structured-language queries spelling out command sequences compare with verb-synonym queries, while embodiment tests vary the environment and robot states.The structured-language condition ablates natural-language understanding; embodiment tests the affordance model and LLM reasoning within it.Source: §E.1 · p. 26

E.3 Adding Skills: Drawer Manipulation

• Drawer manipulation adds learned policies using behavioral-cloning actions and heuristic value functions, plus prompts for chaining open, close, put-in, and take-out skills.The heuristic treats drawer tasks as possible when the robot is next to the drawer.Source: §E.3 · p. 26
• Drawer tasks achieved 100% planning success and 33% execution success, with failures mainly occurring when the robot did not open the drawer wide enough or close it completely.The results are presented in Table 7.Source: §E.3 · p. 29, Table 7 · p. 29

E.4 Chain of Thought Reasoning

• The chain-of-thought prompt supplies scene objects and locations and illustrates explanations and action sequences for single-skill, object-moving, and multi-item requests.The passage gives example rollouts for commands such as holding a sponge, moving grapefruit soda, and bringing snacks.Source: §E.4 · pp. 29–30

E.5 Multilingual Queries

• The multilingual-query evaluation reports plan success rates, with instructions 4–12 translating the first three queries into Chinese, French, and Spanish.The supplied table caption does not give the success-rate values.Source: Table 8 · p. 30

• The top combined score selects the correct skill in an interpretable decision process that supports successful temporally extended execution.Source: Figure 14 · p. 31
• Figures 15 and 17 show long-horizon sequences.Source: Figure 15 · p. 32, Figure 17 · p. 34
• The affordance model failed to identify bags of chips as pickable despite the language model approaching the counter twice, and also failed to identify a sponge as pickable.Source: §E.6 · p. 33
• 84% planning success; of the errors, 65% were LLM errors and 35% were affordance errors.Source: Figure 16 · p. 33