PaperlayerView exact revision on arXiv

Source-linked AI summary

Code as Policies: Language Model Programs for Embodied Control

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, Andy Zeng

arXiv:2209.07753v4cs.RO

TL;DR

The paper addresses language grounding for robots without relying on extensive real-robot training data. It presents Code as Policies, which uses hierarchical code generation to produce executable robot policies, and reports improved code-generation performance alongside capabilities for spatial reasoning and instruction generalization.

  • Problem

    Existing language-grounding methods struggle with unseen instructions or require copious, expensive real-robot training data.

  • Method

    Code as Policies prompts code-writing LLMs to generate executable robot programs that process perception outputs and parameterize control APIs, using hierarchical code generation.

  • Results

    39.8% P@1 on HumanEval is achieved with hierarchical code generation, while CaP supports spatial-geometric reasoning and generalization to new instructions.

  • Takeaways & Limitations

    Code as Policies links language, perception, and action at a specific robot-stack layer while reducing the data requirements associated with end-to-end learning.

  • Takeaways & Limitations

    CaP is constrained by the descriptions available from perception APIs, the available control primitives, prompt examples, and the assumption that instructions are feasible.

Abstract

from arXiv · show

Large language models (LLMs) trained on code completion have been shown to be capable of synthesizing simple Python programs from docstrings [1]. We find that these code-writing LLMs can be re-purposed to write robot policy code, given natural language commands. Specifically, policy code can express functions or feedback loops that process perception outputs (e.g.,from object detectors [2], [3]) and parameterize control primitive APIs. When provided as input several example language commands (formatted as comments) followed by corresponding policy code (via few-shot prompting), LLMs can take in new commands and autonomously re-compose API calls to generate new policy code respectively. By chaining classic logic structures and referencing third-party libraries (e.g., NumPy, Shapely) to perform arithmetic, LLMs used in this way can write robot policies that (i) exhibit spatial-geometric reasoning, (ii) generalize to new instructions, and (iii) prescribe precise values (e.g., velocities) to ambiguous descriptions ("faster") depending on context (i.e., behavioral commonsense). This paper presents code as policies: a robot-centric formulation of language model generated programs (LMPs) that can represent reactive policies (e.g., impedance controllers), as well as waypoint-based policies (vision-based pick and place, trajectory-based control), demonstrated across multiple real robot platforms. Central to our approach is prompting hierarchical code-gen (recursively defining undefined functions), which can write more complex code and also improves state-of-the-art to solve 39.8% of problems on the HumanEval [1] benchmark. Code and videos are available at https://code-as-policies.github.io

I. INTRODUCTION

Code as Policies repurposes code-writing LLMs to translate language into executable robot policies that connect perception, logic, and control. Few-shot prompting and hierarchical generation support generalization, spatial reasoning, contextual parameter selection, and dialogue across robot tasks.

  • Language-grounding methods either struggle with unseen instructions or require copious real-robot training data.
  • Code as Policies uses code-writing LLMs to generate policy programs from natural-language commands, processing perception outputs and parameterizing control APIs.
  • Few-shot examples let LLMs re-compose API calls for new commands, while hierarchical generation recursively defines missing functions and expands policy logic.
  • Code-writing models can infer precise contextual values for ambiguous descriptions such as “faster” or “to the left,” and expose dialogue through a say(text) action.
  • Generated policies combine sequences, conditionals, loops, arithmetic, and third-party libraries to perform spatial-geometric reasoning and multi-step behaviors.
  • The formulation covers reactive policies such as impedance controllers and waypoint-based policies for pick-and-place or trajectory control across multiple robot systems.

II. RELATED WORK

Code as Policies differs from prior language-robot systems by generating executable policy code rather than only plans or relying on a fixed library of skills. Its hierarchical LMP framework composes functions, perception processing, and control primitives for adaptable robot behavior.

  • Prior language-robot work broadly emphasizes semantic interpretation, planning, or learned low-level policies, whereas this work focuses on code generation for expressive robot control.
  • Unlike plans that assume predefined skills for every step, CaP directly generates nested policy code that runs on the robot.
  • CaP combines feedback logic with parameterization of low-level primitives, reducing reliance on fixed, domain-specific skills or language-conditioned policies.
  • Hierarchical LMPs generate subprograms from comments and can recursively define functions, reuse existing functions, and accumulate policy logic over time.

A. Prompting Language Model Programs

Language model programs are prompted with API hints and instruction-to-code examples, enabling Python generation that uses arithmetic, libraries, perception functions, and robot control primitives.

  • Hints provide imports and type information that identify available APIs and guide their use.
  • Few-shot examples pair natural-language comments with solution code, allowing later instructions to refer to earlier prompt content.
  • Instruction-to-code examples teach basic variable retrieval, arithmetic, and list operations through ordinary Python syntax.
  • Third-party libraries: NumPy enables spatial reasoning by representing points as arrays and applying vector operations for movement, extrema, and centers.
  • First-party libraries: First-party perception and control APIs can be exposed through meaningful names in hints and examples, even when absent from training data.
  • First-party libraries: Robot examples combine detected object positions with displacement vectors to move objects toward specified targets.
  • Object grounding: Object references can resolve descriptions, categories, colors, and contextual phrases such as “other block” against a provided object list.

C. Example Language Model Programs (High-Level)

High-level LMPs combine control flow, perception queries, nested calls, and hierarchical function generation to translate complex instructions into reusable robot programs.

  • Control flows: While-loops can implement feedback policies that repeatedly move an object until a spatial condition changes.
  • Composed policies: Nested LMP calls let high-level policies reuse parsers and maintain iterative behavior while remaining within the model’s input-token limit.
  • Hierarchical function generation: LMPs can generate helper functions recursively, allowing rough code sketches to expand into layered implementations.
  • Hierarchical function generation: Undefined functions are created by parsing generated code, invoking a specialized function-generating LMP, and repeating the process depth-first.
  • Composed policies: A generated policy can query objects by spatial and visual properties, then repeatedly move qualifying objects toward the right.
  • Composed policies: The parser can invoke a hierarchically generated area-filtering function to complete an object query.
  • The approach is documented as relying on prompt engineering described in Appendix A.

D. Language Model Programs as Policies

LMP-based policies connect perception states to control actions by programmatically manipulating model outputs according to natural-language instructions.

  • Perception outputs can be transformed into parameters for low-level control APIs, grounding generated programs in a robot’s available action space.
  • LMP policies adapt to unseen instructions, generalize through open-vocabulary perception, and require no additional data collection or model training.
  • Because policies are represented as code, they can be interpreted, modified, and reused across tasks and behaviors.

IV. EXPERIMENTS

Experiments evaluate hierarchical code generation on robotics and general-purpose benchmarks, analyze generalization, and demonstrate robot-system flexibility, while quantitative robot evaluation remains constrained.

  • The experiments assess hierarchical generation across models, compare CaP with baselines in simulated manipulation, and demonstrate multiple robot systems.
  • RoboCodeGen: RoboCodeGen contains 37 robotics function-generation problems spanning spatial reasoning, geometry, and control.
  • RoboCodeGen: RoboCodeGen results report higher pass rates for hierarchical generation and larger models, with Codex models generally performing better.
  • Generalization: Hierarchical generation improves Productivity generalization most when new instructions require longer code or additional logic layers, but only for davinci models.
  • HumanEval: HumanEval evaluations show hierarchical code generation improves performance on general-purpose coding problems as well as robot policy code.

B. CaP: Drawing Shapes via Generated Waypoints

CaP is evaluated across simulated and real robot manipulation settings, using perception APIs and generated waypoint or primitive-based policies. The evaluations examine generalization to unseen instructions and attributes, while highlighting the approach’s dependence on available APIs and demonstrations.

  • B. CaP: Drawing Shapes via Generated Waypoints: The UR5e drawing domain generates and follows 2D waypoints from language commands, using MDETR object detection for perception and an end-effector trajectory API for actions.The setup includes four LMPs for command parsing, object parsing, waypoint parsing, and new-function generation.
  • D. CaP: Table-Top Manipulation Simulation Evaluations: The tabletop manipulation domain evaluates a UR5e arm with a suction gripper on object placement tasks using MDETR detections and a scripted pick-and-place primitive.The simulated benchmark includes eight long-horizon tasks and six new spatial-geometric tasks, with seen and unseen instructions and attributes.
  • D. CaP: Table-Top Manipulation Simulation Evaluations: On unseen tasks and attributes, CLIPort degrades significantly, whereas LLM-based methods retain similar performance and CaP outperforms direct language reasoning.CaP also remains competitive with supervised CLIPort on seen tasks despite using only one example rollout per task.
  • D. CaP: Table-Top Manipulation Simulation Evaluations: Natural-language planners are not applicable to tasks requiring precise numerical spatial-geometric reasoning, while code enables precise numerical computation.The paper reports this comparison alongside additional code-versus-language reasoning results.
  • V. Discussion and Limitations: A mobile-base robot with a 7 DoF arm demonstrates CaP on real-world kitchen navigation and manipulation tasks using ViLD detection and navigation and grasping APIs.The kitchen domain shows deployment across different robot systems.
  • V. Discussion and Limitations: CaP generalizes at the layer that interprets language, processes perception outputs, and parameterizes low-dimensional control inputs, but existing systems are brittle.The approach fits factorized perception-and-control systems and may require larger models trained on domain-specific code.
  • V. Discussion and Limitations: CaP is restricted by what perception APIs can describe, which control primitives are available, and the complexity or abstraction level of demonstrated examples.The method assumes instructions are feasible and cannot determine correctness a priori.

APPENDIX

The appendix details prompt-engineering practices, spatial-geometric reasoning benchmarks, and hierarchical code-generation evaluations. These results show that code-based LMPs support precise computation and that hierarchical generation improves coding performance, while reliability depends on prompt quality.

  • A. Prompt Engineering: Reliable LMP generation requires relevant, specific prompts whose examples and code are free of bugs.The appendix recommends verifying closely related examples and using syntax highlighting to reduce syntax errors.
  • A. Prompt Engineering: Consistent variable and function conventions help LMPs resolve ambiguous names and types across prompt examples.The appendix contrasts inconsistent NumPy-array and Shapely-point interpretations of the name point.
  • A. Prompt Engineering: Explicit third-party imports improve reliability and increase the chance that generated code uses libraries such as NumPy and SciPy.Meaningful first-party function names and specified return formats also guide library usage.
  • A. Prompt Engineering: Code-generation failures include incorrect API calls and variable-type assumptions, which can sometimes be addressed with hints, examples, descriptive names, or concise logic.The authors found automated code editing inconsistent and did not use it in experiments.
  • C. Reasoning with Code vs. Natural Language: The robotics code-generation benchmark tests object selection and position selection from spatial-geometric descriptions.Position-selection answers are correct only when all coordinates fall within 1cm of ground truth.
  • C. Reasoning with Code vs. Natural Language: LMPs achieve accuracies in the high 90s, outperforming Chain of Thought, which outperforms Vanilla reasoning.Code supports precise and multi-step numerical computations through Python and external libraries such as NumPy.

E. Robot Code-Generation Benchmark

The benchmark evaluates robot code generation across five generalization types, comparing flat and hierarchical prompting and generation across three models. Hierarchical generation improves performance for the davinci models, especially on longer, more multi-layered programs.

  • Example Questions: The benchmark tests vector operations, simple controls, shape manipulation, and first-party library use through code-generation questions.The examples include NumPy interpolation, PD control, Shapely shape construction, and imported object-geometry functions.
  • Generalization Analysis: The five generalization types are Systematicity, Productivity, Substitutivity, Localism, and Overgeneralization.They compare prompted examples with new instructions and solutions that recompose, extend, substitute, repurpose, or introduce unseen elements.
  • Generalization Analysis: code-davinci-002 performs best across all generalization types, while code-cushman-001 is especially weak on Productivity and Localism but stronger on Substitutivity.The relative model rankings remain consistent across the generalization categories.
  • Generalization Analysis: Substitutivity is strong across models, whereas Productivity and Localism are more challenging forms of generalization.The analysis associates the former with replacing similar words or categories and the latter with longer answers or maintaining local structures.
  • Generalization Analysis: Hierarchical code generation significantly improves both davinci models, especially on Productivity, but does not improve cushman and lowers its Substitutivity performance.The hierarchical approach helps davinci models write longer and more multi-layered functions.

F. CaP: Reactive Controllers for Toy Tasks

CaP generates simple reactive controllers for cartpole balancing and end-effector impedance control from prompts. These controllers work under specified interfaces but remain limited in complexity, initialization, and execution-feedback tuning.

  • Reactive Controller Generation: CaP generates cartpole and end-effector impedance controllers zero-shot using the same prompt structure.The authors attribute this to knowledge stored in the language model, invoked when the function signature sufficiently specifies the objective.
  • Cartpole Balancing: The cartpole controller uses position, pole angle, and velocity inputs to choose discrete left or right actions that stabilize an unstable fixed point.It requires initialization near upright and actuates the cart beneath the pole.
  • Impedance Control: The impedance controller can control a UR5e robot in PyBullet but omits Coriolis and gravity compensation.The function uses position and velocity errors, gain matrices, and the Jacobian to compute joint torques.
  • Interface Dependence: Hints specifying the expected direction output and PD control are necessary for the generated controller to match the environment API.Without these hints, the resulting function may appear reasonable while producing an incompatible control output.
  • Limitations: More complex continuous controllers may require tuning gains from execution feedback, which CaP does not currently support.The experiments demonstrate simple reactive controllers, while more complex controllers remain an open requirement.

H. Whiteboard Drawing

In the whiteboard domain, CaP translates natural-language drawing commands into 2D waypoint trajectories and robot actions for drawing and erasing shapes. Its prompted functions combine object perception, geometric parsing, transformations, and interaction APIs.

  • Whiteboard Setup: A UR5e robot draws and erases natural-language-described shapes on a whiteboard with a marker attached to its end-effector.The setup also supports voice interaction through speech-to-text and text-to-speech APIs.
  • Prompted Functions: The whiteboard prompt hierarchy includes interfaces for command parsing, object-name parsing, shape-point parsing, point transformations, and function generation.These functions convert language descriptions into reusable geometric subprograms.
  • Perception and Geometry: Perception APIs provide available object names and 2D object-center positions for grounding geometric instructions.Additional tabletop-style APIs expose bounding boxes, segmentation masks, colors, corners, sides, and coordinate denormalization.
  • Robot Actions: The draw primitive follows a sequence of 2D points after moving above the first waypoint and detecting contact with the whiteboard.The erase primitive similarly establishes contact with a hardcoded eraser position before following its trajectory.
  • Demonstrations: Demonstrated instructions include constructing, transforming, erasing, and spatially relating multiple shapes and objects across a sequence.Examples include enlarging and rotating a square, drawing pyramids, and drawing circles or squares around blocks and fruit.

J. Mobile Robot

CaP is deployed on an Everyday Robots mobile manipulator for navigation and object manipulation in a real office kitchen. Prompted functions combine scene parsing, geometric trajectory transformation, navigation, perception, grasping, placement, and dialogue.

  • Mobile Robot Setup: The mobile platform combines a mobile base with a 7DoF arm and uses an RGBD camera for perception in a real-world office kitchen.The setup is shown in Fig. 6.
  • Prompted Functions: The mobile prompt hierarchy includes interfaces for command parsing, object and position parsing, trajectory transformation, and function generation.These functions support language instructions that require geometric navigation or manipulation.
  • Navigation: Navigation APIs expose visible objects, predefined locations, object and location poses, robot pose and heading, and motion-planning calls.The robot can navigate either to a named location or to a 3D position.
  • Manipulation: Manipulation APIs support picking visible objects and placing held objects at 3D positions or on named objects.Picking uses ViLD object detections and scripted grasping primitives.
  • Demonstrations: The demonstrations include repeated geometric navigation, searching for objects while moving, object counting, pick-and-place, and spoken responses.CaP can also retain past robot positions in the Python execution scope as short-term memory for later commands.

K. Simulation Tabletop Manipulation Evaluations

The simulation evaluates Code as Policies on tabletop rearrangement tasks using scripted perception and pick-and-place APIs, organized by seen or unseen instructions and attributes. The evaluation also situates these policies’ broader language and embodiment capabilities, while noting brittleness when action APIs differ substantially.

  • Evaluation setup: The simulated tabletop setup uses a UR5e robot with a Robotiq 2F85 gripper, scripted object detection, and a parameterized pick-and-place primitive.The scene contains colored blocks and bowls for natural-language rearrangement tasks.
  • Evaluation setup: The prompt library includes command parsing, object-name and position parsing, function generation, scene queries, coordinate conversion, and pick-and-place actions.These APIs expose object names, 2D positions, robot-frame coordinates, and placement primitives to generated policies.
  • Task organization: Tasks are divided into seen and unseen instruction or attribute categories, including long-horizon and spatial-geometric families.Seen and unseen attributes vary object colors, directions, distances, magnitudes, ordinal positions, and line orientations.
  • Additional capabilities: Code as Policies can parse commands from non-English languages and emojis, extending the language inputs available to the robot policy.This capability is presented as an inherited benefit of the underlying large pretrained language model.
  • Additional capabilities: Cross-embodiment behavior changes with the available action APIs, but adaptation is brittle when those APIs are very different.The paper suggests that greater robustness may require larger models trained on domain-specific code.
  • Task organization: Table VI reports simulation tabletop manipulation success rates across detailed task scenarios.The supplied table passage identifies the reported measure but does not provide the individual numerical results.
Loading 2209.07753v4…