PaperlayerView exact revision on arXiv

Source-linked AI summary

Language to Rewards for Robotic Skill Synthesis

Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia

arXiv:2306.08647v2cs.ROcs.AIcs.LG

TL;DR

LLMs struggle to produce low-level robot commands, while many robotics applications rely on existing control primitives. This paper uses LLM-generated rewards with model predictive control to synthesize robot behavior, solving 40% more skills than baselines across 17 control problems.

  • Problem

    LLM-based robotics methods often combine preexisting control primitives, while limited relevant training data make it difficult for LLMs to directly produce low-level robot commands.

    Source: §1 · p. 1

  • Method

    The system has an LLM compose reward terms through dialogue and uses model predictive control to optimize actions, enabling language-based interaction with synthesized behavior.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    40% more skills than baselines were solved across 17 control problems, with greater stability on individual skills.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Real-time observation and language feedback let users iteratively shape synthesized robot behavior toward desired motions.

    Source: §4.5 · p. 7

    The system requires manually designed motion-description templates for each robot morphology.

    Source: §5 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

1 Introduction

The paper uses LLM-generated rewards as an interface between language and low-level robot actions, enabling online synthesis and interactive steering of behaviors. Across 17 simulated control problems, the method solved 40% more skills than baselines and was also demonstrated on a real robot arm.

  • Existing LLM robotics methods compose manually engineered or learned primitives because LLMs struggle to produce low-level commands, limiting behavior to the breadth of available primitives.Source: §1 · p. 1
  • The method has LLMs translate task semantics into modular reward functions, which MuJoCo MPC optimizes in real time to synthesize robot motions.Users can interactively steer the resulting behavior through language.Source: §1 · p. 2, Figure 1 · p. 2
  • Because people describe behavioral outcomes rather than motor commands, rewards offer a semantically rich, compositional interface for concise goals and constraints and for interactive steering.Source: §1 · p. 2
  • Prior language-to-reward work [22] [23] [24] [25] [26] [27] usually requires training data; this method instead uses human-LLM dialogue to guide reward and behavior generation.Source: §1 · p. 2
  • Across 17 control problems, the method solved 40% more skills than baselines and was more stable on individual skills; it also demonstrated complex language-guided manipulation on a real robot arm.Source: §1 · p. 2

2 Related Work

Prior approaches map language to actions, code, rewards, or corrections; this work positions LLM-generated parameterized rewards as a data-efficient interface to optimal control and iterative feedback.

Language to Actions

  • End-to-end language-conditioned policies span navigation and manipulation [30] [31] [14] [32] [16] [33] [34] [35], but require substantial data; this work instead delegates low-level actions to an optimal controller.Earlier approaches also mapped templated language to controllers or parsed it into motion primitives.Source: §2 · pp. 2–3

Language to Code

  • Code-generation research spans programming, drawing, and policy generation [36, 8, 37, 38, 39, 40, 41, 3]; this work uses LLM-generated reward code to express low-level policies.The paper reports that reward expressions can produce expressive low-level policies.Source: §2 · p. 3

Language to Rewards

  • Prior language-to-reward methods train domain-specific reward models requiring considerable language-labeled data [23] [22], or use LLMs for rollout scoring requiring many queries [26] [27]; this work produces parameterized rewards for optimization.These approaches include object pushing [25] and drawer opening; AutoRL parameterizes reward functions but lacks a language interface.Source: §2 · p. 3

Incorporating Iterative Human Feedback

  • Correction systems use semantic parsing, data-hungry policies [45], learned cost functions [25], or waypoint edits based on extensive paired data [46] [47]; this work enables data-efficient multi-step LLM reward correction.The learned cost-function approach applies trajectory optimization for control [25].Source: §2 · p. 3

3 Grounding Language to Actions Using Rewards

The system translates language into robot motion by having LLMs configure a structured reward, then using receding-horizon control to optimize actions and support interactive corrections.

3.1 Background and Reward Interface

  • The system uses generic reward residuals, with LLMs composing terms and setting their parameters and weights to express tasks without task-specific residual design.For example, a body-height residual encodes a desired height as a reward parameter.Source: §3.1 · pp. 3–4

3.2 Reward Translator

  • The Reward Translator splits language-to-reward translation into motion description and reward coding, using two LLM stages to map user interactions to reward functions.The Motion Descriptor LLM produces a natural-language description, and the Reward Coder converts it into reward parameters.Source: §3.2 · p. 4, Figure 2 · p. 4
  • The Motion Descriptor completes a template describing common robot movements, producing structured outputs that improve system stability without task-specific prompt examples.This stage addresses complex tasks that direct LLM reward generation may fail to reason through.Source: §3.2 · p. 4
  • The Reward Coder treats reward specification as code writing, using documented reward APIs, an output example, and constraints to set parameters from the motion description.The example specifies the expected response format rather than teaching a task-specific solution.Source: §3.2 · pp. 4–5

3.3 Motion Controller

  • At each control step, MuJoCo MPC replans actions from updated robot states, enabling robust interactive corrections; the system uses iLQG for locomotion and Predictive Sampling for manipulation.The controller sends the action for the current timestamp, then replans as the robot advances.Source: §3.3 · p. 5

4 Experiments

The experiments test whether LLMs paired with MuJoCo MPC can synthesize diverse motions from language, whether rewards outperform action-based interfaces, and whether the system supports interactive and real-robot use.

4.1 Experiment Setup

  • The system used GPT-4 with MuJoCo MPC to evaluate a 12-joint quadruped and a 7-DoF Franka arm paired with a 20-DoF Shadow Hand.The manipulator’s 20-DoF hand is its end-effector, creating a large action space that makes manual controller training challenging.Source: §4.1 · p. 5

4.2 Baselines

  • Baselines were Reward Coder without Motion Descriptor and Code-as-Policies, which generates plans using predefined robot primitive skills.The CaP primitives were designed around common robot commands; their full list was placed in the appendix.Source: §4.2 · pp. 5–6

4.3 Tasks

  • The benchmark comprised nine quadruped tasks and eight manipulator tasks spanning motion control and object interaction.Quadruped tasks tested heading, poses, limb control and locomotion styles; manipulator tasks covered lifting, moving and reorienting varied objects, plus interactions with articulated objects.Source: §4.3 · p. 6

4.4 Evaluation results

  • 11/17 task categories had significantly higher success rates with the proposed method, which was comparable on the rest; reward interfaces improved task coverage and reliability, and Structured Motion Description boosted performance.Each task-method pair used 10 Reward Translator responses, each evaluated 50 times in MJPC. CaP handled tasks expressible by its primitives but failed to generalize to novel low-level skills; Reward Coder alone failed on tasks needing more reasoning, such as opening a drawer.Source: §4.4 · pp. 6–7, Figure 4 · p. 7

4.5 Interactive Motion Synthesis Results

  • Real-time synthesis let users refine a quadruped moon-walk over four instructions and sequentially command a dexterous arm to place an apple in a drawer.Each instruction improved the synthesized behavior toward the desired motion, illustrating natural-language interaction for shaping robot behavior.Source: §4.5 · p. 7, Figure 5 · p. 8

4.6 Real-robot experiments

  • The real-robot implementation transferred to object pushing and grasping, generating relevant reward code that MJPC used to synthesize the motions.The mobile-manipulator pipeline estimated objects from image detections and point clouds, used an AprilTag to detect the table surface, and added a regularization residual to encourage steady, stable movement.Source: §4.6 · pp. 7–8

5 Discussion and Conlusion

The work uses LLM-defined reward functions with MuJoCo MPC to connect natural-language instructions to expressive robot control, and a motion-description template to improve skill acquisition. Its current design requires robot-specific templates and language interaction, and restricts rewards to predefined terms whose parameters the LLM adjusts.

Discussion and Conlusion

  • The system uses reward functions as a semantic-rich LLM interface to MuJoCo MPC, bridging language instructions and expressive low-level robot control.The authors evaluate the approach on a simulated quadruped and dexterous manipulator.Source: §5 · p. 8
  • A motion-description template helps extract robot-motion knowledge from LLMs, and the method outperforms alternatives in stability and number of tasks solved.Comparisons include methods without reward-function interfaces or without the template.Source: §5 · p. 8
  • Robot-specific motion-description templates require manual work, while language-only interaction makes tasks such as walking gracefully difficult to specify.The authors suggest unifying or automating template design and adding multimodal inputs, such as behavior videos.Source: §5 · p. 8
  • Predefined reward terms improve stability but limit flexibility, excluding time-varying rewards; enabling reliable reward design from scratch remains future work.Supporting time-varying rewards would require redesigning the prompt in the current system.Source: §5 · p. 8

A Appendix · A.1 Author Contributions

The appendix assigns five types of research contribution to named authors, covering method work, infrastructure, hardware deployment, advising, and paper writing or revision.

A.1 Author Contributions

  • The contribution list is organized by type, with names ordered alphabetically within each category.The five categories are method, infrastructure development, hardware deployment, project advising, and paper writing/revision.Source: §A.1 · p. 13
  • Method conception, implementation, iteration, and evaluation were attributed to six authors.Nimrod Gileadi, Kuang-Huei Lee, Yuval Tassa, Fei Xia, Peng Xu, and Wenhao Yu.Source: §A.1 · p. 13
  • Infrastructure development was attributed to five authors.Tom Erez, Nimrod Gileadi, Yuval Tassa, Fei Xia, and Wenhao Yu.Source: §A.1 · p. 13
  • Hardware deployment was attributed to nine authors.Chuyuan Fu, Nimrod Gileadi, Leonard Hasenclever, Jan Humplik, Sean Kirmani, Yuval Tassa, Fei Xia, Ted Xiao, and Wenhao Yu.Source: §A.1 · p. 13
  • Project advising was attributed to nine authors.Tom Erez, Nicolas Heess, Brian Ichter, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia, Andy Zeng, and Tingnan Zhang.Source: §A.1 · p. 13
  • Paper writing and revision were attributed to nine authors.Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Nimrod Gileadi, Brian Ichter, Dorsa Sadigh, Fei Xia, Andy Zeng, Wenhao Yu, and Tingnan Zhang.Source: §A.1 · p. 13

A.2 Full task list

The evaluation materials list the tasks and their instructions for a quadruped robot and dexterous manipulation.

A.2 Full task list

  • The evaluation list includes tasks and the instructions used for each task.Source: §A.2 · p. 13
  • Table 1 lists evaluation tasks for the quadruped robot.Source: Table 1 · p. 13
  • Table 2 lists evaluation tasks for dexterous manipulation.Source: Table 2 · p. 13

A.3 Baseline details

The baseline exposes robot control through primitive-skill interfaces: three skills for the quadruped and motion, gripper, and scene-position functions for the dexterous manipulator.

A.3 Baseline details

  • The quadruped baseline uses head_towards, walk, and set_joint_poses to set heading, control walking and turning, and directly set leg joint positions.Prompt examples help the LLM understand joint angles for set_joint_poses.Source: §A.3 · pp. 13–14
  • The manipulator baseline uses end_effector_to, end_effector_open, and end_effector_close for palm motion and hand control, with functions to retrieve object and joint positions.get_object_position(obj_name) and get_joint_position(joint_name) provide access to scene positions.Source: §A.3 · p. 14

A.4 Additional illustrations for real-world results

This subsection provides additional illustrations of the proposed system’s real-world results.

A.4 Additional illustrations for real-world results

  • Figure 7 provides additional illustrations of the proposed system’s real-world results.Source: Figure 7 · p. 14

A.5 Full Prompts

Section A.5 lists the full prompts used in Reward Translator for all experiments in the work.

A.5 Full Prompts

  • The section provides the full Reward Translator prompts used across all experiments.Source: §A.5 · p. 14

i) Motion Descriptor Prompt for Quadruped

The quadruped prompt translates a requested skill into a structured motion description, combining whole-body and navigation targets with optional per-leg adjustments and gait parameters. Rules constrain how the LLM fills and formats that description.

i) Motion Descriptor Prompt for Quadruped

  • The core descriptor specifies torso roll, pitch and height, facing direction or turn rate, and either a destination or forward and sideways speeds.The template sets standing torso height to 0.3 m and defines a horizontal body as zero pitch and roll.Source: i) Motion Descriptor Prompt for… · pp. 14–15
  • Optional foot-lift fields let the prompt set clearance independently for the front-left, back-left, front-right, and back-right feet.A 0.0 m lift means the foot maintains ground contact.Source: i) Motion Descriptor Prompt for… · pp. 14–15
  • Optional per-foot fields can extend each of the four feet forward by a specified distance.Source: i) Motion Descriptor Prompt for… · p. 15
  • Optional per-foot fields can shift each of the four feet inward laterally by a specified distance.Source: i) Motion Descriptor Prompt for… · p. 15
  • Per-leg stepping fields specify frequency, vertical and fore-aft motion, walking direction, and time spent airborne; a separate field sets all four leg phase offsets.Phase offsets range from 0 to 1; legs differing by 0 or 1 move synchronously, while a 0.5 difference puts them opposite in the gait cycle.Source: i) Motion Descriptor Prompt for… · p. 15
  • The response must use the prescribed start and end markers, include only template bullets, omit optional fields unless needed, and stay concise.Source: i) Motion Descriptor Prompt for… · p. 15
  • The LLM must replace numeric placeholders and every choice field with values or a listed option, using its best judgment when uncertain.Source: i) Motion Descriptor Prompt for… · p. 15
  • The prompt defines airborne-time fractions from 0 (always grounded) to 1 (always airborne) and assumes a balancing low-level controller and a robot capable of the requested task.It also gives a 0.65 m upright torso height alongside the 0.3 m standing height.Source: i) Motion Descriptor Prompt for… · p. 15

ii) Reward Coder Prompt for Quadruped … Dexterous Manipulator place an apple in a drawer

The appendix describes embodiment-specific prompts, reward terms, and representations for quadruped and manipulator tasks, alongside evaluation procedures and sampled outputs. It also presents interactive examples showing generated reward code and user feedback refining task targets for a quadruped moonwalk and placing an apple in a drawer.

ii) Reward Coder Prompt for Quadruped

  • The quadruped reward-coder prompt translates motion descriptions into torso targets, foot placement, and stepping parameters, while restricting code to listed functions and requiring plan execution.It specifies mutually exclusive torso location-versus-velocity and heading-versus-turning-speed targets.Source: ii) Reward Coder Prompt for Quadruped · pp. 15–17

iii) Baseline: Code-as-Policies Prompt for Quadruped

  • The quadruped Code-as-Policies baseline exposes joint-angle setting, walking, and heading APIs, with three target angles for each leg’s abduction/adduction, hip, and knee joints.The prompt also provides joint-angle pose examples, including sitting and lifting a foot.Source: iii) Baseline: Code-as-Policies Prompt… · p. 17

iv) Motion Descriptor Prompt for Dexterous Manipulator

  • The manipulator motion-descriptor prompt constrains plans to a template describing palm proximity, object relations, optional rotation or lift targets, and drawer or faucet state.It limits object choices to those listed in the prompt and asks for a complete plan in the specified format.Source: iv) Motion Descriptor Prompt for… · p. 18

v) Reward Coder Prompt for Dexterous Manipulator; vi) Baseline: Code-as-Policies Prompt for Dexterous Manipulator

A.6 Reward functions used in our experiments

  • Generic residuals are internally converted to negative L2 rewards, with weights selected for about the same magnitude and adjustable or zeroable by the reward coder; quadruped foot terms repeat across four feet.Source: §A.6 · p. 20, §A.6.1 · p. 20

A.6.3 Sim-to-Real residual term

  • The sim-to-real regularizer limits joint and end-effector speed, slows objects and matches their speed to the gripper, and encourages opening away from objects and closing near them.These terms are intended to reduce jerky motion and support cleaner grasp transfer.Source: §A.6.3 · p. 21

A.7 Detailed evaluation results for each task

  • For each task, the Reward Translator was queried 10 times and each reward code evaluated 50 times; the controller usually succeeded with a suitable reward but varied more on dynamic or precise tasks.The higher-variance examples included biped standing and upright banana.Source: §A.7 · p. 21

A.8 Example outputs

  • The robot’s standing instruction is to stand upright on two back feet like a human; its configuration sets a torso target of 0.65 and front-foot positions of 0.65, with back feet at 0.0.Source: §A.8.1 · p. 21, Ours · p. 22

Code-as-Policies

  • Code-as-Policies examples instead use explicit joint-angle targets and pose scripts for quadruped legs, while faucet scripts move the end effector to the handle and close it.The snippets include leg-pose helper calls and end-effector approach actions.Source: Code-as-Policies · pp. 23, 24, 26

A.8.2 Sampled results for Turn on faucet task

  • The sampled faucet example is prompted only with the instruction to turn on the faucet.The supplied passage states the task prompt but does not report a success outcome.Source: §A.8.2 · p. 24

A.9 Full outputs for the interactive examples

  • The interactive examples show reward-code outputs for a quadruped moonwalk and a dexterous manipulator placing an apple in a drawer.The appendix presents the examples as full outputs from the system.Source: §A.9 · p. 26

Quadruped moonwalk

  • The quadruped instructions progress from standing upright on two back feet to moonwalking backward at 0.5 m/s with feet swinging forward.The user first removes the requirement to hold the front paws at a fixed height, then clarifies the backward-walking motion.Source: Quadruped moonwalk · pp. 26–28
  • The shown reward setup sets torso targets to height 0.65, angles 90° and 0°, and velocity (-0.2, 0.0), while both back-foot position parameters are None.Source: Quadruped moonwalk · p. 27

Dexterous Manipulator place an apple in a drawer

  • The drawer instructions stage the task: open it, place the apple inside without lifting it, release the apple and move the hand away, then close the drawer.The apple-placement instruction explicitly keeps the drawer open and assumes it is on the floor.Source: Dexterous Manipulator place an apple… · pp. 28–29

Related papers

Loading 2306.08647v2…