Source-linked AI summary
Language to Rewards for Robotic Skill Synthesis
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia
TL;DR
LLMs struggle to translate language directly into hardware-dependent low-level robot actions, while existing robotics methods often rely on semantic planning or predefined primitives. The paper uses LLMs to generate reward parameters and MuJoCo MPC to optimize them into robot motions interactively. Across 17 simulated tasks, it reports solving 40% more skills than baselines and demonstrates complex manipulation on a real robot arm.
Problem
LLMs have limited ability to produce hardware-dependent low-level robot actions, so robotics applications commonly rely on semantic planners or predefined control primitives.
Method
The system uses an LLM-based Reward Translator to generate compositional reward specifications and a MuJoCo MPC Motion Controller to optimize them into robot actions.
Results
Across 17 simulated quadruped and dexterous-manipulator control problems, the proposed formulation solves 40% more skills than baselines and is demonstrated on a real robot manipulator.
Takeaways & Limitations
Reward functions provide an intermediate interface through which language can specify diverse robot behaviors while remaining compatible with low-level control optimization.
Takeaways & Limitations
The system requires manually designed motion-description templates for each robot morphology and relies on language for human interaction.
Abstract
from arXiv · showhide
Large language models (LLMs) have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.
1 Introduction
The paper addresses the gap between LLMs’ semantic knowledge of robot motions and their limited ability to produce hardware-dependent low-level actions. It proposes using language-generated rewards as an interactive interface and evaluates the approach across simulated robots and a real manipulator.
- Motivation: LLM-based robotics methods commonly plan, dialogue, or generate code that composes existing control primitives, but struggle to output low-level robot commands.The limited availability of relevant robot-action data constrains direct language-to-action translation.
- Approach: The proposed system uses an LLM-based Reward Translator to generate reward specifications, which a Motion Controller optimizes into robot motions.The system is designed to support interactive behavior creation from user input and feedback.
- Key insight: Reward functions provide a semantic-rich, modular interface that can connect behavioral language instructions to low-level robot actions.Users can describe outcomes such as standing up rather than specifying motor torques, while modular reward terms represent behaviors, goals, and constraints.
- Contribution: Prior language-to-reward methods often require training data, whereas this approach enables a data-efficient interactive dialogue between humans and the LLM.The dialogue guides reward generation and consequently robot behavior.
- Results: Across 17 simulated quadruped and dexterous-manipulator control problems, the proposed formulation solves 40% more skills than baselines.The evaluation includes diverse locomotion and manipulation skills, followed by deployment on a real robot manipulator.
2 Related Work
Related work has connected language to robot actions, code, rewards, and iterative feedback through parsers, learned models, and code-generation systems. These approaches often depend on restricted action spaces, predefined primitives, or substantial task-specific data.
- Language to Actions: Language-to-action research ranges from templated language parsers and motion primitives to end-to-end models conditioned on natural-language descriptions.Navigation methods often assume low-dimensional graph actions, while manipulation methods commonly use behavioral cloning with language embeddings.
- Language to Code: Language-to-code systems have generated policies and instruction-following programs, while this work targets code generation for expressive reward functions.The paper positions reward-code generation as a way to produce low-level policies.
- Language to Rewards: Language-to-reward methods commonly train domain-specific models that map instructions to reward values or constraints.Prior systems have handled tasks such as object pushing and drawer opening but require considerable language-labeled data.
- Incorporating Iterative Human Feedback: Interactive language-feedback methods include semantic parsers with predefined mappings and learned correction-conditioned policies.The latter approaches are similarly data hungry and therefore fall back on additional constraints.
3 Grounding Language to Actions Using Rewards
The method represents robot-control objectives as compositional reward terms and uses LLMs to translate language into their parameters. A MuJoCo MPC controller then optimizes the generated rewards through repeated planning, while a two-stage translator structures motion descriptions before reward coding.
- 3.1 Background and Reward Interface: The system models robot control as an MDP in which a controller selects actions maximizing expected accumulated reward.The reward connects user instructions to low-level controller optimization.
- 3.1 Background and Reward Interface: The assumed MJPC-compatible reward is a sum of individual terms built from non-negative weights, norms, residuals, and reward parameters.Each residual reaches optimality at zero, and parameters can encode desired values such as a target body height.
- 3.1 Background and Reward Interface: Instead of designing task-specific residuals, the method uses generic residual terms that LLMs compose into complex behaviors.This reduces reliance on domain expertise for constructing rewards for novel tasks.
- 3.3 Motion Controller: A Reward Translator modulates reward parameters and weights, while a MuJoCo MPC Motion Controller optimizes the corresponding action sequence.The controller repeatedly replans over a receding horizon and sends the current action to the robot.
- 3.2 Reward Translator: The translator first expands user input into a templated natural-language motion description, then converts that description into reward code.The Motion Descriptor improves structure and stability, and the Reward Coder uses coding knowledge to generate reward-specifying code.
4 Experiments
The experiments evaluate whether LLM-generated rewards can produce diverse robot motions, outperform primitive-skill interfaces, support interactive behavior shaping, and transfer to real hardware. Across simulated quadruped and dexterous-manipulator tasks, the system is assessed for reliability, task coverage, and real-world pushing and grasping.
- The evaluation asks whether combining LLMs with MJPC can generate diverse, complex robot motions through natural-language commands, whether reward interfaces improve expressiveness, and whether the method applies to real hardware.
- The study compares the proposed system with Reward Coder without Motion Descriptor and Code-as-Policies using predefined robot primitive skills.The simulated systems use a quadruped and a dexterous manipulator modeled in MuJoCo MPC.
- The benchmark contains 17 simulated tasks spanning nine quadruped tasks and eight dexterous-manipulator tasks.The quadruped tasks include heading-direction and body-pose control; manipulator tasks include object interactions and articulated objects.
- 11/17 task categories show significantly higher success rates for the proposed method, with comparable performance on the remaining tasks.Each task-method combination uses 10 Reward Translator responses, each evaluated in MJPC 50 times to measure end-to-end stability.
- Reward interfaces solve more tasks more reliably, while Structured Motion Description further boosts performance significantly.The pass-rate curve measures both system stability across trials and task coverage given sufficient trials.
- Interactive MJPC enables users to shape robot behavior through successive natural-language instructions, including quadruped moonwalking and sequentially placing an apple in a drawer.The system also transfers to real-world object pushing and grasping, using a sim-to-real regularization reward to encourage steady movements.
5 Discussion and Conlusion
The paper presents reward functions as an interface between LLMs and robot control, using MuJoCo MPC to acquire varied skills on simulated robots. It reports better stability and task-solving performance than alternatives, while identifying manual morphology-specific templates and constrained reward design as limitations.
- Discussion and Conclusion: Reward functions interface LLMs with robot control through MuJoCo MPC, combining semantic richness with controller expressiveness.The system also uses a motion description template to extract robot-motion knowledge from LLMs.
- Discussion and Conclusion: The method achieves significantly better stability and solves more tasks than alternatives without reward interfaces or motion description templates.
- Limitations and Future Work: Motion-description templates currently require manual design for each robot morphology, limiting easy extension to novel morphologies.
- Limitations and Future Work: Pre-defined reward terms improve stability but sacrifice flexibility, including support for time-varying rewards.
A.1 Author Contributions
The listed author contributions cover method development, infrastructure, hardware deployment, project advising, and paper writing or revision.
- Method: Method contributions include conception, implementation, iteration, and evaluation by Nimrod Gileadi, Kuang-Huei Lee, Yuval Tassa, Fei Xia, Peng Xu, and Wenhao Yu.
- Infrastructure Development: Infrastructure development was credited to Tom Erez, Nimrod Gileadi, Yuval Tassa, Fei Xia, and Wenhao Yu.
- Hardware Deployment: Hardware deployment was credited to Chuyuan Fu, Nimrod Gileadi, Leonard Hasenclever, Jan Humplik, Sean Kirmani, Yuval Tassa, Fei Xia, Ted Xiao, and Wenhao Yu.
- Project Advising: Project advising was credited to Tom Erez, Nicolas Heess, Brian Ichter, Dorsa Sadigh, Jie Tan, Yuval Tassa, Fei Xia, Andy Zeng, and Tingnan Zhang.
- Paper Writing/Revision: Paper writing and revision were credited to Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Nimrod Gileadi, Brian Ichter, Dorsa Sadigh, Fei Xia, Andy Zeng, Wenhao Yu, and Tingnan Zhang.
A.2 Full task list
This appendix section presents the evaluation task lists for the quadruped robot and dexterous manipulation.
- Full task list: The appendix states that it provides the task list and instructions used for evaluation.
- Quadruped robot: Table 1 lists the tasks used in evaluation for the quadruped robot.
- Dexterous manipulation: Table 2 lists the tasks used in evaluation for dexterous manipulation.
A.3 Baseline details
The baseline details describe primitive interfaces for quadruped locomotion and dexterous manipulation, including motion commands, joint poses, hand control, and scene queries.
- Quadruped robot: The quadruped baseline uses head_towards, walk, and set_joint_poses primitive skills.The walk primitive controls forward, sideways, and turning speeds, while set_joint_poses directly specifies joint positions.
- Quadruped robot: head_towards(direction) specifies a target heading direction for the quadruped robot.
- Dexterous manipulator robot: The dexterous manipulator baseline uses end-effector movement and hand opening or closing, plus object- and joint-position queries.
A.4 Additional illustrations for real-world results
Figure 7 provides additional illustrations of the proposed system’s real-world results.
- Figure 7 shows more illustrations of the proposed system’s real-world results.
A.5 Full Prompts
This section lists the full prompts used by Reward Translator in all experiments.
- The appendix lists the full prompts used in Reward Translator for all experiments.
i) Motion Descriptor Prompt for Quadruped
The quadruped Motion Descriptor prompt expands motion descriptions into structured torso and foot parameters, including optional gait details.
- The prompt specifies torso roll, pitch, height, heading or turning, and optional forward or sideways motion.
- Optional parameters describe individual foot lifts, forward extensions, inward shifts, and stepping trajectories.
- Stepping descriptions include frequency, vertical and forward-back motion, gait direction, and the fraction of time spent in the air.
ii) Reward Coder Prompt for Quadruped
The quadruped Reward Coder prompt exposes structured reward and motion interfaces, while the baseline uses fixed joint-angle and motion primitives; the real-robot system adds a stability residual.
- ii) Reward Coder Prompt for Quadruped: The Reward Coder converts a motion description into a program using predefined functions and permitted libraries.
- ii) Reward Coder Prompt for Quadruped: Quadruped torso targets encode height, pitch, roll, location, velocity, heading, and turning speed.
- ii) Reward Coder Prompt for Quadruped: Foot-position parameters independently control lift height, forward extension, and inward movement, with None disabling each term.
- ii) Reward Coder Prompt for Quadruped: Foot-stepping parameters control frequency, air ratio, phase offset, swing motion, and activation for each named foot.
- iii) Baseline: Code-as-Policies Prompt for Quadruped: The baseline represents quadruped behavior through fixed primitive functions for joint angles, walking, heading, and plan execution.
- iii) Baseline: Code-as-Policies Prompt for Quadruped: Baseline examples directly set four legs’ joint angles to produce sitting and individual-foot lifting behaviors.
- v) Reward Coder Prompt for Dexterous Manipulator: The dexterous-manipulator interface specifies palm proximity, object proximity, orientation, and optional task-specific motion parameters.
- A.6.3 Sim-to-Real residual term: The sim-to-real residual regularizes joint, end-effector, and object velocities while encouraging clean gripper opening and closing during grasp transfer.