PaperlayerView exact revision on arXiv

Source-linked AI summary

RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models

Jingyao Li, Pengguang Chen, Sitong Wu, Chuanyang Zheng, Hong Xu, Jiaya Jia

arXiv:2406.03757v1cs.ROcs.LG

TL;DR

Single-task benchmarks provide limited evidence about robotic task generalization. RoboCoder pairs a benchmark of tasks across entities with an adaptive framework that updates action codes using environmental feedback. The authors report a 36% relative improvement over GPT-4 with humanoid entities.

  • Problem

    Existing robotic benchmarks focus on independently trained single tasks, and current research has limited generalization capabilities.

    Source: §1 · p. 1

  • Method

    The paper introduces an 80-task benchmark across 7 entities and RoboCoder, which uses existing skills and environmental feedback to refine action codes for complex tasks.

    Source: §1 · pp. 1–2

  • Results

    36% relative improvement over GPT-4 with humanoid entities; the authors also report surpassing coding models like GPT-4 in few-shot scenarios, particularly in complex environments.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark tests learning complex tasks from simple skills, while RoboCoder's adaptive feedback approach is presented as improving applicability in open scenarios.

    Source: §1 · p. 2

    The method has only been implemented in simplified simulated environments, has not been validated in the real world, and covers relatively simple tasks that require little physical-world knowledge.

    Source: §5 · p. 9

Abstract

from arXiv · show

The emergence of Large Language Models (LLMs) has improved the prospects for robotic tasks. However, existing benchmarks are still limited to single tasks with limited generalization capabilities. In this work, we introduce a comprehensive benchmark and an autonomous learning framework, RoboCoder aimed at enhancing the generalization capabilities of robots in complex environments. Unlike traditional methods that focus on single-task learning, our research emphasizes the development of a general-purpose robotic coding algorithm that enables robots to leverage basic skills to tackle increasingly complex tasks. The newly proposed benchmark consists of 80 manually designed tasks across 7 distinct entities, testing the models' ability to learn from minimal initial mastery. Initial testing revealed that even advanced models like GPT-4 could only achieve a 47% pass rate in three-shot scenarios with humanoid entities. To address these limitations, the RoboCoder framework integrates Large Language Models (LLMs) with a dynamic learning system that uses real-time environmental feedback to continuously update and refine action codes. This adaptive method showed a remarkable improvement, achieving a 36% relative improvement. Our codes will be released.

1 Introduction

Existing independently trained single-task benchmarks have limited generalization, motivating a benchmark and framework for building complex tasks from simple skills. RoboCoder uses environmental feedback to update action codes and reports improved performance over GPT-4.

1 Introduction

  • Existing benchmarks train on single tasks independently, while this work targets autonomous reuse of existing skills to learn more complex tasks with limited initial mastery.The authors frame this goal as improving applicability in real-world scenarios.Source: §1 · p. 1
  • The benchmark contains 80 manually designed tasks across 7 entities and tests learning complex tasks from simple skills.The entities span humanoids, insect-like robots, mechanical arms, and dynamic control systems.Source: §1 · p. 1
  • RoboCoder searches for similar actions, has an Actor refine retrieved codes when needed, and uses an Evaluator to check, revise, and store successful actions.Evaluator feedback guides revision until the task passes or the maximum iteration count is reached.Source: §1 · p. 2
  • 36% relative improvement over GPT-4 with humanoid entities; in quadruped-robot-dog environments, the reported rate reaches 92%.The paper also reports GPT-4 achieves a 47% pass rate in three-shot scenarios with humanoid entities.Source: §1 · p. 2

2 Related Works

Related work spans high-level action planning, code generation with motion primitives, LLM-guided evolution, and code repair. The paper positions RoboCoder as learning and iteratively refining executable robotic actions in a dynamic action space.

Models for Action Output

  • Vision-language action models have focused on high-level semantic planning, while converting instructions into executable robotic actions remains an open challenge (Guhur et al. (2022); Shridhar et al. (2022); Zhang & Chai (2021); Silva et al. (2022); Jang et al. (2022); Brohan et al. (2023); Jiang et al., 2023; Driess et al. (2023)).The passage names VIMA and PaLM-E as examples.Source: §2 · p. 2

Models for Robot Code Generation

  • Prior robotics code studies primarily use known motion primitives rather than autonomously refining actions (Austin et al., 2021; Chen et al., 2021a; Roziere et al., 2023; Liang et al., 2023; Singh et al., 2022; Wang et al., 2023b; Huang et al., 2023; Wang et al., 2023a; Liu et al., 2023; Silver et al., 2023; Ding et al., 2023; Lin et al., 2023; Xie et al., 2023; Mu et al., 2024).LLM reward-design approaches are described as requiring extensive learning and having limited task generalization.Source: §2 · p. 3

Models for Evolution

  • RoboCoder adds an action-space updating mechanism for iterative self-evolution, framed as enhancing success rate and operational speed.The related-work discussion contrasts this with LLM evolution work in architecture search, prompt and morphology design, and reward design (Chen et al., 2023a; Nasir et al., 2023; Guo et al., 2024; Lehman et al., 2022; Yu et al. (2023); Ma et al. (2023)).Source: §2 · p. 3

Models for Code Repair

3 Methods

RoboCoder builds on an initial action space, searching for relevant actions and iteratively refining candidates with an Actor and Evaluator. Verified actions can update the action space for later tasks.

3.2 Actor

  • The initial action space includes fundamental actions, with detailed human actions and few-shot examples for other entities to test self-expansion.The Actor receives retrieved actions, degrees-of-freedom descriptions, and the target task, then generates action functions for the entity’s motion.Source: §3.1 · p. 4, §3.2 · p. 5

3.1 Action Search

  • The Action Searcher embeds target tasks and action examples, then uses cosine similarity and dual thresholds to select matching actions or a template for generation.A sufficiently high score retrieves an action directly; when scores fall below the lower threshold, the closest action guides generation.Source: §3.1 · p. 4

3.3 Evaluator

  • The Evaluator uses a vision-language model to judge task completion from execution images or, when errors prevent image capture, from action code and error messages.It returns completion information and, when the task is incomplete, suggestions for the Actor to revise the action.Source: §3.3 · pp. 5–6, Figure 5 · p. 6

3 Methods

  • The framework searches for similar actions, has the Actor refine candidates, and sends them to the Evaluator before incorporating successful actions into the action space.Parallel updating tests tasks simultaneously and updates the action space after each iteration.Source: §3.4 · p. 6

3.4 Update Action Space

  • Actions that pass the Evaluator’s criteria are added to the action space, while parallel updating lets difficult tasks benefit from insights gained from simpler ones.As the action space grows, a lightweight searcher can retrieve verified code without rewriting it via an LLM.Source: §3.4 · p. 6

4 Experiments

RoboCoder is evaluated on a seven-entity, 80-task benchmark, with ablations testing feedback, retrieval, and action-space updates. It improves performance over prior methods and model baselines, while a lightweight searcher accelerates inference.

4.3 Ablation Study

  • 20.82% average improvement over w/o feedback contrasts with Code Evaluator’s 1.35% improvement over the first iteration versus RoboCoder’s 31.27% after iterations.The feedback ablation removes reason feedback; the Code Evaluator uses action code and terminal information without action images.Source: Figure 6 · p. 7, §4.3 · p. 8
  • Related Searcher improves over random selection by 15.28% in the first iteration and 5.45% at the end; action-space updates also markedly enhance performance.The action-space ablation tests GPT-3.5 in cycle two for the Human entity.Source: Figure 6 · p. 7, Figure 7 · p. 7, §4.3 · p. 8
  • 0.06 seconds for MiniLMv2 retrieval compares with up to 54 seconds for CodeLLaMA-34B on a single A800 GPU for the Human entity.The Searcher has 21M parameters; the comparison concerns inference speed.Source: Figure 8 · p. 7, §4.3 · p. 9

4.1 Employment Details; 4.2 Test Benchamrk

  • The benchmark covers 80 tasks across Human, Ant, Cartpole, Sektion Cabinet, Franka Panda, Kinova, and Anymal, with GPT-4 as default Actor and GPT4V as Evaluator.MiniLM encodes sentences into a 384-dimensional vector space for similarity assessment.Source: §4.2 · pp. 7–8, §4.1 · p. 7

4.4 Performance

  • 10.34% average enhancement over previous state-of-the-art methods was reported across entities, while the GPT-4 series increased 30.56% over its baseline.The comparison spans coding models including StarCoder, WizardCoder, CodeLLaMA variants, and GPT models.Source: Figure 9 · p. 7, Figure 10 · p. 8, §4.4 · p. 9

5 Limitation

The method is currently tested only in simplified simulated environments, and its tasks are relatively simple and do not require extensive physical-world knowledge.

5 Limitation

  • The method has not been validated in the real world and is limited to simplified simulations with relatively simple tasks.The authors cite hardware limitations and physical-world challenges as reasons for the current simulation-only implementation.Source: §5 · p. 9

6 Conclusion

The conclusion presents RoboCoder and its benchmark as a way to test learning from minimal initial mastery and to update action codes using real-time environmental feedback. It reports improved pass rates, while noting potential misuse of robotic manipulation.

6 Conclusion

  • 36% relative improvement in humanoid pass rates and up to 92% with quadruped robot dogs are reported for RoboCoder.The conclusion describes the approach as an LLM-based adaptive system that dynamically updates action codes from real-time environmental feedback.Source: §6 · p. 9

A Broader impacts

  • The authors describe robotic manipulation as useful to users while warning of potential abuse in other robotic manipulation applications.Source: A Broader impacts · p. 14

B Algorithm

  • The framework searches for a related action, then iterates Actor-generated code and Evaluator feedback up to a maximum iteration count unless the task is complete.The algorithm takes a target task, Searcher, Actor, Evaluator, action space, and iteration limit as inputs.Source: B Algorithm · p. 14

C DOFs

  • The appendix presents Degrees of Freedom properties and illustrates their movements.Source: C DOFs · p. 14

C.1 DOFs Properties

  • The environment property tables document each system’s DOFs, type, stiffness, damping, armature, and motion limits.The section lists Human, Ant, Cartpole, Sektion Cabinet, Franka Panda, Kinova Cabinet, and Anymal.Source: §C.1 · p. 14

C.2 Property Distribution

The property distributions show many zero stiffness and armature settings, widespread motion limits, and default positions centered near zero.

C.2 Property Distribution

  • Stiffness and armature values are often zero, indicating that inertial effects are minimally modeled in most cases.Source: §C.2 · p. 14
  • Tables 1 and 2 are labeled as the Human and Ant DOF properties tables.Source: Table 1 · p. 15, Table 2 · p. 15
  • Most parameters are limited, and default positions are typically centered at zero; rotational limits cluster around ±3.14.Source: §C.2 · p. 15

C.3 DOFs Illustrations

The section presents DOF diagrams for seven entities and describes parameter distributions and DOF-property tables.

D Test Space

The test space uses tasks across seven entities to assess baseline performance and provide standardized tasks for future research.

  • The testing space evaluates the baseline framework across Human, Ant, Cartpole, Sektion Cabinet, Franka Panda, Kinova, and Anymal, selected for their distinct challenges.Source: D Test Space · p. 21
  • Human tasks range from basic movements such as raising an arm to more complex activities such as running.Source: D Test Space · p. 21
  • The task tables are intended to test movement and interaction dynamics and to establish standardized tasks for subsequent research.Source: D Test Space · p. 21, Table 8 · p. 21

E Action Space

The action space provides an initial repertoire across entities, which the framework aims to expand through continued learning and interaction.

F Pass Rates of Code Evaluator

The Image Evaluator is favored over the Code Evaluator for guiding feedback iterations; the framework uses parallel action-space updates across tasks.

F Pass Rates of Code Evaluator

G Update Action Space

  • Parallel updating tests tasks simultaneously and updates the action space after each iteration, allowing simple tasks to be integrated while difficult tasks draw on simpler ones.Source: G Update Action Space · p. 23, Table 10 · p. 24

Related papers

Loading 2406.03757v1…