PaperlayerView exact revision on arXiv

Source-linked AI summary

Lifelong Robot Library Learning: Bootstrapping Composable and Generalizable Skills for Embodied Control with Language Models

Georgios Tziafas, Hamidreza Kasaei

arXiv:2406.18746v2cs.RO

TL;DR

LLM robot policy-code systems are limited by fixed skill libraries and hand-crafted prompts, while end-to-end learning faces data and lifelong-learning challenges. LRLL uses memory, exploration, and skill abstraction to grow a library over time. It outperforms end-to-end and stationary LLM baselines in simulation and demonstrates transfer to real-robot rearrangement.

  • Problem

    LLM policy-code systems rely on fixed skill libraries and carefully engineered prompts, while end-to-end robot learning requires substantial data and is hindered in lifelong settings by catastrophic forgetting.

    Source: §I · p. 1

  • Method

    LRLL retrieves past instruction-code experiences for a frozen LLM, explores and verifies proposed tasks in simulation, and distills experiences into skills that expand the library.

    Source: §I · p. 2

  • Results

    LRLL outperforms end-to-end and stationary LLM baselines, builds generalizable skills, and performs real-robot rearrangement without further adaptation from simulation.

    Source: §I · p. 2, D. Zero-Shot Sim-to-Real Transfer · p. 6

  • Takeaways & Limitations

    The evaluated system builds composable, generalizable, interpretable skills that can transfer to the real world while avoiding gradient-based optimization.

    Source: §V · p. 6

    Vision is limited to referring expressions and attribute classification, demonstrations are language-only, prompts need refinement across domains or LLM engines, and commercial-LLM exploration faces latency and price constraints.

    Source: §V · p. 6

Abstract

from arXiv · show

Large Language Models (LLMs) have emerged as a new paradigm for embodied reasoning and control, most recently by generating robot policy code that utilizes a custom library of vision and control primitive skills. However, prior arts fix their skills library and steer the LLM with carefully hand-crafted prompt engineering, limiting the agent to a stationary range of addressable tasks. In this work, we introduce LRLL, an LLM-based lifelong learning agent that continuously grows the robot skill library to tackle manipulation tasks of ever-growing complexity. LRLL achieves this with four novel contributions: 1) a soft memory module that allows dynamic storage and retrieval of past experiences to serve as context, 2) a self-guided exploration policy that proposes new tasks in simulation, 3) a skill abstractor that distills recent experiences into new library skills, and 4) a lifelong learning algorithm for enabling human users to bootstrap new skills with minimal online interaction. LRLL continuously transfers knowledge from the memory to the library, building composable, general and interpretable policies, while bypassing gradient-based optimization, thus relieving the learner from catastrophic forgetting. Empirical evaluation in a simulated tabletop environment shows that LRLL outperforms end-to-end and vanilla LLM approaches in the lifelong setup while learning skills that are transferable to the real world. Project material will become available at the webpage https://gtziafas.github.io/LRLL_project.

I. INTRODUCTION

LRLL addresses the difficulty of lifelong robot learning by growing reusable skills without gradient-based optimization. It uses human guidance, exploration, memory, and skill abstraction to expand the library over time.

I. INTRODUCTION

  • End-to-end robot learning can require copious data, while gradient-based optimization hinders lifelong learning through catastrophic forgetting.Source: §I · p. 1
  • Existing LLM policy-code systems depend on a fixed skill library and carefully designed prompts, limiting task coverage and requiring experts to adapt them.Source: §I · p. 1
  • LRLL cycles between wake-phase interaction, which grows experience, and sleep-phase reflection, which distills experiences into skills that compose simpler capabilities into complex tasks.The teacher supplies a few demonstrations and hints and is assumed to follow a curriculum in which each cycle's tasks build on the current skill repertoire.Source: §I · p. 1
  • A frozen LLM uses similarity-retrieved instruction-code experiences as context, while an exploration module proposes simulator tasks and an abstraction module adds distilled skills to the library.Refactored experiences are replayed to compress memory; knowledge moves from human guidance through simulation exploration to the library without gradients.Source: §I · p. 2
  • Across a four-stage curriculum, LRLL builds hierarchical, generalizable, interpretable skills, outperforms end-to-end and stationary LLM baselines, and transfers to real-robot rearrangement without further adaptation.The paper also reports ablations and hardware demonstrations.Source: §I · p. 2

II. RELATED WORK

Prior work applies language and LLMs to robot control through planning, learned policies, and code composed from skill libraries. LRLL differs by using interaction and memory to expand a reusable skill library over time.

II. RELATED WORK

  • End-to-end language-conditioned policies can require prohibitive offline data or environment interaction, and their lifelong learning is limited by catastrophic forgetting.Source: §II · p. 2
  • LLM code-generation systems compose primitives into Python policies but generally use stationary libraries and manual prompt engineering; CaP's recursive function definitions are not controlled, reusable, or guided by humans.Source: §II · p. 2
  • Robot memory retrieval has been used to prompt LLMs, while interaction-driven memory expansion has not refined base skills; LRLL distills similar experiences into skills that refactor and compress memory.Source: §II · p. 2
  • Language-guided skill learning has relied on domain knowledge or extensive demonstrations, while related LLM methods train policies through reward learning or behavioral cloning rather than expressing policies as composable code.LRLL instead builds skills from control primitives, retaining interpretability while bypassing policy training from scratch.Source: §II · p. 2

III. METHOD

LRLL repeats a wake-and-sleep cycle: it gathers and verifies task experiences, distills them into library skills, then refactors and replays experiences to reduce memory.

A. LRLL Overview

  • Each cycle starts from language demonstrations containing instruction-policy-success tuples; policy code uses the current library, and the process can continue across an open-ended number of cycles.The success code specifies how privileged simulator data can verify a task, and resulting Python skills can be deployed on a real robot.Source: A. LRLL Overview · p. 3
  • In wake, an LLM proposes tasks from demonstrations, memory, and hints; another LLM generates policies and success checks, and verified experiences are added to memory.Exploration ends at an iteration threshold or when the LLM judges the hinted objectives complete.Source: A. LRLL Overview · p. 3
  • In sleep, experiences are grouped by policy-code abstract syntax trees that match modulo variable and constant names, then an LLM abstracts each cluster into new library functions.Source: A. LRLL Overview · p. 3, Figure 2 · p. 3
  • New skills refactor demonstration policies before wake is replayed from scratch; if a previously successful task fails, its refactored policy-success pair is added to memory.The cycle aims to retain the minimum experiences needed to reproduce wake-phase performance.Source: A. LRLL Overview · p. 3

B. System Components

The system combines pretrained vision grounding, basic motion primitives, similarity-based experience retrieval, and a Python-function skill library with simulator-backed replay.

B. System Components

  • Frozen pretrained vision-language models provide referring-expression grounding and open-vocabulary classification, while control begins with pose movement and gripper opening or closing.Motion planning uses inverse kinematics in end-effector space.Source: B. System Components · p. 3
  • Experiences store task instruction, action, and success code; instruction embeddings support maximum-marginal-relevance retrieval of diverse, similar examples as LLM prompts.Source: B. System Components · p. 3
  • Each skill is a Python API function, and the library tracks skill names, descriptions, and dependencies; a wrapper makes new skills executable in the simulator.Source: B. System Components · pp. 3–4
  • The replay buffer copies current-cycle explored experiences with their simulator states and resets at the start of each cycle.Source: B. System Components · p. 4

C. LLM Prompts

The prompts use retrieved experience examples to guide policy and success-code generation, exploration, and skill abstraction. Exploration proposes task variations and compositions, while abstraction turns example code into reusable functions.

Actor-Critic

  • The actor-critic makes parallel LLM calls for policy and success-code generation, with prompts instantiated after retrieving matching experiences.The prompts expose API modules and retrieved task-code pairs as generation context.Source: C. LLM Prompts · p. 4

Exploration

  • Exploration prompts the LLM to propose task variations and compositions, using teacher hints, current objects, and completed or failed tasks as context.The module uses separate calls for compositions and variations; the authors report this decomposition leads to faster completion.Source: C. LLM Prompts · p. 4

Skill Abstraction

  • Skill abstraction uses two rounds to turn instruction-code examples into Python functions, abstracting code variations and boilerplate while refactoring memory for replay.Generated docstrings serve as skill descriptions, and the examples are rewritten using the generated functions.Source: C. LLM Prompts · p. 4

IV. EXPERIMENTS

The experiments assess LRLL against tabletop-manipulation baselines, test the impact of its proposed contributions, and examine transfer to the real world.

Evaluation aims

  • The evaluation compares LRLL with previous baselines for tabletop manipulation in simulation.This is one of three stated evaluation aims.Source: §IV · p. 4
  • The experiments evaluate the impact of each proposed contribution.This is a separate stated evaluation aim.Source: §IV · p. 4
  • The experiments demonstrate the transferability of LRLL to the real world.This is the third stated evaluation aim.Source: §IV · p. 4

A. Evaluation Setup

The setup evaluates a four-cycle tabletop-task curriculum using generalization and lifelong-transfer splits, with success rates reported across task categories.

Reported measures

  • Table I reports averaged success rates for SA, UA, and UI across the four-cycle curriculum, with 10 trials per instruction.The caption identifies bold entries as the best results.Source: Table I · p. 5

Task curriculum

  • The curriculum spans four cycles: spatial coordination, visual reasoning, object manipulation, and long-horizon rearrangement.The task set contains 41 templates.Source: A. Evaluation Setup · pp. 4–5

Evaluation splits

  • Generalization is tested on seen instructions with seen or unseen attributes and unseen instructions with unseen attributes; forward- and backward-transfer splits assess transfer across cycles.The backward-transfer split uses forward-transfer tasks from the previous cycle to study forgetting or improvement on earlier tasks.Source: A. Evaluation Setup · p. 5

Baselines

  • The baselines include CLIPort, static-prompt LLM code generation, LRLL without sleep, and LRLL without wake.The no-sleep variant retrieves examples without abstraction; the no-wake variant synthesizes skills directly from human demonstrations.Source: A. Evaluation Setup · p. 5

B. Tabletop Manipulation in Simulation

In simulation, LRLL improves on static LLM prompting for unseen attributes and instructions, while the evaluation also examines forward and backward transfer across cycles.

Evaluation procedure

  • The simulated teacher provides up to five demonstrations and multiple test tasks at the beginning and end of each cycle, and experiments are averaged over three teacher seeds.CLIPort is trained incrementally on 1,000 trajectories per task template; agents are tested at each cycle’s end.Source: B. Tabletop Manipulation in Simulation · p. 5

Transfer evaluation

  • Table II reports success rates on unseen task combinations (FT) and a subset of previous combinations (BT) for each cycle and baseline.The evaluation setup describes FT as transfer to new compositions and BT as evaluation on prior-cycle FT tasks.Source: Table II · p. 5

Simulation results

  • ∼6% for unseen attributes and ∼10% for unseen instructions are LRLL’s average success-rate increases over LLM-static.The authors attribute the advantage to task compositions explored during wake and abstracted into shorter, functional policy code during sleep.Source: B. Tabletop Manipulation in Simulation · p. 5

C. Ablation Studies

The ablations find stronger forward transfer with LRLL than with a static LLM baseline, and show benefits from exploration, retrieval, and sleep-phase abstraction. They also identify late-cycle prompt saturation in static prompts.

Forward/Backward Transfer

  • LRLL-no-sleep gains a ∼15% averaged success delta over LLM-static in forward transfer by retrieving explored experiences.The passage reports no actual backward-transfer performance increase for any baseline.Source: C. Ablation Studies · p. 5

Static Prompts vs. Retrieval

  • Static and retrieved prompts differ marginally with a few examples, but static prompting develops late-cycle instabilities from prompt saturation.Retrieval-based baselines keep the LLM actor's context length fixed.Source: C. Ablation Studies · p. 5

Effect of Exploration

  • LRLL-no-wake performs consistently worse across cycles, while exploration supplies the abstractor with more examples for defining skills.Without exploration, one-shot demos often yield one-to-one instruction-to-function mappings without refactoring.Source: C. Ablation Studies · pp. 5–6
  • Explored instruction embeddings cover task variations spanning a broad range of skill compositions from the demonstrations.The t-SNE projections compare train (SA), test (UA+UI), and explored instructions in Visual Reasoning and Rearrangement cycles.Source: Figure 3 · p. 6

Effect of Abstraction

  • Sleep-phase refactoring reduces the experience storage needed for the same performance by 8× and supports high unseen-instruction success with fewer retrieved examples.Refactored code uses simpler skill calls, reducing prompt length and GPT cost.Source: C. Ablation Studies · p. 6

D. Zero-Shot Sim-to-Real Transfer

LRLL is trained in Gazebo and tested on a dual-arm robot without further adaptation. It performs long-horizon rearrangement combining spatial positioning and object-attribute reasoning, with observed errors in motion execution and perception.

D. Zero-Shot Sim-to-Real Transfer

  • The transfer setup uses two UR5e arms, a Kinect, and vision APIs for open-vocabulary detection, attribute recognition, and 4-DoF grasp synthesis.The APIs use MDETR, CLIP, and GR-ConvNet, respectively.Source: D. Zero-Shot Sim-to-Real Transfer · p. 6
  • The real-robot setup includes a catalog of 12 household objects, including fruits, soda cans, and juice boxes.The motion primitives parameterize arm movement and finger opening or closing for the left or right arm.Source: D. Zero-Shot Sim-to-Real Transfer · p. 6
  • After Gazebo training, the real robot performs long-horizon rearrangement combining precise spatial positioning and object-attribute reasoning without further adaptation.Observed errors were mostly due to motion-execution collisions and CLIP misclassifications.Source: D. Zero-Shot Sim-to-Real Transfer · p. 6

V. CONCLUSION & LIMITATIONS

LRLL grows a robot manipulation skill library by generating policies, exploring tasks, and distilling experience into skills. The authors report transferable, composable skills and identify limitations in perception, demonstration input, setup prompts, and exploration cost.

V. CONCLUSION & LIMITATIONS

  • LRLL generates policy code, explores tasks, and distills acquired experiences into new skills, replacing prompt engineering and a static skill library with retrieval and an expandable codebase.The agent writes and verifies the codebase itself.Source: §V · p. 6
  • The learned skills are composable, generalizable, and interpretable, and the authors report transfer to the real world.They attribute avoiding prompt saturation and forgetting phenomena to LRLL's dynamic, gradient-free nature.Source: §V · p. 6
  • Language-only demonstrations limit scaling to skills expressible as symbolic primitive compositions; articulated, contact-rich tasks motivate video or kinesthetic teaching.Other stated boundaries include vision APIs limited to referring expressions and attribute classification, prompts needing refinement across domains or LLM engines, and commercial-LLM exploration latency and price.Source: §V · p. 6

Related papers

Loading 2406.18746v2…