PaperlayerView exact revision on arXiv

Source-linked AI summary

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, Peter Stone

arXiv:2306.03310v2cs.AI

TL;DR

Lifelong decision-making requires transferring both declarative and procedural knowledge, a gap that existing lifelong-learning benchmarks do not systematically measure. LIBERO addresses this with a procedurally generated robot-manipulation benchmark containing four suites and 130 tasks, and experiments show that sequential finetuning has stronger forward transfer while architecture and pretraining choices materially affect performance.

  • Problem

    Lifelong decision-making requires transferring declarative and procedural knowledge, but this mixed knowledge transfer is understudied and lacks systematic quantitative analysis.

  • Method

    LIBERO provides a procedural task-generation pipeline, four benchmark suites totaling 130 manipulation tasks, and human-teleoperated demonstrations for all tasks.

  • Results

    Sequential finetuning outperforms evaluated lifelong-learning methods in forward transfer, while architecture performance varies by knowledge type and naive supervised pretraining can hinder downstream performance.

  • Takeaways & Limitations

    LIBERO enables systematic study of knowledge transfer, policy architectures, lifelong-learning algorithms, task ordering, and pretraining in robot manipulation.

  • Takeaways & Limitations

    The lifelong formulation assumes shared state, action, transition, and horizon spaces across tasks, and the agent loses access to previous tasks during new-task learning.

Abstract

from arXiv · show

Lifelong learning offers a promising paradigm of building a generalist agent that learns and adapts over its lifespan. Unlike traditional lifelong learning problems in image and text domains, which primarily involve the transfer of declarative knowledge of entities and concepts, lifelong learning in decision-making (LLDM) also necessitates the transfer of procedural knowledge, such as actions and behaviors. To advance research in LLDM, we introduce LIBERO, a novel benchmark of lifelong learning for robot manipulation. Specifically, LIBERO highlights five key research topics in LLDM: 1) how to efficiently transfer declarative knowledge, procedural knowledge, or the mixture of both; 2) how to design effective policy architectures and 3) effective algorithms for LLDM; 4) the robustness of a lifelong learner with respect to task ordering; and 5) the effect of model pretraining for LLDM. We develop an extendible procedural generation pipeline that can in principle generate infinitely many tasks. For benchmarking purpose, we create four task suites (130 tasks in total) that we use to investigate the above-mentioned research topics. To support sample-efficient learning, we provide high-quality human-teleoperated demonstration data for all tasks. Our extensive experiments present several insightful or even unexpected discoveries: sequential finetuning outperforms existing lifelong learning methods in forward transfer, no single visual encoder architecture excels at all types of knowledge transfer, and naive supervised pretraining can hinder agents' performance in the subsequent LLDM. Check the website at https://libero-project.github.io for the code and the datasets.

1 Introduction

Lifelong learning aims to build adaptable generalist agents by transferring knowledge across tasks, but decision-making requires transferring both declarative and procedural knowledge. LIBERO benchmarks this gap with procedurally generated robot-manipulation tasks and evaluates architectures, algorithms, ordering robustness, and pretraining.

  • Lifelong learning transfers prior knowledge to new tasks and newly acquired knowledge back to earlier tasks.
  • Decision-making lifelong learning requires transferring both declarative knowledge about entities and concepts and procedural knowledge about actions and behaviors.
  • LIBERO provides a procedural pipeline for continuously generating diverse robot-manipulation tasks that share visual concepts and interactions.
  • 130 language-conditioned tasks are grouped into four suites covering shifts in objects, spatial arrangements, goals, or mixtures, with human-teleoperated demonstrations for every task.
  • Policy architecture matters as much as lifelong-learning algorithms: transformers better abstract temporal information, vision transformers suit rich visual tasks, and convolutional networks suit procedural tasks.
  • Sequential finetuning outperforms evaluated lifelong-learning methods in forward transfer, while semantically rich language embeddings offer no advantage over task-ID embeddings and naive supervised pretraining can hurt downstream performance.

2 Background

The paper formulates robot learning as a finite-horizon Markov decision process and lifelong learning as sequential task-conditioned policy learning. Because earlier tasks and demonstrations are not fully available, training uses behavioral cloning on task-specific demonstrations.

  • Robot learning is modeled as a finite-horizon Markov decision process with state and action spaces, transitions, horizon, initial-state distribution, and reward.
  • In the sparse-reward setting, the reward is replaced by a binary goal predicate, and the policy maximizes expected return.
  • Lifelong learning uses one task-conditioned policy across sequential tasks sharing the state, action, transition, and horizon spaces.
  • The agent loses access to previous tasks while learning the current task, and earlier task data are not fully available.
  • Lifelong imitation learning supplies a small demonstration dataset for each task and trains with behavioral cloning using aggregated observation histories.

3 Research Topics in LLDM

LIBERO is designed around five LLDM questions: transferring different knowledge types, choosing architectures and algorithms, handling task ordering, and understanding pretraining. These questions reflect the mixed declarative and procedural structure of robot manipulation.

  • T1 Transfer of Different Types of Knowledge: LIBERO separates transfer of spatial, object, goal, and entangled knowledge through four task suites.
  • T2 Neural Architecture Design: Effective architectures must abstract multimodal images, language, and robot states while transferring only relevant knowledge to new tasks.
  • T3 Lifelong Learning Algorithm Design: Lifelong-learning algorithms must limit cumulative forgetting because small successive losses can ultimately cause execution failure.
  • T4 Robustness to Task Ordering: Robust lifelong-learning algorithms should tolerate different task orderings because real-world robots often cannot choose which task comes first.
  • T5 Usage of Pretrained Models: The benchmark examines whether pretraining on large datasets benefits subsequent lifelong decision-making learning.

4 LIBERO

LIBERO combines procedural task generation, fixed benchmark suites, demonstration-based policy learning, and representative architectures and lifelong-learning algorithms. Its suites disentangle selected knowledge transfers while also providing an entangled 100-task setting.

  • Procedural Generation: LIBERO generates tasks by extracting behavioral templates, specifying initial object distributions, and defining language-aligned goals with propositional formulas.
  • Procedural Generation: The pipeline uses human-activity language annotations to create task instructions, selects matching scenes and initial configurations, and represents layouts and goals in PDDL.
  • Task Suites: LIBERO offers four suites: SPATIAL, OBJECT, GOAL, and LIBERO-100; the first three disentangle declarative and procedural transfer, whereas LIBERO-100 entangles them.
  • Task Suites: LIBERO-SPATIAL, LIBERO-OBJECT, and LIBERO-GOAL each contain 10 tasks, while LIBERO-100 contains 100 tasks split into LIBERO-90 for pretraining and LIBERO-LONG for downstream lifelong evaluation.
  • Lifelong-Learning Algorithms: The benchmark implements Experience Replay, Elastic Weight Consolidation, and PACKNET as memory-based, regularization-based, and dynamic-architecture lifelong-learning methods.
  • Policies and Training: The implementation includes RESNET-RNN, RESNET-T, and VIT-T vision-language policies, trained with behavioral cloning from 50 human demonstrations per task.

5 Experiments

Experiments evaluate LIBERO across metrics, architectures, lifelong-learning algorithms, language embeddings, task orderings, and pretraining. Results show architecture- and algorithm-dependent transfer, task-order sensitivity, no significant language-embedding differences, and potential harm from basic supervised pretraining.

  • Experimental setup: The study evaluates six research questions spanning architecture, algorithm, language embedding, task ordering, and supervised pretraining effects.Experiments use FWT, NBT, and AUC, computed from success rates.
  • Policy architectures: RESNET-T and VIT-T outperform RESNET-RNN on average, while architecture rankings vary with the lifelong-learning algorithm and task suite.With ER, RESNET-T generally exceeds VIT-T except on LIBERO-OBJECT; with PACKNET, VIT-T is much better on LIBERO-LONG.
  • Lifelong-learning algorithms: SEQL achieves the best FWT across all task suites, while ER is robust across suites and PACKNET performs differently on LIBERO-X and LIBERO-LONG.PACKNET appears effective at preventing forgetting but may lack sufficient capacity for LIBERO-LONG; EWC performs worse than SEQL.
  • Language embeddings: No statistically significant difference appears among BERT, CLIP, GPT-2, and Task-ID language embeddings.The authors suggest sentence embeddings may function mainly as bag-of-words representations that distinguish tasks without exploiting semantic information.
  • Task ordering: Different task orderings produce very different performances for the same algorithm, with the difference statistically significant for PACKNET.The result motivates algorithms or architectures that are robust to varying task orderings.
  • Pretraining: Basic supervised pretraining can hurt downstream lifelong-learning performance, while naive sequential fine-tuning shows better forward transfer in the reported comparison.The findings indicate that better pretraining techniques are needed.

6 Related Work

The paper situates LIBERO among lifelong-learning and robot-learning benchmarks, emphasizing that its task suites target lifelong learning in decision-making. Existing benchmarks cover standard datasets and diverse robot-learning challenges, but LIBERO is curated for LLDM research.

  • Earlier lifelong-learning benchmarks adapted image, segmentation, language-understanding, and video-game datasets for continual learning.
  • Robot-learning benchmarks have addressed meta learning, causality learning, multitask learning, generalization to unseen objects, and compositional learning.
  • LIBERO differs by curating task suites specifically to study lifelong learning in robot manipulation and decision-making.

7 Conclusion and Limitations

LIBERO contributes an extensible robot-manipulation benchmark for LLDM, combining procedural task generation, standardized tasks, and comprehensive experiments. The conclusion identifies future work on architectures, forward transfer, and pretraining, while noting long-term privacy concerns.

  • LIBERO provides a procedural-generation pipeline capable of creating an infinite number of manipulation tasks in simulation.
  • The benchmark instantiates 130 standardized tasks and supports experiments on policy and lifelong-learning algorithm design.
  • Future research should improve spatial and temporal processing, forward transfer, and the use of pretraining for lifelong learning.
  • The paper anticipates no short-term negative societal impacts but identifies user privacy in human-derived LLDM data as a long-term concern.

Checklist

The checklist reports that the paper addresses contributions, limitations, societal impacts, reproducibility, training details, error bars, compute, and asset documentation. The included table is identified but not described further.

  • The authors report discussing their work’s contributions, limitations, and potential negative societal impacts.
  • The authors report providing code, data, reproducibility instructions, training details, and hyperparameter-selection information.
  • The experiments report error bars and the total compute amount and resource types used.
  • The paper reports citing existing assets, addressing licensing where applicable, and releasing new assets through supplemental material or a URL.
  • The checklist marks personally identifiable information, offensive content, and human-subject requirements as not applicable where indicated.

A.1 Neural Architectures

The experiments use three policy architectures—RESNET-RNN, RESNET-T, and VIT-T—and visualize their designs and image encoders. Each architecture incorporates language embeddings differently.

  • The evaluated architectures are RESNET-RNN, RESNET-T, and VIT-T.
  • Figure 6 visualizes the three architectures and shows that they incorporate language embeddings in distinct ways.
  • The image encoders comprise a ResNet-based encoder and a vision-transformer-based encoder.

B Computation

This section describes the lifelong-learning methods, their training context, and the computational setup. The evaluated methods span rehearsal, regularization, and dynamic-architecture approaches.

  • Experiments use a single Nvidia A100 or A40 GPU with 8 16 CPUs for training and evaluation.
  • Lifelong learning seeks continual acquisition and retention of knowledge across an infinite task sequence without catastrophic forgetting.
  • The study implements rehearsal-based ER, regularization-based EWC, and dynamic-architecture PACKNET.ER replays stored samples, EWC constrains updates using parameter importance, and PACKNET prunes, fine-tunes, and freezes parameters.
  • ER stores prior-task data and replays 32 trajectories with each new-task training batch.
  • EWC regularizes updates toward a previous checkpoint using the Fisher information matrix, with an online exponential-moving-average update.The implementation uses γ = 0.9 and λ = 5 · 10^4.
  • PACKNET protects prior-task parameters by iteratively training, pruning, fine-tuning, and freezing network parts.Its implementation prunes 25% of the parameters after training, while available capacity shrinks for later tasks.

C.1 Task Suites

LIBERO provides visualized task suites and an example PDDL scene description for robot manipulation. The task representation specifies objects, regions, initial states, and goal predicates.

  • The benchmark visualizes four task suites: LIBERO-SPATIAL, LIBERO-OBJECT, LIBERO-GOAL, and LIBERO-100.Figure 8 shows initial states because the task goals are shared; the other figures show goal states.
  • The example PDDL scene defines the domain, language instruction, regions, fixtures, objects, and objects of interest.
  • An example task asks the robot to open a cabinet’s top drawer and place a bowl inside it.
  • Region specifications encode targets, coordinate ranges, and yaw rotations for the cabinet, bowl, and plate.

D Experimental Setup

The experimental setup compares five lifelong-learning strategies, including sequential and multitask baselines plus three major lifelong-learning categories.

  • The study evaluates SEQL, MTL, EWC, ER, and PACKNET across task sequences.SEQL learns tasks directly in sequence, whereas MTL learns all tasks simultaneously.
  • SEQL and MTL approximate lower and upper performance bounds for lifelong-learning algorithms.
  • EWC, ER, and PACKNET represent regularization-based, rehearsal-based, and dynamic-architecture approaches, respectively.

E.1 Full Results

The full-results section compares algorithms and policy architectures across task suites using learning curves, loss-versus-success analyses, and attention visualizations. Results show architecture- and metric-dependent behavior, including misleading behavioral-cloning loss and task-irrelevant attention.

  • Full results: Table 8 reports results for three lifelong-learning algorithms and three policy architectures across four task suites, highlighting the top three AUC scores.
  • Learning-curve comparisons: Performance plots compare lifelong-learning algorithms with fixed architectures and architectures with fixed algorithms over 10-task sequences.
  • Loss versus success rates: Behavioral-cloning loss can be misleading because lower loss does not necessarily correspond to higher robot task success.
  • Loss versus success rates: EWC can achieve the lowest loss yet poor success, whereas ER can have the highest loss yet outperform EWC.
  • Loss versus success rates: Success rate, rather than behavioral-cloning loss, is recommended for evaluating whether a policy checkpoint is effective.
  • Attention visualization: Attention maps show that policies often prioritize the robot arm and target-placement area, but may attend to task-irrelevant blank table regions.
  • Attention visualization: These spurious correlations can cause policies to overfit tasks and generalize poorly across tasks.
  • Attention visualization: ViT attention is more temporally consistent than ResNet attention, while PACKNET maintains more consistent attention across learning.
Loading 2306.03310v2…