PaperlayerView exact revision on arXiv

Source-linked AI summary

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei-Fei, Silvio Savarese, Yuke Zhu, Roberto Martín-Martín

arXiv:2108.03298v2cs.ROcs.AIcs.LG

TL;DR

Offline learning from human demonstrations remains difficult to assess because suitable datasets and reproducible methods are limited. The paper studies six algorithms across diverse simulated and real-world tasks and finds that temporal modeling and careful design are especially important, while complex-task learning and real-world transfer show promise.

  • Problem

    Suitable open human datasets and reproducible methods are limited, making offline learning for robot manipulation difficult to assess.

  • Method

    The paper evaluates six offline learning algorithms on human datasets spanning simulated and real-world manipulation tasks, data sources, and observation spaces.

  • Results

    History-dependent models were effective on human datasets, while batch RL struggled; larger human datasets supported complex-task policies and simulation-tuned choices transferred to real-world tasks.

  • Takeaways & Limitations

    Human datasets offer a promising route to proficient policies for challenging manipulation tasks and to reproducible evaluations whose conclusions can transfer to real-world settings.

  • Takeaways & Limitations

    Batch RL methods still need improved ability to learn from suboptimal human datasets, and offline policy selection remains important for real-world settings.

Abstract

from arXiv · show

Imitating human demonstrations is a promising approach to endow robots with various manipulation capabilities. While recent advances have been made in imitation learning and batch (offline) reinforcement learning, a lack of open-source human datasets and reproducible learning methods make assessing the state of the field difficult. In this paper, we conduct an extensive study of six offline learning algorithms for robot manipulation on five simulated and three real-world multi-stage manipulation tasks of varying complexity, and with datasets of varying quality. Our study analyzes the most critical challenges when learning from offline human data for manipulation. Based on the study, we derive a series of lessons including the sensitivity to different algorithmic design choices, the dependence on the quality of the demonstrations, and the variability based on the stopping criteria due to the different objectives in training and evaluation. We also highlight opportunities for learning from human datasets, such as the ability to learn proficient policies on challenging, multi-stage tasks beyond the scope of current reinforcement learning methods, and the ability to easily scale to natural, real-world manipulation scenarios where only raw sensory signals are available. We have open-sourced our datasets and all algorithm implementations to facilitate future research and fair comparisons in learning from human demonstration data. Codebase, datasets, trained models, and more available at https://arise-initiative.github.io/robomimic-web/

1 Introduction

Offline human datasets could help narrow the gap between robot and human manipulation, but robotics lacks suitable large-scale datasets and reproducible benchmarks. This study addresses that gap across diverse manipulation tasks and datasets.

  • Motivation: Robot manipulation has lagged behind other domains despite progress in offline learning from human demonstrations.The paper frames the gap as one between autonomous robot capabilities and the wide range of tasks humans solve effortlessly.
  • Challenges: Fine-grained real-time control makes robotic data collection technically challenging and helps explain the scarcity of large human-provided datasets.Human demonstrations also differ in decision process, data quality, and solution strategy.
  • Study scope: The study evaluates offline policy learning across 8 challenging tasks spanning pick-and-place, multi-arm coordination, and high-precision insertion and assembly.Datasets were collected across operators with varying proficiency.
  • Research gap: Existing studies were limited by a lack of suitable benchmarks and human datasets, often focusing on simple 2D environments or hard-coded-policy data.The paper responds by collecting demonstrations across simulated and real-world manipulation tasks.
  • Main findings: History-dependent models were effective on human datasets, whereas state-of-the-art batch RL algorithms struggled; observation and hyperparameter choices also substantially affected performance.The study also found promise for complex tasks and transfer of insights to real-world scenarios.

2 Challenges in Offline Learning from Human Datasets

The study identifies five recurring challenges in offline learning from human demonstrations: temporal dependence, heterogeneous quality, dataset size, objective mismatch, and sensitivity to design choices.

  • C1: Non-Markovian decision process: Human demonstrations may depend on external factors and past actions rather than only the current observation, motivating history-dependent or temporally abstract models.The paper treats this as a non-Markovian decision-process challenge.
  • Evidence across datasets: Methods modeling temporal correlations performed strongly on human datasets, while batch RL methods such as BCQ performed poorly there despite proficiency on machine-generated data.Table 1 compares success rates across machine-generated, proficient-human, and multi-human datasets.
  • C2: Demonstration quality: Data from multiple humans can vary substantially in proficiency and solution strategy, including trajectory length, movement noise, and mistakes.The study evaluates algorithms on datasets containing such variation.
  • C3: Dataset size: Offline policy performance depends on dataset coverage and size, making the value of collecting additional human demonstrations an important practical question.The paper studies how reducing dataset size affects performance.
  • C4: Objective mismatch: Training loss is only a surrogate for task success, so policy performance can vary across checkpoints and make offline model selection difficult.The study evaluates checkpoints online in simulation to compare selection criteria.
  • C5: Design sensitivity: Agent architecture, observation space, and hyperparameters can strongly affect policy performance, extending sensitivity previously observed on machine-generated datasets.The study examines these choices to derive practical recommendations.

3 Study Design

The study spans eight manipulation tasks, multiple human-data sources and observation spaces, and six offline learning algorithms. It varies dataset size and training choices while evaluating success across checkpoints.

  • Tasks: The benchmark covers 5 simulated and 3 real-world tasks selected to test a broad range of manipulation capabilities.Tasks include simple lifting, placement, precision insertion, multi-arm coordination, and multi-stage assembly.
  • Data sources: Datasets come from machine-generated rollouts, one proficient teleoperator, or six human teleoperators with varying proficiency.The proficient-human set contains 200 demonstrations, while the multi-human set contains 300.
  • Observation spaces: Low-dimensional observations use ground-truth object information, whereas image observations replace it with camera observations while retaining robot state features.The collected sensor streams include external and wrist-mounted cameras.
  • Algorithms and evaluation: The study evaluates BC, BC-RNN, HBC, BCQ, CQL, and IRIS using binary completion rewards and repeated rollout-based success evaluation.Each agent is evaluated every E epochs, and the maximum success rate during training is reported.
  • Design studies: BC-RNN success can drop drastically when observation spaces or hyperparameters change.The figure isolates sensitivity to these design choices in the training protocol.
  • Dataset-size study: Reducing data to 20% or 50% is less damaging for Lift and Can than for more complex tasks, which may benefit from larger human datasets.The smaller subsets were sampled uniformly per contributing human.

4 Experiments

The experiments show that history-dependent modeling, observation design, dataset quality, and training choices strongly affect offline learning from human demonstrations. BC-RNN is consistently effective across challenging human datasets and real-world tasks, while batch RL methods struggle on human data.

  • Algorithm Comparison: BC-RNN outperforms BC across human datasets, with history providing larger gains on longer-horizon and multi-human tasks.The gap is approximately 55% for Transport (PH) versus 5% for Square (PH), and approximately 25% for Square (MH) versus 5% for Square (PH).
  • Algorithm Comparison: Batch RL algorithms perform poorly on human datasets despite strong results on mixed-quality agent-generated data.The experiments compare BCQ and CQL on human demonstrations and find that neither performs particularly well there.
  • Suboptimal Human Data: Adding lower-quality demonstrations uniformly improves BC-RNN, although mixed-quality data remains below high-quality human data on Square.BC-RNN reaches 55.3%, 73.3%, and 74.0% on 200-demo Square mixtures, compared with 84.0% on the high-quality 200-demo PH dataset.
  • Suboptimal Human Data: In the Can-Paired diagnostic dataset, most batch RL algorithms struggle to distinguish successful from failed demonstrations, while IRIS improves over BC-RNN by 5%.The dataset contains paired successful and failed demonstrations for each of 100 task initializations.
  • Observation Space: Image-based BC-RNN nearly matches ground-truth-observation performance, except on Transport (MH), and improves over BC on Square and Transport.This supports the feasibility of learning from raw sensory observations for real-world manipulation.
  • Observation Space: Removing pixel-shift randomization causes 47% and 35% relative drops on Square and Transport, while removing wrist images causes 9% and 43% drops.Both wrist-camera images and image randomization substantially affect image-based policy performance.
  • Hyperparameters: Increasing the learning rate from 1e-4 to 1e-3 drops image-agent performance by 35%-63%, while deterministic policies reduce MH performance, especially on low-dimensional Transport by 58%.Larger MLPs also reduce performance uniformly, suggesting overfitting to dataset actions.
  • Dataset Size: Less complex tasks reach 75%-100% success using 20% of the data, whereas complex Square and Transport tasks improve substantially as dataset size increases.These results support the value of large human datasets for challenging manipulation tasks.

5 Discussion

The study identifies temporal modeling, observation design, data quality, and offline policy selection as central factors in learning from human demonstrations. It also finds promise for complex tasks and transfer from simulation to real-world settings.

  • Lessons: History-dependent models are particularly effective on human datasets compared with algorithms without temporal context.The study highlights BC-RNN, HBC, and IRIS as effective examples.
  • Lessons: State-of-the-art batch RL algorithms remain much worse at learning from suboptimal human datasets than from suboptimal machine-generated datasets.The study recommends benchmarking batch RL methods on human data and combining history dependence with value learning.
  • Lessons: Offline policy selection remains important because the best evaluation checkpoint is difficult to identify in real-world settings.The discussion calls for better offline methods to select an evaluation policy.
  • Lessons: Observation space and hyperparameters substantially affect policy learning, including proprioception, pixel-shift randomization, and wrist-camera images.Low-dimensional policies are sensitive to proprioceptive features, while image-based policies depend on randomization and wrist views.
  • Opportunities: Large-scale human datasets show promise for challenging manipulation because more data improved complex-task policies and proficient Tool Hang policies were learned without task-specific tuning.Tool Hang was the study’s most complex task.
  • Opportunities: Hyperparameters tuned on simulated tasks transferred directly to real-world datasets and tasks.Tool Hang and real tasks were used for evaluation rather than hyperparameter tuning.

B Dataset Details

The dataset details use average trajectory length to characterize dataset variants and determine evaluation rollout horizons. Trajectory length serves as a proxy for demonstration quality.

  • Dataset statistics: Table 4 reports mean and standard deviation of average trajectory lengths for each dataset variant.These statistics characterize the trajectory-length distribution across datasets.
  • Dataset quality: Longer demonstrations indicate lower dataset quality because less proficient humans took more time to complete tasks.Trajectory length is treated as a proxy for dataset quality.
  • Evaluation: Average trajectory length determines the evaluation rollout horizon for each dataset.The rollout horizon is dataset-specific rather than fixed across all variants.

B.1 Data Collection

The study constructs machine-generated and human demonstration datasets, defines an offline manipulation-learning setup, and evaluates imitation-learning and batch-RL algorithms under task-completion rewards.

  • Dataset construction: Machine-generated datasets contain mixtures of expert and suboptimal trajectories collected from RL-agent checkpoints on Lift and Can.Other tasks were excluded because the RL algorithm could not solve them even with substantial tuning.
  • Dataset construction: Human datasets were collected through remote teleoperation by operators with varying proficiency and geographic distance from the simulation server.The operators included better, okay, and worse quality groups.
  • Dataset construction: The Can-Paired dataset contains 200 demonstrations arranged as 100 matched good-bad trajectory pairs from identical task initializations.Good trajectories place the can correctly, whereas bad trajectories toss it outside the workspace.
  • Dataset preparation: Smaller datasets were sampled uniformly per contributing human and split into 90% training and 10% validation subsets.Uniform sampling avoided bias toward higher- or lower-quality demonstrations.
  • Problem formulation: The offline setting learns a policy from trajectory data without collecting additional samples from the Markov decision process.The formulation uses an offline dataset of state-action-reward trajectories.
  • Training signals: Binary task-completion rewards and episode-end signals are used by batch offline reinforcement-learning algorithms.A reward is one when the next state belongs to the goal set, and zero otherwise.
  • Algorithms: BC clones demonstrated actions, BC-RNN models temporal dependencies, HBC predicts subgoals hierarchically, and IRIS adds value learning to HBC’s high-level policy.BCQ learns value estimates while constraining candidate actions using a generative action model.

D.2 Hyperparameter Selection Procedure

Hyperparameters are selected separately for machine-generated, proficient-human, and multi-human datasets, then transferred to held-out tasks including Tool Hang and real-world settings. The procedure tests architecture, optimization, and representation choices across algorithms.

  • Selection protocol: Hyperparameters are tuned separately for machine-generated, proficient-human, and multi-human dataset settings.Shared settings are selected using representative datasets and then applied across environments within each setting.
  • Selection protocol: Tool Hang and all real tasks are evaluation-only, so no hyperparameter tuning occurs on those tasks.This tests whether simulation-derived insights transfer to the hardest simulation task and real robot datasets.
  • Scope: HBC and IRIS are excluded from image-based training because their subgoal reconstructions may be problematic for high-dimensional images.This is an explicit scope limitation of the comparison.
  • BC and BC-RNN: Lower learning rates perform better consistently for BC and BC-RNN.The comparison evaluates 1e−3 and 1e−4 learning rates.
  • BC and BC-RNN: BC benefits from higher-capacity action networks, whereas BC-RNN performs better without an additional MLP.The authors hypothesize that the RNN already has sufficient capacity and that larger actor networks can overfit.
  • CQL: For CQL, deterministic backups outperform nondeterministic backups, although the gains can be marginal.The deterministic variant is selected for subsequent experiments.
  • CQL: For low-dimensional CQL experiments, increasing batch size from 100 to 1024 improves stability and performance, while image experiments use batch size 8 due to GPU memory limits.The batch-size choice therefore differs by observation modality.
  • Final settings: The finalized hyperparameters are organized by algorithm and dataset setting in Tables 6–15.Columns distinguish proficient-human, multi-human, and machine-generated settings.

E Additional Details on Task Environments

The study uses Panda arms in matched simulation and real-world manipulation environments, with low-dimensional or image observations and task-specific randomization. Policies operate at 20 Hz, while real demonstrations are collected from robot sensors and cameras at approximately the same rate.

  • Task demonstrations: Appendix examples show complete trajectories for Transport, Tool Hang, and real-world Tool Hang to illustrate their task stages.
  • Workspace and setup: Simulation tasks use MuJoCo and robosuite with Panda robotic arms, while physical experiments use a Franka Emika Panda arm.
  • Robot control: Each arm outputs a 7-dimensional action containing translation, axis-angle rotation, and gripper control.Actions are produced at 20 Hz and converted into target poses for a 500 Hz operational-space controller.
  • Workspace and setup: The physical workspace combines fixed table-mounted cameras with a wrist-mounted camera, and demonstrations are collected from robot sensors at approximately 20 Hz.The simulation and real tasks were designed to approximately match appearance, dimensions, and initialization randomizations.
  • Observation spaces: Low-dimensional agents receive proprioception and task-specific object observations, whereas image agents receive external and wrist-camera images.Proprioception contains end-effector pose and gripper positions for each arm.
  • Observation spaces: Image inputs use task-dependent camera counts and resolutions, including four views for Transport and 240 by 240 images for Tool Hang.Other simulation tasks use 84 by 84 images, while most real tasks use 120 by 120 images.

G Additional Results on Policy Selection

Policy selection is difficult because checkpoint performance can fluctuate substantially and validation loss does not reliably identify the best policy. Success rate may continue increasing even as validation loss worsens, making online evaluation important.

  • Checkpoint variability: Policy checkpoint performance can vary substantially during training, including during later stages and on simpler image-based tasks.This variance makes offline policy selection difficult in real-world settings where many rollouts per checkpoint are impractical.
  • Validation-based selection: Using 30% rather than 10% validation data does not make validation loss a reliable policy-selection criterion.On Square (PH), the best policy reaches 80.7±0.9 versus 2.7±1.9 for the lowest-validation-loss policy; on Transport (PH), the corresponding values are 64.0±2.8 and 0.7±0.9.
  • Training versus evaluation: Success rate can keep increasing while validation loss increases substantially, because the best validation loss occurs earlier than the best policy performance.The best validation loss occurs around epochs 100–300, whereas best performance occurs much later.
  • Training versus evaluation: The results show that validation loss is a poor measure of policy performance for selecting offline policy checkpoints.

H.1 Machine-Generated Datasets

On image-based machine-generated datasets, BCQ outperforms the other compared methods, whereas on multi-human image datasets, batch RL methods are omitted because of poor performance and BC-RNN improves over BC.

  • Machine-generated datasets: BCQ outperforms the other evaluated methods across image-based Machine-Generated datasets.The authors note that newer batch RL methods shown to work on pixel observations might perform better.
  • Multi-human datasets: Batch RL methods are excluded from most multi-human image-data comparisons because of poor performance on human datasets.Can-Paired is the exception among the reported human-data image results.
  • Multi-human datasets: BC-RNN improves over BC on every evaluated multi-human image dataset, with especially strong gains on lower-quality data.

I.1.1 BCQ

BCQ and CQL are highly sensitive to design and hyperparameter choices on human datasets. BCQ’s perturbation actor can sharply reduce performance, while CQL benefits from larger batches and selected algorithmic variants.

  • BCQ sensitivity: BCQ’s perturbation actor causes performance drops exceeding 80% on some human datasets, including Can-PH.
  • BCQ sensitivity: Matching BCQ’s action-sampler hyperparameters to BC does not eliminate BCQ’s underperformance relative to BC.The matched settings use the same learning rate, MLP architecture, and Gaussian Mixture Model.
  • CQL sensitivity: A 10× smaller learning rate decreases CQL success rate by over 50%, while reducing batch size from 1024 to 100 decreases performance by over 30%.These results indicate sensitivity to both learning rate and batch size on low-dimensional Lift datasets.
  • CQL sensitivity: CQL’s deterministic backup and Lagrange variants produce substantial improvements on Machine-Generated data but only marginal improvements on Proficient-Human data.
  • Evaluation choices: During evaluation, the GMM low-noise trick samples a mode rather than the full learned distribution.The learned standard deviation of each mode is replaced with 1e-4.

K Additional Results on Multi-Human Datasets

This section reports supplementary Multi-Human dataset results and detailed tables corresponding to the paper’s main-text studies. The additional observation-space results are consistent with the reported proficient-human findings, while further tables cover suboptimal human-data mixtures and several ablations.

  • Observation Space Study: The Multi-Human observation-space study reproduces the main-text analysis, with results and conclusions reported as consistent.The main text presented the corresponding study on Single-Human or proficient-human datasets.
  • Suboptimal Human Data: Tables 23 and 24 report success rates averaged over 3 seeds across Multi-Human Lift and Transport subsets.The subsets mix demonstrations from “Better”, “Adequate”, and “Worse” human operators, with Table 24 covering image observations.
  • Detailed Tables: The supplementary tables provide detailed results corresponding to the main-text studies in Figures 2, 3, and 4a.These include observation-space analysis, BC-RNN hyperparameter sensitivity, dataset-size ablations, and policy-selection criteria.
  • Ablation Results: Table 26 documents BC-RNN hyperparameter sensitivity corresponding to Figures 2b and 2c.It expands the main-text analysis with a dedicated table of hyperparameter results.
  • Ablation Results: Tables 27 and 28 report proficient-human and Multi-Human dataset-size ablations corresponding to Figure 3.The tables separate the two dataset types rather than combining their ablation results.
  • Policy Selection: Table 29 compares policy selection using lowest validation loss or the final checkpoint against the best-performing policy.It corresponds to Figure 4a and reports how performance decreases under the alternative selection criteria.
Loading 2108.03298v2…