PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Offline, Test Online: A Real Robot Learning Benchmark

Gaoyue Zhou, Victoria Dean, Mohan Kumar Srirama, Aravind Rajeswaran, Jyothish Pari, Kyle Hatch, Aryan Jain, Tianhe Yu, Pieter Abbeel, Lerrel Pinto, Chelsea Finn, Abhinav Gupta

arXiv:2306.00942v2cs.ROcs.AIcs.CVcs.LG

TL;DR

Robot learning lacks affordable, comparable real-world evaluation and sufficiently broad shared data. TOTO combines offline training on a public manipulation dataset with remote testing on shared hardware, and initial baselines show challenging but feasible generalization; manual evaluation and a single current setup remain limitations.

  • Problem

    Robotics benchmarking is hindered by expensive hardware, cross-lab variation, and limited shared real-world data.

    Source: §I · p. 1

  • Method

    TOTO provides a public manipulation dataset for offline learning and shared hardware for remote evaluation on common tasks.

    Source: §I · p. 1, B. Tasks · p. 2, C. Dataset · p. 3

  • Results

    Initial baselines show that TOTO tasks are challenging yet feasible, with VINN performing best in train locations and offline-RL methods achieving some success.

    Source: B. Policy Learning Results · p. 5, §VII · p. 6

  • Takeaways & Limitations

    TOTO enables researchers to train, remotely evaluate, and directly compare methods without acquiring hardware or collecting task data.

    Source: §I · p. 1, §VII · p. 6

    Reward measurement and resets currently require manual steps, and expanding to additional robots may introduce visual differences.

    Source: A. Limitations and Future Work · p. 6

Abstract

from arXiv · show

Three challenges limit the progress of robot learning research: robots are expensive (few labs can participate), everyone uses different robots (findings do not generalize across labs), and we lack internet-scale robotics data. We take on these challenges via a new benchmark: Train Offline, Test Online (TOTO). TOTO provides remote users with access to shared robotic hardware for evaluating methods on common tasks and an open-source dataset of these tasks for offline training. Its manipulation task suite requires challenging generalization to unseen objects, positions, and lighting. We present initial results on TOTO comparing five pretrained visual representations and four offline policy learning baselines, remotely contributed by five institutions. The real promise of TOTO, however, lies in the future: we release the benchmark for additional submissions from any user, enabling easy, direct comparison to several methods without the need to obtain hardware or collect data.

I. INTRODUCTION

TOTO addresses robotics benchmarking barriers by combining a shared offline dataset with remotely accessible hardware, enabling fairer comparisons across methods. It is designed to lower participation costs while preserving real-world evaluation.

I. INTRODUCTION

  • Robotics lacks common benchmarks because labs differ in hardware, objects, conditions, and infrastructure, while acquiring and installing equipment limits participation.Source: §I · p. 1
  • Existing standardized tasks and shared physical challenges still leave training infrastructure and method-development conditions to individual labs.Source: §I · p. 1
  • Offline training enables common-data evaluation, avoiding the cross-lab variability of online real-world training and the transfer concerns of simulation.Source: §I · p. 1
  • TOTO combines a publicly released manipulation dataset with shared hardware so participants train on identical data and receive apples-to-apples evaluations.Source: §I · p. 1
  • TOTO lowers entry barriers by letting researchers train, remotely evaluate, and compare against baselines without setting up hardware or collecting task data.Source: §I · p. 1

A. Shared Tasks and Environments

TOTO extends shared-task and shared-robot benchmarking by pairing remote real-world evaluation with an open dataset and offline learning focus. Its contribution is a common platform that matches training data and physical evaluation.

A. Shared Tasks and Environments

  • Unlike shared tasks or environments alone, TOTO provides a common physical setup so users need not recreate identical hardware conditions in their own labs.Source: A. Shared Tasks and Environments · p. 2

B. Shared, Remote Robots

  • Remote shared robots reduce the need for users to acquire hardware, while TOTO specifically targets generalization through varied lighting, unseen objects, and image-based observations.Source: B. Shared, Remote Robots · p. 2

C. Open-Source Robotics Datasets

  • TOTO addresses the difficulty of interpreting open robotics datasets by pairing its 2,898-trajectory initial dataset with matching real-world evaluation that can grow through user submissions.Source: C. Open-Source Robotics Datasets · p. 2

D. Offline Robot Learning

III. THE TOTO BENCHMARK

  • TOTO centers on manipulation, supplying hardware, benchmark tasks, an initial dataset, and a common evaluation protocol.Source: §III · p. 2

A. Hardware

The benchmark uses a real robot arm and RGB-D observations to evaluate pouring and scooping under varied object and location conditions. The task design emphasizes manipulation generalization beyond standard setups.

A. Hardware

  • The hardware uses a Franka Emika Panda with joint-position control and an Intel D435 RealSense camera for RGB-D observations.Source: A. Hardware · p. 2
  • Users may lower control frequency, and absolute joint-angle actions allow training data to be subsampled by frame stride.Source: A. Hardware · p. 2

B. Tasks

  • The task suite contains pouring and scooping, with randomized starts, noisier target locations, varied objects, and unnormalized rewards based on transferred material weight.Source: B. Tasks · pp. 2–3
  • Scooping transfers material from a bowl into a spoon across three bowl types, three materials, and six training locations.Source: B. Tasks · p. 3
  • Pouring transfers granular material from a gripped cup into a target cup across four target cups, two materials, and six training locations.Source: B. Tasks · p. 3

C. Dataset

TOTO releases a diverse manipulation dataset and evaluates policies across seen and unseen objects, materials, and locations. The protocol uses repeated seeds and reports trial means and variances.

C. Dataset

  • The dataset contains 1,000–2,000 trajectories per task, with images, joint-target actions, joint states, and rewards collected through multiple techniques.Source: C. Dataset · p. 3
  • Teleoperation provides most trajectories through intuitive end-effector control, with multiple operators contributing diversity and mostly successful trials.Source: C. Dataset · p. 3
  • Behavior-cloning rollouts add action noise while using state-based policies trained for each target location.Source: C. Dataset · p. 3
  • Replay of teleoperated trajectories adds further diversity because lighting changes with time of day.Source: C. Dataset · p. 3

D. Evaluation Protocol

  • Evaluation uses two unseen objects, one unseen material, three seen and three unseen locations, and 18 trials per method for each location split.Source: D. Evaluation Protocol · p. 3

IV. BENCHMARK USE

TOTO enables remote offline training and shared-hardware evaluation through an accessible workflow and documented software infrastructure.

A. User Workflow

  • TOTO users train policies locally on a shared real-world dataset and submit them for evaluation on common hardware.Users receive video, reward, and success-rate feedback rather than low-level evaluation data.Source: A. User Workflow · p. 3
  • Evaluation is supervised and currently takes around 12 hours, while the benchmark emphasizes offline learning rather than real-time results.Source: A. User Workflow · pp. 3–4
  • A public leaderboard will report anonymous evaluation scores, and later user-policy data may be added to the original dataset.Source: A. User Workflow · p. 4

B. Software Infrastructure

  • The starter kit provides documented training code, open-source baselines, trajectory data, pretrained models, and a containerized environment.A stub environment lets users test agent compatibility before submission.Source: B. Software Infrastructure · p. 4
  • A standard agent interface and configuration format specify how users initialize policies and map observations to actions.Source: B. Software Infrastructure · p. 4

V. BASELINES

TOTO establishes baseline questions about visual representations for manipulation and compares in-domain and generic pretrained models contributed by multiple labs.

A. Visual Representation Baselines

B. Policy Learning Baselines

TOTO compares policy learners ranging from nearest-neighbor imitation to offline reinforcement learning, using RGB observations and optional frozen visual encoders.

B. Policy Learning Baselines

  • The baselines span nearest-neighbor querying, behavior cloning, and offline reinforcement learning, with RGB images processed by some frozen vision models.Source: B. Policy Learning Baselines · p. 4
  • BC predicts 50-step action sequences between replanning steps, filters training data to successful trajectories, and outperforms purely closed- or open-loop variants.Source: B. Policy Learning Baselines · p. 4
  • VINN [36] combines BYOL embeddings with nearest-neighbor action selection and adopts the same 50-step quasi-open-loop execution as BC.Source: B. Policy Learning Baselines · p. 4
  • IQL [38] concatenates frozen MoCo (In-Domain) image embeddings with robot joint angles as its input state.Source: B. Policy Learning Baselines · p. 4
  • Decision Transformer models actions from RGB embeddings, joint angles, trajectory history, and a target-return prompt.Source: B. Policy Learning Baselines · p. 4
  • Figure 3 compares vision representations with BC, separating models trained on TOTO data from generic models and contrasting training with unseen object locations.Source: Figure 3 · p. 5

A. Visual Representation Comparison Using BC

The experiments compare visual representations and policy learners across train and test locations, showing strong in-domain visual performance but degradation for policy generalization.

A. Visual Representation Comparison Using BC

B. Policy Learning Results

  • Offline reinforcement learning methods achieve some success unlike in RB2, likely supported by TOTO’s larger and more diverse dataset.Source: B. Policy Learning Results · p. 5
  • Scooping is non-Markovian because the spoon is above the bowl before and after the action, yet IQL performs decently without history while DT has only moderate success.Source: B. Policy Learning Results · p. 5
  • VINN performs best on training locations, but all policy-learning methods degrade on unseen locations, with VINN particularly affected by nearest-neighbor dependence on training trajectories.Source: B. Policy Learning Results · p. 5, Table III · p. 5

C. Dataset Size Ablation

The ablation finds that more data generally improves BC success, but gains diminish near the full dataset, while offline action-error metrics provide a useful but imperfect real-world signal.

C. Dataset Size Ablation

  • Training on more data generally leads to higher success, while using all trajectories performs worse than using only successful trajectories for BC.The ablation compares top 5%, 25%, 50%, 100%, and approximately 70% successful trajectories.Source: C. Dataset Size Ablation · p. 5
  • VINN performs best on training locations, but its performance and that of other methods degrade on unseen locations.The supplied figure passage identifies this comparison but provides no numerical values.Source: Figure 4 · p. 6, Table IV · p. 6
  • The TOTO dataset is approximately the right size for policy learning because increasing training data from 50% to 70% does not substantially improve performance.Additional data may still improve visual representation learning.Source: C. Dataset Size Ablation · p. 6

D. Metrics for Offline Policy Evaluation

  • Offline evaluation estimates joint-angle and end-effector-pose action errors against expert demonstrations from a validation set of 100 trajectories.These metrics compare predicted actions with demonstrated actions using mean squared error.Source: D. Metrics for Offline Policy Evaluation · p. 6
  • As validation errors decrease, real-world reward increases, while an overtrained 2,000-epoch policy has higher error and lower reward.The metrics provide a signal for likely real-world success but should not fully guide algorithm development alone.Source: D. Metrics for Offline Policy Evaluation · p. 6

VII. DISCUSSION

TOTO’s discussion presents a feasible community workflow for remotely evaluating locally trained policies, while identifying manual evaluation and scaling constraints for future work.

VII. DISCUSSION

  • Collaborators locally train policies and submit them for remote evaluation on shared hardware, demonstrating the feasibility of TOTO’s intended user workflow.The initial baselines also indicate that the benchmark tasks are challenging, particularly for generalization.Source: §VII · p. 6

A. Limitations and Future Work

  • TOTO’s evaluation protocol still requires manual reward measurement and resetting, with spills potentially requiring further manual intervention.The current process measures transferred material and returns it to the original object.Source: A. Limitations and Future Work · p. 6
  • Expanding TOTO to additional robots could increase evaluation capacity, but visual differences across robots require additional demonstrations.The expansion could also support a larger task set, including assigning different robots to different tasks.Source: A. Limitations and Future Work · p. 6
  • As demand grows, an evaluation job queue is planned to prioritize requests and schedule them according to available robots.Source: A. Limitations and Future Work · p. 6

VIII. APPENDIX

The appendix describes a two-task manipulation benchmark, its reward data, and the community challenges for contributing visual representations or agent policies.

A. Task Specifications

B. Dataset

  • Reward distributions are reported for both tasks, with multiple pouring peaks attributed to differences in the weights of training materials.Source: B. Dataset · p. 8, Figure 8 · p. 8
  • Rewards are unnormalized weights in grams of material successfully scooped or poured, directly measuring the amount of manipulated material.Source: B. Dataset · p. 8

C. Get Started with TOTO

  • TOTO invites participants to submit BC agents using pretrained visual representations or agent policies using custom or provided representations.The benchmark provides instructions, code, and dataset access through its website.Source: C. Get Started with TOTO · p. 8
Loading 2306.00942v2…