Source-linked AI summary
RoboNet: Large-Scale Multi-Robot Learning
Sudeep Dasari, Frederik Ebert, Stephen Tian, Suraj Nair, Bernadette Bucher, Karl Schmeckpeper, Siddharth Singh, Sergey Levine, Chelsea Finn
TL;DR
Robotic learning needs diverse data to generalize beyond single robots and environments, but collecting separate datasets is costly. RoboNet pools large-scale real-world experience across robots and domains and combines it with visual foresight and inverse models. Pre-training on RoboNet followed by limited fine-tuning supports transfer to unseen hardware, although model capacity and task simplicity constrain performance.
Problem
Robotic learning aims to generalize across open-world settings, but experiments commonly remain small-scale, single-domain, and single-robot with limited data sharing.
Method
RoboNet is an extensible, self-supervised dataset of multi-robot interaction experience used with visual foresight and supervised inverse models.
Results
Pre-training on RoboNet supports zero-shot transfer to novel objects, viewpoints, and table surfaces, while around 400 trajectories enable transfer to unseen grippers and robot platforms.
Takeaways & Limitations
Shared multi-robot data can provide useful priors for rapidly acquiring vision-based skills on unseen robot hardware.
Takeaways & Limitations
The evaluated tasks are relatively simple, low-fidelity manipulation tasks, limiting immediate practical use.
Abstract
from arXiv · showhide
Robot learning has emerged as a promising tool for taming the complexity and diversity of the real world. Methods based on high-capacity models, such as deep networks, hold the promise of providing effective generalization to a wide range of open-world environments. However, these same methods typically require large amounts of diverse training data to generalize effectively. In contrast, most robotic learning experiments are small-scale, single-domain, and single-robot. This leads to a frequent tension in robotic learning: how can we learn generalizable robotic controllers without having to collect impractically large amounts of data for each separate experiment? In this paper, we propose RoboNet, an open database for sharing robotic experience, which provides an initial pool of 15 million video frames, from 7 different robot platforms, and study how it can be used to learn generalizable models for vision-based robotic manipulation. We combine the dataset with two different learning algorithms: visual foresight, which uses forward video prediction models, and supervised inverse models. Our experiments test the learned algorithms' ability to work across new objects, new tasks, new scenes, new camera viewpoints, new grippers, or even entirely new robots. In our final experiment, we find that by pre-training on RoboNet and fine-tuning on data from a held-out Franka or Kuka robot, we can exceed the performance of a robot-specific training approach that uses 4x-20x more data. For videos and data, see the project webpage: https://www.robonet.wiki/
1 Introduction
RoboNet addresses limited data sharing and poor robustness in robotic learning by pooling diverse real-world experience across robots and environments. The paper evaluates whether this shared data supports broad generalization and rapid adaptation.
- Motivation and contribution: RoboNet contains roughly 162,000 trajectories and nearly 15 million video frames from 7 robots, hundreds of objects, varied viewpoints, and environments.Data collection is autonomous, minimally supervised, self-supervised, and designed for extension to new hardware, sensors, and policies.
- Motivation and contribution: The paper studies whether sharing data across robots and environments enables pre-training models that transfer priors to new robots and scenes.The intended outcome is faster learning without re-collecting all experience for each environment.
- Methods: The evaluation combines visual foresight and inverse-model approaches for self-supervised learning from autonomously collected data.Visual foresight uses action-conditioned video prediction to plan toward user-specified goals.
- Results: RoboNet-pretrained models generalize zero-shot to novel objects, viewpoints, and table surfaces, and with around 400 trajectories adapt to unseen grippers and robot platforms.The adapted models outperform robot-specific and environment-specific training in the reported experiments.
2 Related Work
Prior robotics research studied transfer across selected domains, dynamics, morphologies, viewpoints, objects, tasks, or environments. RoboNet instead emphasizes a shared, large-scale dataset spanning multiple robots and institutions to study broad transfer.
- Positioning RoboNet: Most deep-learning robotics work focuses on a single robot and domain, whereas RoboNet targets one model that generalizes across multiple robots and domains.The framing connects to multi-task, lifelong-learning, and meta-learning research.
- Positioning RoboNet: Large-scale self-supervised robotics methods share RoboNet’s data-driven perspective, but RoboNet specifically studies transfer across multiple robots and environments for cross-institutional data sharing.The paper uses visual foresight to support task generalization without reward signals.
- Transfer-learning literature: Related approaches study transfer from simulation, across dynamics or morphologies, through viewpoints, or across objects, tasks, and environments.These methods address particular transfer axes rather than RoboNet’s combined multi-robot, multi-domain setting.
- Dataset diversity: Inspired by ImageNet, RoboNet emphasizes diversity by combining 7 robots, data from 4 institutions, dozens of backgrounds, and hundreds of viewpoints.This breadth makes broad generalization in robotics experimentally testable.
3 Data-Driven Robotic Manipulation
The paper frames manipulation as image-based object relocation learned from self-supervised image-action trajectories. It evaluates action selection through predictive visual control and supervised inverse models.
- Data-driven assumption: The approach assumes common-sense knowledge, robot kinematics, and object physics are learned implicitly from data rather than supplied explicitly.This is a data-driven modeling assumption rather than an input to the controller.
- Problem formulation: The task is moving objects to specified locations by pushing or grasping and placing, using camera images rather than explicit environmental models.Data collection uses randomly sampled actions and requires only initial action programming and periodic introduction of objects.
- Visual foresight: Visual foresight predicts future images from current images and action sequences, then optimizes actions for desired designated-pixel motion.Executing the first action, observing a new image, and replanning implements image-based model-predictive control.
- Inverse models: The inverse model predicts actions needed to move from a current observation and context frames to a goal image.The experiments use a one-step supervised model and repeat prediction in closed loop for multi-step control.
4 The RoboNet Dataset
RoboNet is an open, extensible repository of self-supervised robot interaction data collected across hardware, environments, viewpoints, and institutions. Its shared action representation and contribution infrastructure support training and filtering across varied conditions.
- Dataset scope: RoboNet combines experience from 7 robots and 4 institutions under loose scene specifications, producing diversity in viewpoints, objects, tables, and lighting.The dataset is intended to grow through contributions from research labs.
- Action representation: All trajectories use end-effector position and rotation deltas plus a gripper-joint dimension, creating a unified action space across robot arms.End-effector parameterization extends naturally to arms with different degrees of freedom and avoids requiring camera calibration.
- Data collection: Initial trajectories are collected with randomly sampled actions from hand-engineered distributions, commonly combining a diagonal Gaussian with an automatic grasping primitive.The collection process is designed to provide self-supervised interaction data.
- Environmental diversity: Dataset variation spans robot arms, grippers, arenas, camera configurations, lab settings, backgrounds, lighting, and camera exposure, while the project uses RGB video.Additional sensor modalities are possible in principle, but are outside this project’s chosen setup.
- Use and contribution: Users can filter RoboNet by attributes such as robot or gripper, and the project provides code, collection scripts, storage conventions, and an upload platform.Uploaded trajectories receive manual quality checks against dataset standards.
5 Robot-Agnostic Visual Control: Model Training and Experiments
This section evaluates whether RoboNet-trained visual control models generalize across viewpoints, backgrounds, and unseen robots, using zero-shot tests and few-shot adaptation. Diverse pre-training improves viewpoint and environment transfer, while fine-tuning enables strong performance on new robots despite limitations from under-fitting and overly broad pre-training.
- Experimental scope: RoboNet experiments test zero-shot generalization to novel viewpoints and backgrounds, plus few-shot adaptation to entirely new robotic platforms.The study also examines how data breadth and quantity affect generalization and whether predictive models transfer across robots and domains.
- Zero-shot viewpoint and background generalization: Models trained on 90 viewpoints achieve lower final distance to the goal than single-view models on both seen and novel viewpoints.The evaluation compares five object-relocation tasks across two camera views.
- Zero-shot viewpoint and background generalization: 14.4 ± 2 cm average final distance demonstrates comparable performance when the multi-view model is tested in a substantially different environment.The result comes from a pushing task in a new setting and suggests effective generalization to novel surroundings.
- Few-shot adaptation to new robots: Fine-tuning RoboNet-pre-trained models on about 300–400 target-robot trajectories substantially improves performance over training from scratch on all three test robots.For Kuka and Franka, models fine-tuned with 400 samples outperform counterparts trained on all RoboNet data from the respective robot.
- Few-shot adaptation to new robots: Pre-training on Sawyer data performs significantly better than pre-training on all RoboNet data for Baxter adaptation.The authors hypothesize that the broader model under-fits, so more diversity can sometimes hurt performance relative to a visually similar subset.
- Model capacity experiments: Increasing model size from 200M to 500M parameters reduces held-out per-pixel error from 0.104 ± 0.057 to 0.0847 ± 0.045, indicating under-fitting.The authors report under-fitting in RoboNet video prediction models and suggest higher-capacity models may improve performance.
- Inverse-model reaching: A RoboNet-trained inverse model performs visual reaching on Sawyer and Franka, including a held-out viewpoint, but performs poorly on complex object-manipulation tasks.Its comparatively greedy action selection is identified as the reason for weaker performance on more complex tasks.
6 Discussion
RoboNet combines large-scale, diverse robotic experience with learning methods to study generalization across robots and environments. Results support rapid adaptation to unseen hardware, while model capacity, task complexity, video prediction quality, and random-policy data collection remain limitations.
- RoboNet combines experience from 7 robots, multiple environments and backgrounds, over 100 camera viewpoints, and 4 geographic locations.
- The experiments evaluate visual foresight and inverse models across varying viewpoints, grippers, and robots.
- Fine-tuning RoboNet-pretrained models can quickly acquire vision-based skills on unseen robot hardware.
- Video-prediction models with ≤75M parameters heavily underfit RoboNet, while even 500M-parameter models show underfitting.
- The evaluated tasks are relatively simple, low-fidelity manipulations such as pushing and pick-and-place, limiting immediate practical use.
- Predetermined random policies make data collection scalable but limit more complex and nuanced interactions.
- Adaptation to new domains is possible with a few hundred trajectories, suggesting that collecting variety across domains may matter more than quantity.
A.2 Sampling-based Planning
Visual foresight plans manipulation by predicting how actions transform images and designated-pixel distributions, then optimizing action sequences with sampling-based model-predictive control.
- Sampling-based Planning: Designated-pixel distributions undergo the same predicted transformations as images, producing predicted pixel locations.
- Sampling-based Planning: Planning cost is the time-averaged expected distance between the predicted pixel distribution and the goal pixel position.
- Sampling-based Planning: MPPI searches for action sequences by sampling candidates and updating their mean with exponential weighting.
- Sampling-based Planning: The planner applies only the first action before replanning, producing an MPC-like control loop.
B Data Collection Details
RoboNet standardizes most robot actions around Cartesian end-effector control while using randomized exploration, including a grasping primitive to increase grasp events.
- Most robots use Cartesian end-effector actions with restricted rotation and a gripper joint.The state contains gripper XYZ position, yaw, and gripper joint angle; per-robot safety bounds constrain operation.
- Data collection samples actions from diagonal Gaussians or a grasp-biased distribution, with hand-tuned robot- and dimension-specific variances.
- A hard-coded grasping primitive closes the gripper near a z-level threshold and opens it when the lifted arm is not carrying an object.
- Google R3 uses random pushing motions, while Fetch explores randomly only in the x and y dimensions.
C Database Implementation Details
The RoboNet database stores trajectories as filterable entities and provides Python-based access to HDF5 data with MP4-encoded videos.
- Each trajectory is stored separately with filterable attributes, and new trajectory attributes can be added easily.
- Users can access and structure the data through a Pandas Python API.
- Data uses HDF5 storage, while videos use MP4 encoding for efficiency.
D Description of Benchmarking Tasks
The control benchmarks evaluate object relocation on Kuka, Franka, and Sawyer robots using fixed starts and goals, with success judged after measuring remaining distance.
- Object relocation tasks move objects from fixed initial positions toward fixed goal positions marked on a table.
- Success is determined by human judges after the remaining distance to the goal is measured using a tape.
- The benchmarking setups cover the Kuka, Franka, and Sawyer robot platforms.
E Experimental evaluation of adaptation to unseen gripper
The unseen-gripper evaluation equips Sawyer with a visually and physically different Robotiq gripper and compares zero-shot, new-data-only, Sawyer-pretrained, and RoboNet-pretrained adaptation.
- 300 trajectories were collected with a Robotiq 2-finger gripper that differed substantially from the Weiss Robotics gripper used in other Sawyer trajectories.
- Four models were compared, including zero-shot RoboNet training, new-data-only training, Sawyer-only pretraining, and full RoboNet pretraining before fine-tuning.
- The Sawyer-only pretrained model performed best after fine-tuning and achieved performance comparable to in-domain generalization.
- The more diverse RoboNet-pretrained model performed worse, likely because limited capacity caused underfitting.