Source-linked AI summary
BridgeData V2: A Dataset for Robot Learning at Scale
Homer Walke, Kevin Black, Abraham Lee, Moo Jin Kim, Max Du, Chongyi Zheng, Tony Zhao, Philippe Hansen-Estruch, Quan Vuong, Andre He, Vivek Myers, Kuan Fang, Chelsea Finn, Sergey Levine
TL;DR
Robot learning needs large, reusable datasets that support generalization beyond the environments and institutions where data are collected. BridgeData V2 addresses this need with diverse robotic manipulation data and evaluates multiple goal- and language-conditioned learning methods. The evaluation reports generalization across tasks, environments, and institutions, with performance improving as model capacity, dataset size, and skill diversity increase.
Problem
Robot-learning datasets need broad task and environment coverage so researchers can reuse them beyond the institution and scenes where they were collected.
Method
BridgeData V2 combines varied robotic manipulation trajectories with goal-image and language conditioning, and the paper evaluates six offline learning methods.
Results
BridgeData V2 supports policies that generalize across tasks, environments, and institutions, while performance improves with model size and dataset size and diversity.
Takeaways & Limitations
BridgeData V2 is presented as a useful publicly shared resource for scalable robot-learning research across multiple learning algorithms and laboratories.
Takeaways & Limitations
The dataset mainly covers low-precision behaviors and does not address forceful, highly dynamic, or low-tolerance industrial manipulation.
Abstract
from arXiv · showhide
We introduce BridgeData V2, a large and diverse dataset of robotic manipulation behaviors designed to facilitate research on scalable robot learning. BridgeData V2 contains 60,096 trajectories collected across 24 environments on a publicly available low-cost robot. BridgeData V2 provides extensive task and environment variability, leading to skills that can generalize across environments, domains, and institutions, making the dataset a useful resource for a broad range of researchers. Additionally, the dataset is compatible with a wide variety of open-vocabulary, multi-task learning methods conditioned on goal images or natural language instructions. In our experiments, we train 6 state-of-the-art imitation learning and offline reinforcement learning methods on our dataset, and find that they succeed on a suite of tasks requiring varying amounts of generalization. We also demonstrate that the performance of these methods improves with more data and higher capacity models, and that training on a greater variety of skills leads to improved generalization. By publicly sharing BridgeData V2 and our pre-trained models, we aim to accelerate research in scalable robot learning methods. Project page at https://rail-berkeley.github.io/bridgedata
1 Introduction
BridgeData V2 addresses the challenge of building reusable robot-learning datasets by combining broad task and environment coverage with support for diverse learning methods. Its evaluation finds that the dataset supports generalization and that performance improves with model size, dataset size, and diversity.
- Motivation: Large-scale robot learning requires datasets whose policies generalize across diverse real-world tasks and environments.The proposed recipe combines large demonstration datasets with expressive policies trained by behavioral cloning or offline reinforcement learning.
- Motivation: Existing datasets often cover only a few tasks and environments, limiting reuse by researchers who cannot exactly reproduce the original scene.The paper therefore emphasizes coverage broad enough to support new laboratory settings.
- BridgeData V2: 50,365 demonstrations cover 13 skills across 24 environments, expanding the original Bridge Dataset by more than 7 times.The dataset also includes 9,731 trajectories from a heavily randomized pick-and-place policy.
- BridgeData V2: BridgeData V2 supports open-vocabulary task specification through either goal images or language instructions and is evaluated with six state-of-the-art learning methods.The dataset was collected in scenes containing many possible skills, supporting multi-task learning under different method assumptions.
- Findings: The study finds that BridgeData V2 enables generalization across tasks, environments, and institutions, while performance improves with model size and dataset size and diversity.These findings are presented as evidence that the dataset is useful for broad offline robot-learning research.
2 Related Work
Prior robot-learning datasets commonly focus on single behaviors, tasks, or environments, which limits cross-environment reuse. BridgeData V2 instead targets broad task and domain coverage, public accessibility, and compatibility with multiple learning approaches.
- Prior datasets: Many prior datasets focus on a single task, domain, or behavior, while multi-task datasets often remain difficult for other researchers to reuse.Single-environment collection can require researchers to reproduce the original scene closely.
- BridgeData V2: BridgeData V2 is designed to support cross-environment generalization in researchers’ own laboratories through coverage of many tasks and domains.This design reduces dependence on exact replication of the data-collection environment.
- BridgeData V2: Unlike some large, diverse multi-task datasets, BridgeData V2 uses a publicly available robot to facilitate academic research.The related-work discussion contrasts this with prior datasets built on proprietary robots.
- BridgeData V2: BridgeData V2 combines demonstrations, autonomously collected data, and language descriptions to accommodate methods with different assumptions.The paper specifically motivates support for multi-task imitation learning, offline reinforcement learning, and language-conditioned imitation learning.
- Evaluation: The paper evaluates a wide variety of methods because prior robotic manipulation datasets were typically tested with only one or a few methods.This evaluation is intended to demonstrate compatibility with multiple learning approaches.
3 BridgeData V2
BridgeData V2 is a publicly available, multi-task dataset collected with an inexpensive robot setup across varied environments, camera configurations, objects, and workspace positions. Its composition spans foundational manipulation, environment interaction, and more complex skills, with trajectories labeled after collection.
- Design goals: The dataset aims to support generalization to novel tasks, environments, and institutions with goal-image or natural-language task conditioning.These goals guide both data collection and dataset composition.
- System setup: The system uses a WidowX 250 arm, one fixed RGBD camera, two randomized RGB cameras, and one wrist camera at 640x480 resolution and 5 Hz.Demonstrations are collected through VR-controller teleoperation.
- System setup: The publicly available robot setup costs approximately $4,000 and can be assembled with a turnaround time under two weeks.The setup consists of publicly available parts.
- Data collection: Data collection varies tasks, environments, objects, camera poses, and workspace positions while placing multiple feasible tasks in each environment.This protocol encourages policies to use task specifications rather than infer tasks solely from observations.
- Data collection: A heavily randomized scripted policy adds suboptimal pick-and-place trajectories that can be used or excluded during training.The autonomous data enables faster collection across many objects and may support methods that benefit from suboptimal data, such as offline reinforcement learning.
- Data annotation: Because trajectories are not task-labeled during collection, crowdsourced annotators describe each trajectory afterward, emphasizing moved objects’ final locations.Post-hoc descriptions provide task information for the dataset.
- Dataset composition: A skill groups trajectories with similar motions despite different objects or arrangements, whereas a task groups trajectories matching a similar language instruction.Thus, the same pick-and-place motion can represent different tasks when applied to different objects.
- Dataset composition: The dataset includes 13 skills ranging from foundational pick-and-place, pushing, and reorienting to folding, stacking, tool use, and furniture manipulation.Foundational skills comprise a large portion because they apply broadly and may transfer to more complex behaviors.
4 Offline Learning Methods
The evaluation covers goal-conditioned imitation and reinforcement learning methods alongside language-conditioned behavioral cloning, including a large transformer policy. These methods differ in conditioning, action modeling, and model architecture while using image observations.
- Method families: The study evaluates six state-of-the-art offline learning methods with goal-image or language conditioning.This selection reflects BridgeData V2’s intended support for open-vocabulary task specification and varied method assumptions.
- Goal-conditioned methods: GCBC encodes the observation and goal with ResNet-34 before predicting a robot action through fully connected layers.It serves as a standard goal-conditioned behavioral-cloning baseline.
- Goal-conditioned methods: D-GCBC replaces the baseline policy representation with a diffusion process to model multimodal action distributions.Training uses the DDPM objective.
- Goal-conditioned methods: ACT uses transformers and a conditional VAE objective to generate multimodal action sequences rather than single actions.The method applies action chunking to imitation learning.
- Goal-conditioned methods: CRL formulates goal-conditioned reinforcement learning as representation learning with log-linear goal-conditioned value functions.Its formulation is analogous to contrastive learning.
- Language-conditioned methods: LCBC encodes natural-language instructions with MUSE and conditions a ResNet-34 image encoder using FiLM before producing actions.The resulting representation feeds a fully connected policy network.
- Language-conditioned methods: RT-1 is a large transformer that tokenizes image and language inputs and predicts discretized low-level robot actions.It uniquely uses observation history and higher image resolution among the evaluated methods.
5 Experiments
The experiments evaluate whether BridgeData V2 supports multi-task learning, generalization to unseen objects and environments, cross-institution transfer, and scaling with model and dataset capacity. Results show non-zero success under several generalization settings, while performance improves with larger models, more data, and greater skill diversity.
- The experiments test broad multi-task learning with goal- and language-conditioned methods, generalization to new objects and environments, cross-institution transfer, and scaling effects.The evaluation is intended to assess the dataset's utility for large-scale robot learning rather than rigorously compare learning-method design choices.
- Seen tasks: Seen-task success rates are averaged over 10 trials, with comparable goal-conditioned methods and RT-1 outperforming LCBC.
- Unseen tasks: Both language- and goal-conditioned methods achieve non-zero success on tasks with unseen objects, environments, or both, demonstrating broad generalization.Language-conditioned methods particularly struggle with unseen objects because their names are not grounded in the dataset.
- Method behavior: D-GCBC sometimes produces jerky behaviors because it lacks observation histories and action chunking, indicating oscillation between modes in its action distribution.
- Cross-institution evaluation: Both goal- and language-conditioned methods achieve non-zero success at a new institution, although performance is somewhat worse in Lab 2.RT-1 shows only a small degradation, while goal-conditioned methods degrade more; the evaluations are zero-shot without new Lab 2 data.
- Scaling analysis: Higher-capacity models, larger datasets, and greater skill diversity improve performance, including on an unseen pick-and-place task.The skill-diversity comparison used approximately equal-sized datasets: 28k trajectories with 3 skills versus 27k with all 13 skills.
6 Discussion, Limitations, and Future Work
BridgeData V2 supports policies that generalize across tasks, environments, and institutions, while scaling with model size, dataset size, and skill diversity. Its main limitations are restricted task dynamics, single-institution collection, and dependence on an accessible but nonstandardized robot arm.
- BridgeData V2 enables policies from multiple learning algorithms to generalize across tasks, environments, and institutions.
- Higher-capacity models and larger training sets improve goal-conditioned behavior cloning performance.
- Policies trained on 13 skills outperform those trained on 3 skills on an unseen task despite similarly sized datasets.The comparison used 20 trials per policy.
- The dataset mainly covers low-precision behaviors and does not address forceful, dynamic, or low-tolerance industrial manipulation.Examples outside its coverage include throwing, moving heavy objects, and industrial insertion.
- BridgeData V2 was collected at one institution, and standardizing its accessible low-cost robot arm across researchers may be difficult.The authors identify broader environment coverage and multi-robot datasets as future directions.
A Data Statistics
The appendix identifies where dataset skills and sensor modalities are documented. Table 5 lists the skills, while Figure 6 breaks down sensor coverage.
- Table 5 provides the full list of skills in the dataset.The list follows the paper’s definition of skill in Section 3.3.
- Figure 6 breaks down which portions of the dataset include which sensors.
- The dataset statistics reference both skill definitions and sensor-coverage breakdowns as complementary descriptions.
B Learning Method Implementation Details
The implementation appendix presents the evaluation-task set and explains how goal-conditioned methods obtain training goals. Evaluation tasks are documented in Figure 7.
- The complete set of evaluation tasks is shown in Figure 7.
- Goal-conditioned methods receive both an observation and a goal.
- During training, each goal is selected by uniformly sampling a future observation from the same trajectory.
B.1 Goal-conditioned behavior cloning
The goal-conditioned behavior-cloning implementations encode observation-goal pairs with image encoders and predict actions, while related methods adapt diffusion or chunking choices to the setting.
- Observation and goal images are stacked channel-wise, encoded with ResNet-34, and mapped through three 256-unit layers to predict robot actions.Training uses random crops, resizing, color jitter, Adam, a 3e-4 learning rate, and 2000-step linear warmup.
- The diffusion behavior-cloning implementation conditions an action-distribution diffusion process on the observation-goal image encoding without learning a value function.It uses a DDPM-style objective.
- ACT is adapted from single-task control to goal-conditioned learning with chunk size 5 for the shorter, lower-frequency trajectories.The passage reports better performance with the reduced chunk size.
B.4 Contrastive RL
The implementation uses visual encoders and separate conditioning mechanisms for contrastive RL and language-conditioned behavioral cloning. Training also uses extended action sequences and multi-view RGB-D data collection.
- Contrastive RL: Contrastive RL processes observation and goal images separately through a shared ResNet-34 encoder and two MLP representation heads.The shared encoder is used by both the value function and policy.
- Language-Conditioned Behavioral Cloning: LCBC uses a frozen MUSE language encoder and FiLM-conditioned ResNet blocks to predict actions from language and image observations.FiLM conditioning is applied at the end of every ResNet block.
- RT-1: The RT-1 implementation increases transformer sequence length from 6 to 15 to accommodate longer episodes.Actions are scaled to [-1, 1] and tokenized with a vocabulary of 256.
- Data Collection: Data collection combines a fixed RGB-D camera, randomized alternative views, and a wrist camera mounted on the gripper.The robot was teleoperated with a Meta Quest 2 VR headset.
D Comparison to the Original Bridge Data
BridgeData V2 is evaluated against training on the original Bridge Data combined with PTR data. The additional BridgeData V2 data significantly improves GCBC performance on unseen pick-and-place tasks.
- Dataset Comparison: GCBC performance is compared between original Bridge Data plus PTR data and the full BridgeData V2 dataset.The comparison isolates the effect of training with the additional data released in BridgeData V2.
- Unseen-Task Performance: BridgeData V2 training significantly improves performance on unseen pick-and-place tasks.The result is reported for GCBC evaluated on unseen tasks.
- Evaluation Focus: The comparison tests whether the expanded dataset provides an advantage beyond the earlier Bridge Data and PTR combination.The reported evaluation focuses specifically on unseen pick-and-place tasks.