PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Behavior Cloning with Open Data, Training, and Evaluation

Arthur Allshire, Himanshu Gaurav Singh, Ritvik Singh, Adam Rashid, Hongsuk Choi, David McAllister, Justin Yu, Yiyuan Chen, Huang Huang, Pieter Abbeel, Xi Chen, Rocky Duan, Phillip Isola, Jitendra Malik, Fred Shentu, Guanya Shi, Philipp Wu, Angjoo Kanazawa

arXiv:2606.27375v1cs.RO

TL;DR

Behavior-cloning research faces costly, difficult-to-reproduce data collection, training, and real-world evaluation. ABC releases an open manipulation stack and studies training and architecture choices across real and simulated settings. Simulation evaluations correlate strongly with real-world performance, offering a way to assess policies before costly physical evaluation.

  • Problem

    Behavior-cloning progress is difficult to reproduce and evaluate because it requires robot hardware, demonstrations, training infrastructure, and costly real-world testing.

    Source: §1 · p. 2

  • Method

    ABC releases an open behavior-cloning stack with a large bimanual dataset, hardware and training infrastructure, simulation data, and comparisons of DiT and VLA training and architecture choices.

    Source: §1 · p. 2, §2 · p. 3, §3.2 · p. 6

  • Results

    Simulation task progress correlates with real-world performance at r = 0.91, while strict success correlates at r = 0.85.

    Source: §4 · p. 9

  • Takeaways & Limitations

    The simulation stack can help researchers debug policy design choices before expensive real-world evaluations, including for researchers without hardware.

    Source: §4 · p. 9

Abstract

from arXiv · show

We introduce ABC, a fully open-source stack for manipulation with behavior cloning. At its core is ABC-130K: the largest open-source teleoperation dataset to date, featuring 3,500 hours of data spanning over 130K episodes across 195 diverse tasks. Furthermore, we open-source our accessible hardware setup, training infrastructure, and simulation pipeline. We also release 400 hours of sim-teleop data and provide a co-training recipe that produces correlated simulation and real-world evaluation, offering a reliable proxy for ablating model-design and training decisions before costly real-world evaluation. We explore various training recipes and compare common architectural choices for Diffusion Transformers (DiT) and Vision-Language-Action (VLA) models, grounding our findings in real-world evaluations. The resulting policies successfully execute dexterous tasks such as box folding and extracting credit cards from wallets. By providing a reproducible toolkit, we aim to place researchers on an equal footing, establishing the necessary foundation to learn the ABCs of Behavior Cloning together as a community.

1 Introduction

Behavior-cloning research is difficult to reproduce and evaluate because it requires substantial hardware, data, training infrastructure, and costly real-world testing. ABC addresses these challenges with an open end-to-end stack and uses it to explore model and training choices for dexterous manipulation.

1 Introduction

  • ABC releases an open end-to-end behavior-cloning stack centered on ABC-130K, with hardware, training and deployment code, and a simulation pipeline with 400 hours of simulation teleoperation data.The stack also demonstrates sim-to-real evaluation correlation for iterating on design choices without physical robots.Source: §1 · p. 2
  • The released dataset contains 3,500 hours of real-world interaction across more than 130,000 episodes and 195 tasks, including dexterous manipulation.Tasks include folding boxes and retrieving credit cards from wallets.Source: §1 · p. 2
  • The authors explore training recipes and architectural choices for diffusion transformers and vision-language-action models using offline metrics, simulation, and real-world evaluations.They evaluate vision encoders, conditioning schemes, and pretraining recipes.Source: §1 · p. 2
  • The resulting baseline policies execute long-horizon and dexterous tasks, including inserting six AirPods in a row, unlocking a lockbox with a key, folding a cardboard box, and packing a student bag.Model weights and real-world evaluation rollouts are publicly released.Source: §1 · p. 2
  • The stack targets gaps in existing open datasets, which may lack scale, rely on bespoke expensive hardware, or omit training infrastructure and evaluation suites.The paper frames its goal as accelerating exploration of manipulation training paradigms that remain unsettled.Source: §1 · p. 2

2 ABC-130K Dataset

ABC-130K is a large-scale bimanual manipulation dataset designed to combine complex, diverse tasks with an inexpensive robot. It includes task-level variation and collection metadata, with subgoal annotations in a substantial subset.

2 ABC-130K Dataset

  • The dataset targets large-scale dexterous data for low-cost bimanual systems, whereas cited alternatives include single-arm datasets, heterogeneous collections, and data on a $30,000 robot.The related-work comparison identifies MolmoAct2 as the closest comparison, with 720 hours on the YAM platform.Source: §2 · p. 3
  • ABC-130K contains 134,806 episodes across 195 tasks, totaling 3,553 hours of bimanual manipulation data.Its tasks span primitives including pick-and-place, folding, handover, insertion, tool use, and assembly.Source: §2 · p. 3
  • The 195 tasks are organized into seven primitive categories, with objects and initial configurations varied within each task.Figure 3 presents random top-camera frame samples illustrating task, background, and object diversity.Source: §2 · p. 3, Figure 3 · p. 3
  • ABC-130K includes teleoperator IDs and collection timestamps, while a 1,552-hour subset adds subgoal annotations for contiguous subtrajectories.The paper shows how these annotations and metadata can be used for policy conditioning.Source: §2 · p. 3

3 ABC-Models

The section compares ABC-DiT and ABC-VLA policies, varying model architecture, conditioning, and training scale. The studies use real-world evaluations alongside offline metrics to assess design choices.

3.1 ABC-DiT

  • Replacing AdaLN visual conditioning with pooled cross-attention improved DiT task progress, and replacing CLIP with DINOv3 improved it further.The authors use DINOv3 with pooled cross-attention for ABC-DiT, a 2B-parameter model.Source: Table 1 · p. 4, §3.1 · p. 5

3.2 ABC-VLA

  • 61.4% aggregate progress for AdaLN exceeded 32.6% for FAST co-training and 11.7% for vanilla cross-attention in the VLA connector comparison.The comparison fixed the backbone, action head, and training recipe; the authors suggest diffusion gradients can work with VLM features given a suitable interface.Source: Table 1 · p. 4, §3.2 · p. 6

3.3 Scaling compute across batch size and training steps

  • At batch sizes of 1,536 and 4,608, DiT outperformed VLA, while at 9,216 VLA made a larger performance jump and could outperform DiT, which remained more compute-efficient.Figure 5 compares real-world task success and progress at 100K and 200K training steps.Source: §3.3 · p. 6, Figure 5 · p. 7

3.4 Offline Metrics for Policy Evaluation

  • Training loss and validation action error were negatively correlated with real-world performance, but validation loss was not significantly correlated.Validation action error had the strongest negative correlation and is meaningful only when the number of diffusion steps is held fixed; these offline metrics help iterate but cannot reliably predict task success or reveal failure modes.Source: §3.4 · p. 7, Figure 8 · p. 8

4 ABC-Sim

ABC-Sim provides a simulation suite and teleoperation data for evaluating manipulation policies. Across 12 checkpoints, simulation evaluations strongly correlate with real-world performance.

4 ABC-Sim

  • ABC-Sim addresses offline metrics’ inability to measure task success or expose failure modes by providing environments with task success rates and watchable rollouts.The simulation environments are intended to let researchers iterate without robot hardware.Source: §4 · p. 8
  • The release includes ten MuJoCo tasks, with a Blender pipeline for generating higher-fidelity, path-traced renderings.The tasks range from turning mugs right-side up to pouring beads and in-hand object handover.Source: §4 · pp. 8–9, Figure 10 · p. 9
  • The stack includes over 400 hours of VR teleoperation data and the tools to collect similar simulation demonstrations.Operators teleoperate YAM arms with GELLO leader arms; the simulation runs at 240Hz and sends visual observations to the headset at 60Hz.Source: §4 · p. 9
  • Simulation task progress correlates with real-world progress at r = 0.91, while strict success rates correlate at r = 0.85 across 12 checkpoints.The authors propose using the simulation stack to debug policy design before costly real-world evaluations or for research without hardware.Source: Figure 9 · p. 8, §4 · p. 9

5 Real-World Capabilities

Policies trained on ABC-130K demonstrate real-world capabilities, including dexterous tasks that benefit from richer dataset labels and targeted DAgger recovery data. The evaluations cover pretraining, downstream finetuning, and box folding.

5 Real-World Capabilities

  • Per-episode operator identities and subtask annotations can steer policy behavior at inference time, enabling finer-grained control beyond adding trajectories alone.The examples include selecting operator style, trading off motion smoothness against visual responsiveness, and resolving temporal ambiguity.Source: §5 · pp. 9–10

5.1 Pretraining Capabilities

  • During 200,000-step pretraining on ABC-130K, simulation performance tends to increase for both ABC-DiT and ABC-VLA as training continues.The authors evaluate each of the three real-world tasks 50 times.Source: §5.1 · p. 10, Figure 12 · p. 10, Figure 11 · p. 10

5.2 Downstream single-task Finetuning

  • Across four dexterous tasks, finetuning from checkpoints pretrained on more data yields stronger single-task policies than finetuning from less data or training from scratch.The comparison includes ABC-130K’s 3,500-hour corpus and a 7,000-hour internal corpus.Source: §5.2 · p. 10, Figure 13 · p. 11

5.3 DAgger

  • For box folding, an 80:10:10 mixture of previous-round data, current interventions, and other current-round episode data performs better than training on whole episodes or interventions alone.Intervention-only training can exploit spurious correlations between caged source data and uncaged intervention data.Source: Figure 14 · p. 11, Figure 15 · p. 11, §5.3 · p. 12

6 Related Work

The related work spans single-arm and bimanual datasets as well as large-scale behavior-cloning systems. ABC-130K is positioned as a large, low-cost bimanual dataset with dexterous tasks, while the accompanying open infrastructure addresses barriers to comparing policy designs.

6.1 Large-scale open datasets for robotics

  • Earlier open datasets include single-arm collections and heterogeneous multi-embodiment data, while ALOHA-based platforms remain moderately expensive and newer bimanual datasets differ in scale and task coverage.BridgeData-V2 [6], DROID [4], RH20T [29], and Open X-Embodiment illustrate the earlier dataset landscape; ALOHA [30] [32] and MolmoAct2-BimanualYAM [8] represent more recent bimanual efforts.Source: §6.1 · p. 12
  • ABC-130K offers over 3,500 hours across 195 tasks on an $8,000 bimanual YAM station, including dexterous skills such as card retrieval and folding.AgiBot-World is the closest dataset in scale, with roughly 3,000 hours across roughly 200 tasks on a $30,000 platform, but emphasizes long-horizon macro-manipulation.Source: §6.1 · p. 12

6.2 Large-scale Behavior Cloning

  • Closed training recipes, proprietary datasets, and unknown compute budgets limit broader evaluation of policy-design choices in large-scale behavior cloning.The authors position their dataset and accompanying training and evaluation infrastructure as an open foundation for studying these choices.Source: §6.2 · p. 12

7 Infrastructure

The project releases systems infrastructure alongside its data and models, covering data loading, real-robot software, inference, and DAgger collection.

7 Infrastructure

  • The project provides efficient data loading for training from multiple video streams.The implementation details are in Appendix B.1.Source: §7 · p. 13
  • The release includes a lightweight, extensible framework for real-robot code and inference optimizations for ABC-DiT and ABC-VLA.The implementation details are in Appendices C and D, respectively.Source: §7 · p. 13
  • The project releases its DAgger collection setup using passive leader arms.Implementation details are in Appendix F.Source: §7 · p. 13

8 Requests for Research

The paper identifies open questions in behavior cloning and positions ABC as a reproducible ecosystem for exploring them.

8 Requests for Research

  • ABC is presented as a foundation for studying open questions in large-scale real-world robot learning.The listed questions include history conditioning, scaling laws for behavior cloning, and reinforcement-learning fine-tuning.Source: §8 · p. 13
  • History conditioning asks whether policies should use past observations rather than only the current frame.The paper describes this as conditioning on a sequence of past observations.Source: §8 · p. 13
  • Scaling-law work would examine how behavior-cloning performance changes with model and data size.Source: §8 · p. 13
  • The paper also proposes studying reinforcement-learning fine-tuning of ABC base models.Source: §8 · p. 13

A ABC-130K Task Taxonomy

ABC-130K groups its 195 manipulation tasks into seven primitive categories based on contact modes and precision requirements, with task variants consolidated in its statistics.

A ABC-130K Task Taxonomy

  • The taxonomy groups 195 tasks into seven primitive categories intended to reflect similar contact modes and precision requirements.Source: A ABC-130K Task Taxonomy · p. 18
  • Pick-and-place and fine pick-and-place cover transfers and multi-object arrangements, including millimeter-level placement and insertion.The categories comprise 67 tasks and 793 hours for pick-and-place, and 39 tasks and 736 hours for fine pick-and-place.Source: A ABC-130K Task Taxonomy · p. 18
  • Folding covers garments and other deformables, while insertion/ejection covers everyday-object fits such as pens into caps.Folding comprises 36 tasks and 883 hours; insertion/ejection comprises 19 tasks and 441 hours.Source: A ABC-130K Task Taxonomy · p. 18
  • Tool-use tasks involve an intermediate tool or mechanism, while sorting requires classifying multiple objects.The categories comprise 16 tool-use tasks and 321 hours, and 8 sorting tasks and 205 hours.Source: A ABC-130K Task Taxonomy · pp. 18–19
  • Tying and untying covers long, thin deformables and bimanual coordination; task-name variants with the same behavior count as one task in statistics.Tying/Untying comprises 10 tasks and 175 hours.Source: A ABC-130K Task Taxonomy · p. 19

B Training & Model Details

  • The training appendix documents the dataloader, model training and architecture, model sizes, and FLOP counts; models use NVIDIA H200 GPUs.Source: B Training & Model Details · p. 19

B.1 Data Loading

This appendix describes the systems and model details behind ABC’s training and deployment, including video-loading methods, model configurations, and inference optimizations.

B.1 Data Loading

  • The custom abcdl dataloader supports distributed loading of large datasets, and constant-GOP video encoding enables analytical frame indexing and lower per-decode reads.CFR frame mapping reduces mean filesystem reads from 9.75 MB to 0.14 MB per decode; the format also supports local filesystems and cloud storage.Source: §B.1 · pp. 19–21, Figure 17 · p. 20, Figure 19 · p. 20

B.2 Architecture Details

  • ABC-DiT pairs a DINOv3 vision encoder with a large DiT action model, while ABC-VLA uses a Gemma 3 VLM and a smaller diffusion action head.ABC-DiT uses 12 query tokens per image and a 32-layer DiT; ABC-VLA attention-pools VLM features for adaLN conditioning.Source: §B.2 · p. 21, Figure 20 · p. 22

B.3 Training Details

  • Both model families use learning-rate warmup, AdamW, gradient clipping, and a global batch size of 9,216.ABC-DiT trains on 12 H200 nodes and ABC-VLA on 8 H200 nodes with gradient accumulation.Source: §B.3 · p. 21

B.4 Architecture Sizes and FLOPS

  • ABC-DiT’s larger action heads reduce training loss efficiently, whereas ABC-VLA concentrates parameters and compute in its Gemma-SigLIP backbone.ABC-VLA uses eight independent diffusion draws because the VLM pass is shared across draws while only the small action head is repeated.Source: §B.4 · p. 22, Table 2 · p. 22

C Real Robot Setup

  • ABC is built around a low-cost bimanual YAM workstation and a lightweight robot codebase for collection, evaluation, and deployment.Source: C Real Robot Setup · p. 22

C.1 Hardware

  • The workstation uses two 6-DoF YAM arms and three cameras; its white enclosure narrows background diversity but isolates fine-manipulation learning from background variation.The authors report that some policies trained only in the enclosure transfer to in-the-wild settings.Source: §C.1 · pp. 22–23, Figure 22 · p. 23

C.2 Real Robot Code

  • The robot framework uses decentralized nodes with publish/subscribe communication and timing controls to operate hardware modules at predefined rates.Source: §C.2 · p. 23

D Inference Optimization

  • ABC-DiT latency falls from 63 ms to 36.3 ms, while ABC-VLA latency falls from 47.8 ms to 17.5 ms after inference optimizations.Source: D Inference Optimization · p. 25

E.1 Task description and evaluation

The appendix specifies downstream task evaluations and finetuning, a passive-leader-arm intervention method, and experiments on policy conditioning.

E.1 Task description and evaluation

  • Downstream evaluations cover credit-card removal, LEGO sorting, pen-cap insertion, and bottle-cap unscrewing, with two-minute budgets except for three minutes for card removal.The appendix provides task descriptions and an evaluation rubric.Source: §E.1 · p. 25

E.2 Finetuning recipe

  • ABC-DiT is the cheaper finetuning base except for credit-card removal, where ABC-VLA finetuning performs markedly better.Both model families are finetuned for 200,000 gradient steps, with checkpoints inspected every 50,000 steps.Source: §E.2 · p. 25

F DAgger Infrastructure

  • The DAgger setup collects interventions using conventional passive leader arms without additional hardware, mapping leader-arm pose changes to follower-arm commands.It uses forward kinematics to compute the leader pose delta and inverse kinematics to obtain follower joints.Source: F DAgger Infrastructure · pp. 25–26

G Evaluation Details

H Policy Conditioning

  • Conditioning studies examine operator identity, executed-action prefixes, and language subtask prompts on t-shirt folding.The experiments use a finetuned CLIP cross-attention policy and evaluate ten rollouts per condition on two unseen shirts.Source: H Policy Conditioning · p. 26

H.1 Operator-ID conditioning

Operator-ID and subtask prompts steer behavior at inference, while action prefixes trade responsiveness against smoothness. The section also describes how these conditioning signals are sampled during pretraining.

H.1 Operator-ID conditioning

  • Filtering to Op-0 yields mean rubric score 3.3, 2/10 completions, and 369 s, worse on every metric than the all-operator baseline (3.8, 4/10, 302 s).The filtered model was trained to 30k rather than 100k steps after longer training led to observed overfitting.Source: §H.1 · pp. 26–27
  • Marginalized inference improves over the unconditioned baseline, while Op-0 yields the highest-quality folds and completion rate and Op-1 longer, lower-quality folds.Op-0-conditioned rollouts also re-execute earlier subtasks more often, consistent with stricter stage-transition criteria in its demonstrations.Source: §H.1 · pp. 27–28

H.2 Action prefix conditioning

  • Action prefixes smooth interchunk motion but can override visual evidence after a missed grasp; shortening the prefix combats this at some smoothness cost.The illustrated prefix-conditioned generation continues toward the bin despite the missed grasp, whereas the unconditional generation attends to the visual observation.Source: §H.2 · p. 28, Figure 26 · p. 29

H.3 Subtask conditioning

  • Subtask prompts expose the current task stage, helping disambiguate visually similar intermediate and terminal states whose correct next actions differ.A planner, classifier, or human operator can set the subtask at inference.Source: §H.3 · p. 29
  • With subtask prompting, the policy avoids re-flattening an already-folded shirt, unlike the unprompted policy, which did so in 5 of 10 trials.The prompted policy still failed to finish within the time limit in one trial after a missed grasp and recovery errors.Source: §H.3 · pp. 29–30

H.4 Conditioning during pretraining

  • During pretraining, prefix lengths are sampled uniformly from 0–7; operator and subtask annotations are each used with probability 0.2, with subtask use limited to examples carrying it.Subtask metadata is used for 20% of the time it is available.Source: §H.4 · p. 30

Related papers

Loading 2606.27375v1…