PaperlayerView exact revision on arXiv

Source-linked AI summary

Robot Learning with Super-Linear Scaling

Marcel Torne, Arhan Jain, Jiayi Yuan, Vidaaranya Macha, Lars Ankile, Anthony Simeonov, Pulkit Agrawal, Abhishek Gupta

arXiv:2412.01770v3cs.ROcs.AIcs.LG

TL;DR

Robot-learning data is difficult to obtain at scale because real-world demonstrations require substantial human effort, while simulation still demands scene and behavior creation. CASHER crowdsources 3D digital twins, collects behavioral data in simulation with demonstration-bootstrapped RL, and uses generalist-policy generalization to reduce later human demonstrations. CASHER reaches 62% zero-shot success after training on 56 environments and improves few-shot fine-tuning performance by 36% and 30% over imitation learning baselines.

  • Problem

    Robot-learning data is difficult to obtain at scale because real-world demonstrations require substantial human effort, while simulation still demands scene and behavior creation.

    Source: §I · p. 1

  • Method

    CASHER crowdsources 3D digital twins, collects behavioral data in simulation with demonstration-bootstrapped RL, and uses generalist-policy generalization to reduce later human demonstrations.

    Source: §III · p. 3, B. Amortized Data Collection · p. 4

  • Results

    CASHER reaches 62% zero-shot success after training on 56 environments and improves few-shot fine-tuning performance by 36% and 30% over imitation learning baselines.

    Source: A. Zero-Shot Scaling Laws Analysis · p. 5, D. Analysis of CASHER on More Tasks · p. 6

  • Takeaways & Limitations

    CASHER supports real-world transfer while shifting continual data collection from human demonstrations toward model-generated simulated demonstrations.

    Source: §I · p. 1, §V · p. 7

    The method shifts burden to compute, whose time still exceeds real-world demonstration collection, and simulation cannot yet accurately represent some objects such as liquids and deformable objects.

    Source: §V · p. 7

Abstract

from arXiv · show

Scaling robot learning requires data collection pipelines that scale favorably with human effort. In this work, we propose Crowdsourcing and Amortizing Human Effort for Real-to-Sim-to-Real(CASHER), a pipeline for scaling up data collection and learning in simulation where the performance scales superlinearly with human effort. The key idea is to crowdsource digital twins of real-world scenes using 3D reconstruction and collect large-scale data in simulation, rather than the real-world. Data collection in simulation is initially driven by RL, bootstrapped with human demonstrations. As the training of a generalist policy progresses across environments, its generalization capabilities can be used to replace human effort with model generated demonstrations. This results in a pipeline where behavioral data is collected in simulation with continually reducing human effort. We show that CASHER demonstrates zero-shot and few-shot scaling laws on three real-world tasks across diverse scenarios. We show that CASHER enables fine-tuning of pre-trained policies to a target scenario using a video scan without any additional human effort. See our project website: https://casher-robot-learning.github.io/CASHER/

I. INTRODUCTION

Robot learning needs abundant, high-quality data, but real-world teleoperation scales only linearly with human effort and simulation still requires costly scene, task, and behavior creation. CASHER addresses this by crowd-sourcing real-world digital twins and amortizing behavior collection through model generalization.

I. INTRODUCTION

  • CASHER targets superlinear data scaling with sublinear human effort for robot learning.The system addresses the scarcity and cost of high-quality robotic training data.Source: §I · p. 1
  • Teleoperation remains limited because data collection cost scales linearly with human involvement and remains far below vision-and-language data scales.Real-world collection requires expert supervision and physical robots.Source: §I · p. 1, §II · pp. 1–2
  • Simulation reduces physical collection costs but still requires substantial human effort for realistic scenes, rewards, and successful behaviors.Random procedural environments are unlikely to match natural scene distributions or produce successful behaviors reliably.Source: §I · p. 1
  • CASHER shifts content creation to non-experts using 3D reconstruction and reduces behavior-collection effort as generalist policies generalize across environments.This creates a data flywheel in which model-generated data increasingly replaces human demonstrations.Source: §I · p. 1, §II · p. 2
  • The paper contributes real-to-sim-to-real continual collection, scanned deployment fine-tuning without additional demonstrations, and zero-shot and few-shot scaling analyses.The related-work context contrasts this approach with prior real-to-sim systems that often fail to generalize across scenarios.Source: §I · p. 1, §II · pp. 2–3

III. AMORTIZED DATA SCALING FOR LEARNING GENERALIST POLICIES THROUGH REAL-TO-SIM-TO-REAL

CASHER builds a large-scale real-to-sim-to-real pipeline by crowdsourcing digital twins of real scenes and collecting behavioral data in simulation. This supports realistic, decentralized data collection across many environments without requiring physical robot setups.

III. AMORTIZED DATA SCALING FOR LEARNING GENERALIST POLICIES THROUGH REAL-TO-SIM-TO-REAL

  • CASHER combines accessible digital-twin generation, multi-environment model learning, and scan-based fine-tuning to scale robotic manipulation data collection.Its pipeline uses model generalization to amortize human demonstrations as more environments are encountered.Source: §III · p. 3

A. Real-to-Sim Scene Synthesis

  • Crowdsourced scans provide simulated scenes with real-world geometry, visuals, and physics while enabling safe, decentralized, asynchronous collection without physical robot setups.The approach is intended to scale across diverse environments and large numbers of non-experts.Source: A. Real-to-Sim Scene Synthesis · p. 3

B. Amortized Data Collection

CASHER bootstraps a generalist policy with human demonstrations and reinforcement learning, then uses its cross-environment generalization to generate simulated trajectories and reduce later human effort. The resulting trajectories train a final visuomotor policy through supervised learning.

B. Amortized Data Collection

  • Human demonstrations initialize the first environment batch, whose demonstration-bootstrapped RL trajectories are distilled into a perception-based generalist policy.The system partitions environments into batches of size K.Source: B. Amortized Data Collection · p. 3
  • The generalist policy is deployed in later simulated environments to collect paired visual, action, and state trajectories instead of querying humans.These model-generated trajectories are used to expand the training dataset.Source: B. Amortized Data Collection · pp. 3–4
  • Demonstration-bootstrapped RL trains state-based policies, with human fallback on environments where the generated policy performs below the success threshold.The resulting policies generate trajectories for both successful and fallback environment sets before visuomotor distillation.Source: B. Amortized Data Collection · p. 4
  • The pipeline repeats over environment batches, using the accumulated dataset to train a visuomotor generalist policy by supervised learning.Trajectory generation and policy training are continually repeated as new batches arrive.Source: B. Amortized Data Collection · p. 4
  • A generalist policy trained across environments increasingly replaces human demonstrations, making data collection more human-efficient and eventually self-sustaining.Perfect generalization is unnecessary; sufficient generalization can bootstrap demonstration-augmented learning.Source: B. Amortized Data Collection · p. 4

C. Fine-tuning of Generalist Policies on Deployment

CASHER fine-tunes a pretrained generalist policy for a scanned target environment by generating successful simulated trajectories without human intervention. The resulting state policy is distilled back into a visuomotor policy for real-world deployment.

C. Fine-tuning of Generalist Policies on Deployment

  • A scanned target environment lets the pretrained generalist policy generate successful trajectories that bootstrap state-based RL without external human intervention.The method operates on a digital twin of the deployment environment.Source: C. Fine-tuning of Generalist Policies… · p. 4
  • Figure 3 evaluates zero-shot scaling on bowl, cup, and mug placement, simulation-to-real performance correspondence, and robustness across broader environments.The figure’s three panels cover scaling, sim-real correlation, and broader-environment evaluation.Source: Figure 3 · p. 5
  • The state-based policy is distilled into a fine-tuned visuomotor policy that retains generalist capabilities while improving success in the target environment.Real-world transfer uses successful rollouts generated by the state policy.Source: C. Fine-tuning of Generalist Policies… · p. 5

IV. EXPERIMENTAL EVALUATION

CASHER evaluates zero-shot scaling and continual data collection across real-world tasks, showing improved performance with more environments while reducing human demonstrations and compute demands.

IV. EXPERIMENTAL EVALUATION

  • The experiments evaluate scaling laws, amortized human data, fine-tuning, cross-task transfer, and multi-object extrapolation using real-world tasks and held-out scenes.The setup includes bowl, mug, and cup placement plus box placement, with policies evaluated across institutions and unseen scenes.Source: §IV · p. 5

A. Zero-Shot Scaling Laws Analysis

  • 62% zero-shot success was reached after training on 56 environments, with real-world performance improving as the number of training environments increased.Across eight additional kitchens, success increased from 16% to 60% when training environments rose from 9 to 56.Source: A. Zero-Shot Scaling Laws Analysis · p. 5
  • The evaluation includes few-shot box placement, cabinet opening, and multi-object sink placement as complementary tests of generalization.Figure 5 covers these three evaluation settings.Source: Figure 5 · p. 6

B. Amortized Human Data Needed Through Continual Data Collection

  • Continual data collection reduced the human demonstrations needed across sequential environment batches while achieving higher performance than human-only collection.Compute requirements also tended to decrease as scaling progressed, because stronger policies required fewer trials to obtain successful rollouts.Source: Figure 4 · p. 5, B. Amortized Human Data Needed Through… · p. 6
  • CASHER’s continual policy outperformed a policy trained solely from human demonstrations, possibly because autonomous behaviors remain closer to previously learned behaviors.The authors hypothesize that additional variability in human demonstrations can make learning harder.Source: B. Amortized Human Data Needed Through… · p. 6

C. Fine-Tuning of Generalist Policies

CASHER improves weak generalist policies through scanned or few-shot fine-tuning and transfers across additional manipulation tasks, while scaling human effort sublinearly but shifting burden toward compute.

C. Fine-Tuning of Generalist Policies

  • 55% average performance improvement was achieved through scanned deployment fine-tuning without additional human demonstrations.The evaluated policies had been trained on 36 environments and initially performed at ≤20% on two target scenes.Source: C. Fine-Tuning of Generalist Policies · p. 6
  • 54% average success-rate improvement was achieved by few-shot fine-tuning with 10 demonstrations per environment.The selected base policies performed at ≤20% before fine-tuning.Source: C. Fine-Tuning of Generalist Policies · p. 6

D. Analysis of CASHER on More Tasks

  • Few-shot fine-tuning improved performance by 36% for box placement and 30% for cabinet opening relative to an imitation-learning baseline with 0% success.Performance continued increasing with more training environments and had not reached saturation.Source: D. Analysis of CASHER on More Tasks · p. 6

V. CONCLUSION

  • CASHER transfers learned policies to the real world and supports both zero-shot and fine-tuning results while scaling across environments with decreasing human effort.The conclusion frames the system as a crowdsourced-simulation approach for scaling robot learning.Source: §V · p. 7
  • The method demonstrates superlinear data scaling with respect to human demonstrations but shifts a substantial burden to compute and cannot yet accurately simulate some objects.The stated examples include liquids and deformable objects, and simulation compute still exceeds real-world demonstration time.Source: §V · p. 7

APPENDIX

The appendix documents CASHER’s amortized data-collection algorithm and provides method, task, implementation, evaluation, hardware, crowdsourcing, and compute details.

APPENDIX

  • The appendix covers amortized data collection, real-to-sim transfer, autonomous fine-tuning data collection, and teacher-student distillation.These topics are listed as method details for CASHER.Source: APPENDIX · p. 10
  • Task, implementation, evaluation, hardware, crowdsourcing, and compute-resource specifications are provided in dedicated appendix sections.The detailed evaluation includes robustness experiments with disturbances and distractors.Source: APPENDIX · p. 10

A. Amortized Data Collection

  • Algorithm 1 initializes a vision-based generalist policy, samples crowdsourced digital twins, rolls out the policy, filters successful rollouts, and fine-tunes with reinforcement learning.The algorithm processes batches of environments and aggregates successful simulated trajectories.Source: A. Amortized Data Collection · p. 10
  • Failed environments receive human demonstrations, after which PPO fine-tuning and teacher-student distillation update the generalist policy.Human collection is therefore targeted to environments where the current policy fails.Source: A. Amortized Data Collection · p. 10

B. Real-to-Sim Transfer of Scenes

CASHER crowdsources real-world scene scans into articulated digital twins, then uses simulation and progressively more autonomous data collection to train generalist policies across environments.

B. Real-to-Sim Transfer of Scenes

  • Crowdsourced mobile scans provide diverse digital twins with natural clutter, layouts, colors, lighting, and positional variation for generalist-policy training.Scenes are reconstructed from videos or image sequences and can be scanned in under five minutes using commercial mobile software.Source: B. Real-to-Sim Transfer of Scenes · p. 10
  • The GUI transfers scanned scenes into Isaac Sim, supports scene articulation, and adds task objects and sites for sinks or cabinets.The interface accommodates static and highly articulated environments.Source: B. Real-to-Sim Transfer of Scenes · p. 10, Figure 7 · p. 10

C. Autonomous Data Collection

  • Each articulated environment is bootstrapped with 10 keyboard-teleoperated demonstrations before reinforcement-learning fine-tuning.The demonstrations use a discretized action space with 14 end-effector actions.Source: B. Real-to-Sim Transfer of Scenes · p. 10

1) Multi-task bootstrapped RL fine-tuning::

  • Training 10 scenes in parallel across 2048 environments yields an effective 10x speedup while producing a policy that works across all 10 scenes.The experiments use PPO fine-tuning with sparse rewards and privileged state information.Source: 1) Multi-task bootstrapped RL… · pp. 10–11

2) Teacher-student distillation:

  • Teacher-student distillation converts state-based policies into colored-point-cloud policies using 1000 trajectories per scene, reaching 56 environments and 6,720,000 state-action pairs.The dataset combines camera-rendered and synthetically generated point clouds, while retaining prior-batch data to avoid catastrophic forgetting.Source: 2) Teacher-student distillation · p. 11

D. Fine-tuning

CASHER adapts generalist policies through few-shot supervised fine-tuning while defining task actions and sparse success rewards for sink and cabinet manipulation.

2) Few-shot Supervised Fine-tuning::

  • Few-shot supervised fine-tuning updates only the final fully connected layers of a pretrained generalist policy using human visuomotor demonstrations.The visual-processing layers remain frozen, allowing performance improvements from a small number of real-world demonstrations.Source: 2) Few-shot Supervised Fine-tuning · p. 11

VIII. TASK DETAILS

  • The task action space contains 14 discretized end-effector actions: six positional, six rotational, and two gripper actions.Position changes are ±0.03 m, orientation changes are ±0.02 radians, and the gripper actions open or close.Source: §VIII · p. 11
  • Sparse rewards define success for placing objects into sinks or cabinets and opening cabinets using geometric, uprightness, gripper, or joint-angle conditions.The sink condition requires distance below 0.25 together with an upright object and open gripper; cabinet opening requires joint position above 0.65.Source: §VIII · pp. 11–12
  • The point-cloud student policy combines a 128-dimensional point-cloud embedding with 9 robot-state dimensions and outputs probabilities over 14 actions.Its encoder uses local point features, a 3D U-Net, and separate max- and average-pooling outputs.Source: §VIII · p. 12

A. Simulation details

CASHER trains privileged state-based policies in Isaac Sim and compares them with large-scale real-world-data baselines, which do not solve the sink task zero-shot or after few-shot tuning.

A. Simulation details

  • CASHER uses Isaac Sim for physics-accurate, photorealistic simulation and employs high-fidelity SDF collision meshes for scanned scenes.Other objects retain default convex-decomposition collision meshes and physical parameters.Source: A. Simulation details · p. 12

1) State-based policy:

  • The simulated state-based policy receives privileged state and outputs a probability distribution over 14 discrete end-effector actions.This state-based training is performed in simulation before transferring behavior to visuomotor policies.Source: 1) State-based policy · p. 12

X. BASELINE COMPARISONS WITH LARGE SCALE REAL WORLD DATA COLLECTION

  • Neither OpenVLA nor Octo successfully completes the zero-shot object-to-sink task, although OpenVLA performs qualitatively better than Octo.Both models were pretrained on more than 800k Open X-Embodiment real-world trajectories and received RGB images with language instructions.Source: §X · p. 12
  • Fine-tuning Octo with 10 demonstrations still produces zero success, despite qualitatively better behavior than its zero-shot result.The comparison is reported as evidence that substantially more data is needed for these tasks across varied scenarios.Source: §X · p. 12

XI. DETAILED EVALUATION RESULTS

The detailed evaluation tests CASHER policies under multi-object, dim-lighting, messy-kitchen, and human-disturbance conditions across real-world hardware setups.

A. Evaluation on Multi-Object Scenes

  • Multi-object extrapolation succeeds 80% for placing two objects and 10% for placing all three, despite training only on single-object scenes.The evaluation allows six trials to place three kitchen objects sequentially into the sink.Source: A. Evaluation on Multi-Object Scenes · p. 13

B. Evaluation on Scenes Involving Disturbance and Distractors

  • Dim lighting reduces sink-task success to 30% when the policy was trained only with sufficient lighting.The scenario uses minimal lighting and is evaluated as a robustness condition.Source: B. Evaluation on Scenes Involving… · p. 13
  • Messy-kitchen scenes with dirty dishes and tableware in the sink also achieve 30% task success.Training used a clean sink, so the evaluation introduces realistic clutter absent during training.Source: B. Evaluation on Scenes Involving… · p. 13
  • Human disturbance, implemented by pushing the object during evaluation, yields 50% successful sink-task trials.This test directly probes robustness to changes in object position during execution.Source: B. Evaluation on Scenes Involving… · p. 13

XII. HARDWARE SETUP

  • Real-world experiments use two Franka Panda arms on mobile tables at two institutions with different kitchen settings and paired depth cameras.Each setup uses two calibrated Intel RealSense D435i cameras to create aligned point-cloud maps.Source: §XII · p. 13

XIII. CROWDSOURCING

CASHER’s crowdsourcing experiments use kitchen scans and simulation-based policy evaluation across sink, cabinet, and disturbance scenarios. The section also reports the compute and time required for teleoperated demonstrations, RL fine-tuning, trajectory collection, and distillation.

XIII. CROWDSOURCING

  • 29 sink scenes and 26 cabinet scenes were used, including 22 and 18 scenes collected through crowdsourcing, respectively.Source: §XIII · pp. 13, 15
  • Zero-shot sink evaluation covers bowls, mugs, and cups placed on either side of the sink.Source: Table VI · p. 14
  • Few-shot evaluations compare imitation-learning baselines with supervised fine-tuning for sink and cabinet placement, including cabinet grasping and placement success.Source: Table VII · p. 14, Table VIII · p. 14
  • The multi-object evaluation rolls out three objects for six episodes and measures success by the average episodes needed to place specified numbers of objects into the sink.Source: Table IX · p. 14
  • Robustness experiments cover multi-object, dim-lighting, messy-kitchen, and human-disturbance scenarios, with success rates reported in Table X.Source: Table X · p. 14, Figure 10 · p. 15

XIV. COMPUTE RESOURCES

  • Creating an environment and collecting 10 teleoperated demonstrations takes about 1 hour, RL fine-tuning about 20 hours, and teacher-student distillation 5 days.Source: §XIV · p. 15
Loading 2412.01770v3…