PaperlayerView exact revision on arXiv

Source-linked AI summary

ZeroBot: Learning from Scratch in Minutes with Generative Real2Sim

Ivan Kapelyukh, Xiaohan Zhang, Stephen James, Laura Herlant, Edward Johns

arXiv:2609.34010v1cs.ROcs.CV

TL;DR

ZeroBot addresses learning novel-object manipulation without demonstrations, policy pre-training, or prior object models. It builds a mesh from a single view and trains a policy in simulation using a geometry-aware contact action space. Real-robot experiments report successful transfer across manipulation tasks and learning from scratch in minutes.

  • Problem

    Learning novel-object manipulation to a goal pose without demonstrations, policy pre-training, or known object models remains a challenging setting, while real-world RL can require human supervision for safety and resetting.

    Source: §I · p. 1

  • Method

    ZeroBot generates a complete object mesh from a single view and uses it in simulation for massively parallel RL with an action space that selects high-value contact states.

    Source: §I · p. 2, B. Building a Simulation Using… · p. 3

  • Results

    Real-robot experiments showed successful sim2real transfer, outperforming baselines using partial meshes or nearest-neighbor retrieval, while learning manipulation from scratch in minutes.

    Source: C. Real2Sim2Real Evaluation Results · pp. 5–6, §VI · p. 8

  • Takeaways & Limitations

    The results demonstrate that image-to-3D models can support rapid, autonomous robot learning through real2sim, despite imperfect generated meshes.

    Source: §VI · p. 8

    ZeroBot inherits image-to-3D precision limits and assumes a static background, constant simulated mass, and known articulation parameters.

    Source: §V · p. 7

Abstract

from arXiv · show

We present ZeroBot, a real2sim framework for learning a robot manipulation task from scratch in minutes under challenging conditions: zero human demonstrations, zero policy pre-training, and zero known object models. Given only a single view of an object and a goal pose for that object, ZeroBot uses image-to-3D generative models to obtain a complete object mesh, which is used in simulation for large-scale parallel reinforcement learning. To accelerate training, we introduce an action space which leverages the generated geometry and learned value function to sample states involving robot-object contact. When evaluated on real-world tasks including grasping, pushing, articulated object interaction, and multi-stage manipulation, ZeroBot achieves an 87% success rate with an average training time of 119 seconds. These results show the value of using image-to-3D models in a real2sim framework for rapid, autonomous robot learning.

I. INTRODUCTION

ZeroBot targets rapid learning of novel-object manipulation without demonstrations, policy pre-training, or prior object models. It builds a scene-specific simulation from a single view and uses a geometry-aware action space to learn and deploy a policy.

I. INTRODUCTION

  • ZeroBot addresses novel-object manipulation to a goal pose with zero demonstrations, policy pre-training, and known object models.The target is learning from scratch in minutes across tasks including grasping, pushing, rolling, and multi-stage interaction.Source: §I · p. 1
  • Real-world RL can require human supervision for safety and environment resetting, while simulation scales with available compute.Building transferable simulations of deployment scenes remains challenging.Source: §I · p. 1
  • Image-to-3D generative models let ZeroBot build a complete object mesh from one view, avoiding known-model assumptions and manual scanning.The approach addresses scanning's time and human-effort costs and its inability to reconstruct occluded surfaces.Source: §I · p. 1
  • ZeroBot uses generated geometry to select high-value robot-object contact states, accelerating policy learning and supporting deployment.The learned value function selects a contact location; motion into contact is followed by execution of policy actions.Source: §I · pp. 1–2
  • The framework combines single-view mesh generation, simulation learning, and zero-shot real-robot deployment without demonstrations or policy pre-training.Its contributions also include evaluations of generated meshes against retrieval, partial meshes, and scanned geometry.Source: Figure 1 · p. 1, §I · p. 2

II. RELATED WORK

Prior work uses simulation, visual reconstruction, and image-to-3D models for robotics, but often relies on scene datasets, manual scans, offline asset generation, demonstrations, or pre-trained policies. ZeroBot instead generates an object mesh from one view and learns manipulation from scratch.

II. RELATED WORK

  • Prior efficient action spaces target generalization to novel objects and may take days to train, whereas ZeroBot tailors its geometry-based action space to learning from scratch in minutes.Simulation also allows learning across many robots in parallel.Source: §II · p. 2
  • Visual real2sim approaches often require manual scans or multiple cameras; ZeroBot generates complete object geometry from a single view and uses mesh-based tracking with a state-based deployment policy.The tracking uses Foundation Pose to mitigate the visual sim2real gap.Source: §II · p. 2
  • Image-to-3D models have supported grasp sampling and offline simulation assets, but prior approaches include pre-trained grasp networks, optimization-heavy generation, or human demonstrations and annotations.ZeroBot generates a mesh in seconds and learns manipulation autonomously from scratch in minutes.Source: §II · p. 2

A. Problem Setting and Assumptions

The task is to move an object from its initial pose to a user-provided goal using only an RGB-D view, without demonstrations, policy pre-training, or known object models. ZeroBot builds a scaled mesh-based simulation and learns a state-based policy for deployment.

A. Problem Setting and Assumptions

  • The task provides an RGB-D image and a goal pose relative to the object's initial pose, with no demonstrations, policy pre-training, or known object models.The setting primarily covers single rigid objects, with extensions to articulated objects; it assumes a clear upright, occlusion-free object view and a static background.Source: A. Problem Setting and Assumptions · p. 2

B. Building a Simulation Using Generative Priors

  • ZeroBot generates an object mesh from one view, aligns it with depth to recover metric scale, and uses the mesh to build a simulation for goal-pose-conditioned reinforcement learning.The image-to-3D pipeline uses InstantMesh and batched ICP; its mesh is also used for tracking and planning at deployment.Source: B. Building a Simulation Using… · p. 3, Figure 2 · p. 3
  • A state-based policy uses Foundation Pose and the generated mesh to estimate object pose during simulation setup and real-robot execution.The authors state that a pixels-to-actions policy is infeasible when training from scratch must take minutes.Source: B. Building a Simulation Using… · p. 3

C. Simulation Training and Execution with Contact Actions

ZeroBot trains a policy and value function in parallel simulation, then uses contact states on the generated mesh to accelerate learning. The learned value function guides contact selection during training and real-robot deployment.

Simulation training overview

  • The policy and value function are trained in Isaac Gym with PPO and a dense goal-flow reward, using the real object pose as the simulation start state.Initial poses are randomized when testing generalization; that is not the paper’s focus.Source: C. Simulation Training and Execution… · p. 3

Motivation for action space

  • Sampling contact states avoids spending simulator time on free-space motion, so the robot can start interacting and collecting reward sooner.The learned value function scores candidate contact states against the current scene state.Source: C. Simulation Training and Execution… · pp. 3–4

Action space algorithm

  • The action space samples gripper poses from generated-mesh surface points, removes physically invalid contacts, and scores the remaining states with the learned value function.Contact types vary by experiment; gripper opening is used for grasping and closing for pushing.Source: C. Simulation Training and Execution… · p. 4
  • During training, contact states are selected with a softmax-based probability, while execution selects the maximum-value state and moves there with motion planning.Value-function warmup begins with mostly uniform contact sampling to reduce vulnerability to poor initialization.Source: C. Simulation Training and Execution… · p. 4
  • After each contact, the policy executes 6D relative end-effector actions with residuals added to a down-scaled goal-flow vector; multi-stage tasks repeat contact selection.The robot samples new contacts after its policy actions at the current contact.Source: C. Simulation Training and Execution… · p. 4

Real-robot execution

  • At deployment, ZeroBot reuses the learned value function to select a high-value sampled contact with a collision-free inverse-kinematics solution, then plans and executes the policy.The motion plan avoids collisions with the object and environment.Source: C. Simulation Training and Execution… · p. 4

IV. EXPERIMENTAL EVALUATION

The evaluation asks whether ZeroBot can learn manipulation from scratch in minutes and transfer across object models, viewpoints, tasks, and initial poses. It also examines design choices and extensions to articulated objects.

A. Research Questions

  • The evaluation asks whether manipulation can be learned in minutes without demonstrations or policy pre-training, and compares complete generated meshes with partial meshes and mesh retrieval.It also asks whether the framework’s performance changes on challenging viewpoints and novel initial poses.Source: A. Research Questions · pp. 4–5
  • The experiments examine performance when task-critical geometry is barely visible or unobserved, and which design decisions contribute most to accelerating training..Source: A. Research Questions · p. 5
  • The evaluation also asks whether the framework extends to articulated objects and how mesh generation compares with a ground-truth scanned mesh..Source: A. Research Questions · p. 5

B. Experiment Setup

The real-world evaluation uses six tasks on a Franka arm, spanning pushing, grasping, multi-stage contact, novel poses, and occluded geometry. The setup compares generated meshes with partial-mesh and nearest-neighbor baselines.

Hardware

  • The evaluation uses a 7-DoF Franka Research 3, wrist RGB-D input, external RGB-D tracking, and two A6000 GPUs, with 256 parallel simulation agents for training.Foundation Pose tracks the object during execution; the wrist camera also reconstructs the background.Source: B. Experiment Setup · p. 5

Evaluation tasks

  • MultiStageBook requires nudging before a 6-DoF grasp, while PushUpRamp tests precise can pushing or rolling and SlideBowl requires pushing from its unseen far side.MultiStageBook uses the same policy and value function across two contact actions.Source: B. Experiment Setup · p. 5
  • NovelPoses tests grasping from randomized and novel poses; PullFromShelf requires collision-aware bottle extraction, and GraspHandle tests reconstruction of a partly occluded handle.NovelPoses randomizes training position within a 20x20 cm area and z-axis orientation.Source: B. Experiment Setup · p. 5

Task success

  • Task success is determined by a goal region, and merely grasping the book does not count as success in MultiStageBook..Source: B. Experiment Setup · p. 5

Baselines

  • ZB-NoPrior uses a partial point-cloud mesh without a 3D prior, while ACDC-NN retrieves a database mesh; all three methods use the same RL training framework.ACDC is a real2sim framework; ZeroBot generates a mesh rather than retrieving one.Source: B. Experiment Setup · p. 5

C. Real2Sim2Real Evaluation Results

ZeroBot reports higher real-robot success rates than the evaluated baselines, with results linking complete generated meshes to transfer and performance on challenging geometry. The evaluation also reports learning useful tasks in minutes without demonstrations or policy pre-training.

Real2Sim2Real evaluation results

  • ZeroBot outperforms the partial-mesh ZB-NoPrior baseline on all tasks, with incomplete geometry associated with simulation difficulty and real-world grasp collisions.The PushUpRamp partial mesh makes simulation success difficult; in GraspHandle and MultiStageBook, grasps that succeed in simulation can collide with the real object.Source: C. Real2Sim2Real Evaluation Results · p. 5
  • Generated meshes achieve higher success rates than nearest-neighbor retrieval on less common objects and higher-precision tasks such as PushUpRamp.ACDC-NN performs reasonably on common objects such as bottles.Source: C. Real2Sim2Real Evaluation Results · p. 5
  • SlideBowl and GraspHandle results show transfer when task-critical geometry is unobserved, including a bowl’s far side and a partly occluded tumbler handle..Source: C. Real2Sim2Real Evaluation Results · pp. 5–6
  • ZeroBot achieves higher success rates than ZB-NoPrior and ACDC-NN across the six real-robot tasks.Figure 5 compares 10 end-to-end repeats per method and task, for 180 runs total.Source: Figure 5 · p. 6
  • ZeroBot learns useful manipulation tasks in minutes without demonstrations or policy pre-training, within the stated assumptions of the problem setting.Figure 6 plots cumulative real-world successes against training time; training ends at a simulation success threshold or a 400-second limit.Source: C. Real2Sim2Real Evaluation Results · p. 6, Figure 6 · p. 6

D. Training Speed Comparison

ZeroBot’s geometry-informed contact action space substantially speeds learning on SlideBowl, outperforming the compared alternatives without auxiliary rewards. The comparison also shows that residual actions explore slightly faster early than direct actions.

D. Training Speed Comparison

  • ZeroBot learns much faster than the baselines, and both ZeroBot variants significantly outperform them.The comparison focuses on SlideBowl, where all baselines make some task progress.Source: D. Training Speed Comparison · pp. 6–7, Figure 8 · p. 7
  • NoContAct, using only a Cartesian policy, barely learns after 30 minutes; adding contact reward or using open-loop trajectory optimization also plateaus.ZeroBot does not need auxiliary rewards and outputs a closed-loop policy.Source: D. Training Speed Comparison · p. 6
  • Residual actions explore slightly faster early than direct actions, while geometry-grounded contact sampling accelerates exploration relative to CRL-Octree.The plotted maximum success rate is averaged over 10 runs, with a 95% confidence interval; checkpoints use 256 parallel environments and 10 evaluation episodes.Source: D. Training Speed Comparison · pp. 6–7, Figure 8 · p. 7

E. Extending our Framework to Articulated Objects

ZeroBot extends to opening a microwave door using a single-view generated mesh, with articulation parameters assumed known. Generated geometry reaches 80% real-robot success, while oracle geometry performs better and takes longer to create.

E. Extending our Framework to Articulated Objects

  • The task is grasping and opening a microwave door; articulation parameters such as joint limits and door segmentation are assumed given.InstantMesh generates the whole object mesh from one view, and Foundation Pose tracks the door pose for the policy.Source: E. Extending our Framework to… · p. 7
  • Single-view mesh generation achieves 80% real-robot success, although ground-truth geometry yields higher sim2real success rates.Mesh creation took 23 seconds for generation versus 443 seconds for the oracle mesh, including manual scanning.Source: E. Extending our Framework to… · p. 7, Figure 9 · p. 7
  • The oracle comparison uses a Polycam mesh made from 164 manual-scan views; both methods use ZeroBot’s contact action space for policy learning.RialTo’s original demonstration requirement and training time exceeding two days make a system-level comparison infeasible in this setting.Source: E. Extending our Framework to… · p. 7

V. LIMITATIONS AND FUTURE WORK

ZeroBot’s current scope is constrained by image-to-3D precision, static-background assumptions, constant simulated mass, and known articulation parameters. The conclusion reports successful real-world transfer from generated meshes and faster training with the proposed action space.

V. LIMITATIONS AND FUTURE WORK

  • Current image-to-3D precision, a static-background assumption, constant simulated mass, and known articulation parameters constrain the framework’s scope.Inferring physics and articulation properties automatically, and extending to more dynamic multi-object scenes, are left to future work.Source: §V · p. 7

VI. CONCLUSION

  • ZeroBot generates object meshes and learns manipulation policies in simulation without human demonstrations or policy pre-training.The conclusion describes useful tasks and successful sim2real transfer from generated meshes.Source: §VI · pp. 7–8
  • Real-robot experiments report that generated meshes outperform partial-mesh and nearest-neighbor baselines, while ablations show the action space accelerates training.The paper presents image-to-3D models as a promising real2sim technique for rapid, autonomous robot learning.Source: §VI · p. 8
Loading 2609.34010v1…