Source-linked AI summary
MolmoB0T: Large-Scale Simulation Enables Zero-Shot Manipulation
Abhay Deshpande, Maya Guru, Rose Hendrix, Snehal Jauhri, Ainaz Eftekhar, Rohun Tripathi, Max Argus, Jordi Salvador, Haoquan Fang, Matthew Wallingford, Wilbert Pumacay, Yejin Kim, Quinn Pfeifer, Ying-Chun Lee, Piper Wolters, Omar Rayyan, Mingtong Zhang, Jiafei Duan, Karen Farley, Winson Han, Eli Vanderbilt, Dieter Fox, Ali Farhadi, Georgia Chalvatzaki, Dhruv Shah, Ranjay Krishna
TL;DR
Robotics foundation-model training is constrained by limited disclosure of real-world data and by the assumption that simulation alone cannot support robust manipulation. The paper introduces an open, large-scale procedural simulation pipeline and trains three policy classes, achieving zero-shot transfer to unseen real-world objects and environments, including 79.2% success on tabletop pick-and-place across 4 settings versus 39.2% for π0.5.
Problem
Robotics foundation-model training relies heavily on real-world data, while the field widely assumes simulation alone is insufficient for robust manipulation without real-world adaptation.
Method
The paper introduces MolmoBot-Engine and MolmoBot-Data, then trains MolmoBot, MolmoBot-Pi0, and MolmoBot-SPOC on procedurally generated simulation trajectories.
Results
79.2% success rate is achieved by MolmoBot on real-world tabletop pick-and-place across 4 settings versus 39.2% for π0.5, with zero-shot transfer to unseen objects and environments.
Takeaways & Limitations
The results support zero-shot sim-to-real transfer for both static and mobile manipulation using large, diverse simulated training data.
Takeaways & Limitations
MolmoBot-Engine is constrained to assets that can currently be accurately simulated, leaving contact-rich, deformable, fluid, and granular tasks open.
Abstract
from arXiv · showhide
A prevailing view in robot learning is that simulation alone is not enough; effective sim-to-real transfer is widely believed to require at least some real-world data collection or task-specific fine-tuning to bridge the gap between simulated and physical environments. We challenge that assumption. With sufficiently large-scale and diverse simulated synthetic training data, we show that zero-shot transfer to the real world is not only possible, but effective for both static and mobile manipulation. We introduce MolmoBot-Engine, a fully open-source pipeline for procedural data generation across robots, tasks, and diverse simulated environments in MolmoSpaces. With it, we release MolmoBot-Data, a dataset of 1.8 million expert trajectories for articulated object manipulation and pick-and-place tasks. We train three policy classes: MolmoBot, a Molmo2-based multi-frame vision-language model with a flow-matching action head; MolmoBot-Pi0, which replicates the $π_0$ architecture to enable direct comparison; and MolmoBot-SPOC, a lightweight policy suitable for edge deployment and amenable to RL fine-tuning. We evaluate on two robotic platforms: the Franka FR3 for tabletop manipulation tasks and the Rainbow Robotics RB-Y1 mobile manipulator for door opening, drawer manipulation, cabinet interaction, and mobile pick-and-place. Without any real-world fine-tuning, our policies achieve zero-shot transfer to unseen objects and environments. On tabletop pick-and-place, MolmoBot achieves a success rate of 79.2% in real world evaluations across 4 settings, outperforming $π_{0.5}$ at 39.2%. Our results demonstrate that procedural environment generation combined with diverse articulated assets can produce robust manipulation policies that generalize broadly to the real world. Technical website: https://allenai.github.io/MolmoBot
1 Introduction
The paper challenges the assumption that simulation alone cannot produce robust real-world manipulation policies, arguing that aggressively scaled, diverse simulation can enable zero-shot transfer. It introduces an open data-generation pipeline and evaluates three policy classes across tabletop and mobile manipulation.
- Motivation: Large-scale simulation is presented as a sufficient substrate for zero-shot transfer to real-world manipulation without real-world fine-tuning.The approach spans diverse environments, embodiments, articulated assets, and tasks, without photorealistic rendering or explicit domain adaptation.
- Data and pipeline: MolmoBot-Engine procedurally generates manipulation data across robots, tasks, and simulated environments, producing 1.7 million expert trajectories in MolmoBot-Data.The dataset covers articulated object manipulation and pick-and-place, using environments from MolmoSpaces.
- Policies: The study trains MolmoBot, an augmented Molmo2-based policy, alongside MolmoBot-Pi0 and the lightweight MolmoBot-SPOC.MolmoBot uses a flow-matching action head coupled to intermediate vision-language representations, while MolmoBot-Pi0 enables controlled architectural comparison.
- Evaluation: The policies are evaluated on the Franka FR3 for tabletop tasks and the Rainbow Robotics RB-Y1 for mobile manipulation.The mobile platform covers door opening, drawer manipulation, cabinet interaction, and mobile pick-and-place.
- Results: 79.2% success rate is achieved by MolmoBot on real-world tabletop pick-and-place across 4 settings, compared with 39.2% for π0.5.The policies transfer zero-shot to unseen real-world objects and environments and outperform π0.5 in the reported evaluations.
- Results: 46.7% success rate is achieved by MolmoBot-Pi0, compared with 39.2% for π0.5 under the same architecture.This controlled comparison indicates that MolmoBot-Data supports strong performance independently of the replicated architecture.
- Implication: The authors argue that access to large, diverse, open simulation pipelines may be more important than an irreducible sim-to-real barrier for general-purpose manipulation.They open-source the components needed to generate and use the simulation data.
2 Related Work
Prior work connects generalist robot policies to imitation learning, vision-language-action models, and large-scale datasets, while simulation increasingly addresses the cost of real-world data collection. MolmoBot extends this direction with an open, scalable simulation pipeline spanning robots, tasks, environments, and articulated manipulation.
- Imitation learning: Imitation learning has progressed from behavior cloning to hierarchical methods, temporal abstractions, and generative policies for manipulation.Recent diffusion-based approaches have demonstrated strong performance on manipulation benchmarks.
- Vision-language-action models: Vision-language-action models unify language understanding, perception, and control, with larger models and multitask datasets supporting broad task coverage.RT-1, RT-2, and π0 exemplify this progression, including flow-matching representations for continuous action generation.
- Datasets and simulation: Large-scale real-world datasets provide heterogeneous demonstrations across tasks and embodiments but are costly and logistically difficult to collect.DROID is cited as an example of a large manipulation-demonstration dataset for contemporary VLA models.
- Datasets and simulation: Simulation-based research has explored synthetic demonstrations for grasping, manipulation, planning, navigation, and humanoid control.The cited efforts include GraspVLA, InternVLA variants, and related synthetic-data approaches.
- MolmoBot: MolmoBot contributes an open pipeline and generated expert-trajectory dataset that combine procedural scenes with rigid and articulated assets for real-world transfer without demonstrations.The stated scope includes different robots, tasks, and diverse environments.
- Articulated and mobile manipulation: Articulated and mobile manipulation remain difficult because of complex contact dynamics, partial observability, and the need to coordinate navigation, perception, and manipulation.Large-scale manipulation systems have more often focused on fixed-base tabletop settings.
- Articulated and mobile manipulation: The study addresses mobile manipulation with a single evaluation spanning tabletop and mobile platforms and multiple tasks rather than only task-specific articulated-object solutions.The mobile tasks include mobile pick-and-place and articulated interactions such as door opening.
3 MolmoBot-Engine: A scalable manipulation data engine
MolmoBot-Engine is an open-source procedural pipeline that generates diverse manipulation data by varying environments, objects, robot configurations, and execution conditions. It scales to millions of episodes across tabletop and mobile platforms while using simulation-based generation and replanning.
- 3.1 MolmoSpaces environments and assets: MolmoBot-Engine procedurally generates scalable manipulation training data in diverse MolmoSpaces environments.It builds on procedurally generated indoor scenes and supports task-specific object placement and environment adaptation.
- 3.1 MolmoSpaces environments and assets: The engine samples task-relevant objects, randomizes visual and physical parameters, and replans until it finds a successful trajectory.This combines scene adaptation, domain randomization, and iterative planning during data generation.
- 3.2 Robot configuration: Robot episodes vary initial joint configurations, action commands, object poses, camera conditions, and gripper execution to improve robustness.Action noise is proportional to commanded displacement, and object poses are randomized within collision and reachability constraints.
- 3.2 Robot configuration: Data generation supports both fixed-base Franka FR3 tabletop manipulation and mobile RB-Y1 manipulation.The RB-Y1 includes a holonomic base, torso, head, and two arms, while the Franka is a 7-DoF arm with a parallel-jaw gripper.
- 3.5 Expert planners: 1.7M episodes comprise 295M frames across more than 11k target assets, more than 9k receptacle assets, and more than 94k environments.Using 100 NVIDIA A100 GPUs, generation reached approximately 660 successful episodes per GPU-hour and exceeded 88 hours of robot experience per wall-clock hour.
4 Models and training
The paper trains three policy classes on synthetic manipulation data, combining vision, language, proprioception, and optional spatial point conditioning with action-prediction architectures. Training includes MolmoBot, an architecture-matched MolmoBot-Pi0 comparison, and lightweight MolmoBot-SPOC variants for static and mobile manipulation.
- Three policy classes—MolmoBot, MolmoBot-Pi0, and MolmoBot-SPOC—are trained on MolmoBot-Data for architecture comparison and manipulation control.
- MolmoBot: MolmoBot combines RGB observations, language instructions, robot state, and optional 2D point coordinates with a Molmo2-based flow-matching action head.The action head uses cross-attention to vision-language features and iteratively denoises action chunks conditioned on a timestep embedding.
- MolmoBot: MolmoBot processes up to three frames per camera view, while multi-frame variants use current observations and frames sampled eight steps earlier.The F = 3 variant represents the current state and states approximately 0.5 and 1 second earlier.
- Action representation: MolmoBot predicts continuous joint-space targets for actuated joints and, on RB-Y1, additionally predicts linear and angular mobile-base velocity commands.The policies use either absolute joint positions or joint-position deltas; mobile manipulation uses delta policies.
- Alternative policies: MolmoBot-Pi0 exactly follows the π0 architecture, enabling controlled comparison, while MolmoBot-SPOC uses a lightweight transformer with parallel action decoding.SPOC encodes visual, language, proprioceptive, and optional point inputs, then predicts D × T action tokens in one forward pass with T = 16.
5 Experiments
The experiments evaluate zero-shot sim-to-real transfer across tabletop and mobile manipulation, simulation benchmarks, and training-data ablations. Policies trained only on simulation outperform strong baselines in several real-world and simulated settings, while performance depends on data scale and composition.
- Real-world evaluations: MolmoBot policies transfer zero-shot to unseen real-world objects and environments across Franka FR3 and RB-Y1 tasks without real-world post-training.
- Simulation evaluations: 64.1% average simulation success for MolmoBot (F=2), compared with 10.1% for π0.5 zero-shot, while MolmoBot-Pi0 reaches 41.8%.
- Simulation evaluations: 93.5% success on Pick MSProc for MolmoBot (F=2) versus 48.0% for π0.5-Finetune, with larger gaps on more variable tasks.
- Data ablations: Pick performance improves with more demonstrations, whereas increasing unique training houses has little downstream effect when total data is fixed.
6 Conclusion
The paper argues that large-scale procedural simulation can enable effective zero-shot transfer for static and mobile manipulation. It releases the training pipeline and identifies simulation fidelity as the main boundary for extending the approach.
- MolmoBot-Engine generates MolmoBot-Data for three policy classes evaluated on Franka FR3 and Rainbow Robotics RB-Y1 platforms.
- 79.2% real-world success on tabletop pick-and-place demonstrates zero-shot transfer to unseen objects and environments without real-world fine-tuning.
- The approach is constrained by assets that can currently be simulated accurately, especially for contact-rich, deformable, fluid, and granular-dynamics tasks.
A Additional Data Details
RB-Y1 motion planning uses a GPU-accelerated, collision-aware trajectory generator over a 23-DOF whole-body model.
- cuRobo models the RB-Y1 as a 23-DOF chain combining a holonomic base, torso, and two 7-DOF arms.
- Given a target end-effector pose, cuRobo solves inverse kinematics with 64 seeds before computing a collision-free trajectory.
A.2 Referral Expressions
Referral expressions are sampled to distinguish the target object from distractors within the task context. The procedure uses CLIP-based similarity contrasts and a temperature-controlled softmax.
- The task context is the set of relevant scene objects, and expressions maximize CLIP similarity to the target relative to distractors.
- Candidate expressions are filtered by CLIP thresholds and sampled with a softmax temperature of 2 ⋅10^-2 over similarity contrasts.
A.2.1 Train-Time Task Prompt Randomization
Training varies task instructions and object references to expose the policy to diverse language formulations for the same manipulation tasks.
- Task prompts are procedurally randomized during training to increase instruction diversity.
- The policy samples among task-prompt templates with varying wording and phrasing for each training example.
- Object referral expressions are sampled from multiple valid alternatives, with a bias toward shorter expressions before insertion into prompts.
B.1 DROID Evaluation Environments
DROID evaluations span four real-world environments that vary objects, receptacles, support surfaces, clutter, distractors, and location. These settings test pick-and-place across diverse scene configurations.
- Across the environments, the evaluation covers objects, receptacles, clutter, distractors, and support surfaces under varied spatial configurations.
- Kitchen: The kitchen contains 4 objects and 2 receptacles, with easy and hard placements across 8 tasks.
- Workroom: The workroom uses 5 objects and 2 receptacles for 10 tasks, including configurations with an additional spork distractor.
- Bedroom: The bedroom evaluates 4 objects across 2 receptacles, including cluttered banana placements and a bed instead of a table.
- Office: The office combines 8 objects, 7 receptacles, varying clutter and distractors, and evaluation at a different institution and geographical location.
B.2 Simulation Evaluation
The simulation evaluation uses fixed control and horizon settings, multiple renderers and camera configurations, and several task-dataset combinations. StereoVLA receives a matched front-on camera setup and filtered episodes.
- The policy runs with a 66ms policy interval and horizons of 303 steps / 20 seconds for pick tasks and 606 steps / 40 seconds for pick-and-place tasks.
- The filament renderer uses environment illumination of 12000 candela by default.
- StereoVLA is evaluated with front-on cameras and filtered episodes containing visible target objects, yielding 92 Pick Classic episodes and 91 Pick episodes.
- The benchmark includes Thor, Objaverse, MuJoCo, Filament, DROID, DROID-Light, and random-camera configurations across pick and pick-and-place tasks.
C Zero-Shot Evaluation on External Simulation Benchmarks
The paper adapts SIMPLER and LIBERO to the DROID platform for zero-shot evaluation without in-domain data or task-specific fine-tuning. Results show strong baseline ranking but low and benchmark-sensitive absolute performance.
- Benchmark adaptation: SIMPLER and LIBERO are reimplemented with the DROID platform to support unified, reproducible zero-shot evaluation.
- Benchmark adaptation: Evaluation uses no in-domain data or task-specific fine-tuning on the adapted benchmarks.
- SIMPLER: 45.8% is achieved on SIMPLER’s “put carrot on plate,” while all models fail on the remaining tasks.
- LIBERO: 36.6% vs. 3.8% average is reported for MolmoBot-Img versus baselines in LIBERO’s no-distractor setting.
- Interpretation: The adapted benchmarks preserve baseline rank ordering but have structural limitations for assessing generalist policies.