Source-linked AI summary
MolmoSpaces: A Large-Scale Open Ecosystem for Robot Navigation and Manipulation
Yejin Kim, Wilbert Pumacay, Omar Rayyan, Max Argus, Winson Han, Eli VanderBilt, Jordi Salvador, Abhay Deshpande, Rose Hendrix, Snehal Jauhri, Shuo Liu, Nur Muhammad Mahi Shafiullah, Maya Guru, Ainaz Eftekhar, Karen Farley, Donovan Clay, Jiafei Duan, Arjun Guru, Piper Wolters, Alvaro Herrasti, Ying-Chun Lee, Georgia Chalvatzaki, Yuchen Cui, Ali Farhadi, Dieter Fox, Ranjay Krishna
TL;DR
Robot-learning benchmarks need broader, more scalable coverage of the long tail of real-world environments, objects, and tasks than physical evaluation alone can provide. MolmoSpaces supplies an open, simulator-agnostic ecosystem and MolmoSpaces-Bench, whose evaluations show strong sim-to-real correlation while exposing policy sensitivities and grasp-generation constraints.
Problem
Existing robot benchmarks underrepresent the diverse environments, objects, and task specifications needed to measure generalization to everyday situations.
Method
MolmoSpaces unifies large-scale indoor scenes, annotated object assets and grasps, compatible simulators, and an eight-task zero-shot benchmark for navigation and manipulation.
Results
Pearson and Spearman correlations between MolmoSpaces-Bench and 752 RoboArena pick tasks were 0.96 and 0.98, respectively, while evaluations revealed sensitivity to instruction phrasing and initial robot pose.
Takeaways & Limitations
The open ecosystem supports scalable benchmark creation, diverse training-data generation, and controlled analysis of generalist policy behavior across unseen settings.
Takeaways & Limitations
Grasps generated on isolated objects can fail during in-situ testing because surrounding geometry and gravity are initially ignored, requiring robustness-oriented pipeline updates.
Abstract
from arXiv · showhide
Deploying robots at scale demands robustness to the long tail of everyday situations. The countless variations in scene layout, object geometry, and task specifications that characterize real environments are vast and underrepresented in existing robot benchmarks. Measuring this level of generalization requires infrastructure at a scale and diversity that physical evaluation alone cannot provide. We introduce MolmoSpaces, a fully open ecosystem to support large-scale benchmarking of robot policies. MolmoSpaces consists of over 230k diverse indoor environments, ranging from handcrafted household scenes to procedurally generated multiroom houses, populated with 130k richly annotated object assets, including 48k manipulable objects with 42M stable grasps. Crucially, these environments are simulator-agnostic, supporting popular options such as MuJoCo, Isaac, and ManiSkill. The ecosystem supports the full spectrum of embodied tasks: static and mobile manipulation, navigation, and multiroom long-horizon tasks requiring coordinated perception, planning, and interaction across entire indoor environments. We also design MolmoSpaces-Bench, a benchmark suite of 8 tasks in which robots interact with our diverse scenes and richly annotated objects. Our experiments show MolmoSpaces-Bench exhibits strong sim-to-real correlation (R = 0.96, \r{ho} = 0.98), confirm newer and stronger zero-shot policies outperform earlier versions in our benchmarks, and identify key sensitivities to prompt phrasing, initial joint positions, and camera occlusion. Through MolmoSpaces and its open-source assets and tooling, we provide a foundation for scalable data generation, policy training, and benchmark creation for robot learning research.
1 Introduction
MolmoSpaces addresses the difficulty of evaluating generalist robot policies across the long tail of environments, objects, and tasks by providing a large-scale, open simulation ecosystem. Its benchmark evaluates zero-shot policies in unseen environments and reveals both sim-to-real validity and sensitivity to distribution shifts.
- Existing benchmarks provide limited signal because many manipulation evaluations focus on short-horizon skills in a single scene.
- Real-world generalization requires evaluation across a broad combinatorial distribution of environments, objects, and task specifications.
- Simulation enables systematic, reproducible evaluation across thousands of controlled scenarios but must support diversity, physical realism, articulated interaction, and long-horizon tasks.
- Over 230k indoor environments, more than 130k object models, and over 42M annotated grasps across 48k interactive objects support scalable evaluation and generalization to novel objects.
- MolmoSpaces-Bench evaluates eight base tasks zero-shot in never-before-seen environments and is open-source and extensible for dataset and benchmark creation.
- Zero-shot evaluations distinguish policies across unseen environments and objects, while minor changes in instruction phrasing or initial robot pose can significantly reduce success for some policies.
- By expanding simulated environments and making them openly accessible, MolmoSpaces supports more rigorous measurement of generalization and future diverse training-data generation.
2 Related work
Prior robotics simulation and benchmarking efforts provide scalable components but often specialize in navigation, limited task settings, or closed evaluation pipelines. MolmoSpaces combines large-scale diverse scenes and objects with cross-simulator support and controlled distributional analysis across navigation and manipulation.
- Modern simulators provide scalable, safe, repeatable, and physically grounded platforms for robot learning and evaluation.
- Large-scale efforts diversify simulated assets and scenes, including internet-scale objects, procedurally generated houses, LLM-guided environments, and combinations of real scans and designed layouts.
- MolmoSpaces unifies 230K scenes, 130K object assets, and more than 48K grasp-annotated objects across MuJoCo, IsaacSim, and ManiSkill.
- Real-world benchmarking is difficult to scale because of hardware heterogeneity, sensing and control differences, and evaluation time and labor.
- Simulation benchmarks enable scalable, reproducible, controlled variation and systematic stress testing for manipulation and navigation.
- Existing sim-to-real benchmarks can show strong correlation but may be limited to tabletop manipulation, partial reconstruction, or proprietary evaluation pipelines.
- MolmoSpaces-Bench evaluates zero-shot generalist policies across navigation, manipulation, and mobile manipulation using over 20K scenes and more than 22K interactable objects.
3 MolmoSpaces
MolmoSpaces combines large, diverse scene and object collections with annotated grasps, simulator support, and scalable tooling for robotic evaluation. Its scene and grasp pipelines include physical validation, while remaining failures expose practical scene- and object-level constraints.
- MolmoSpaces-Scenes: Over 230k indoor environments span varied layouts and scene types, including handcrafted, procedural, and LLM-generated datasets.The scene collections include residential multiroom houses and diverse non-residential environments.
- MolmoSpaces-Objects: More than 130k rigid and articulated object models provide semantic and physical metadata for evaluating generalization to novel objects.The assets include THOR and Objaverse objects distributed across many categories and synsets.
- Simulation Infrastructure: The ecosystem supports MuJoCo, IsaacSim, and ManiSkill through converted assets, scene loaders, and occupancy maps for collision-free robot initialization.The same scenes can also use different MuJoCo rendering engines.
- Scene Quality Testing: Over 95% of MuJoCo environments pass stability and manipulation checks, although MSMultiType articulation reaches 63% because of scene-layout constraints.Subset evaluations in Isaac Sim also show consistently high pass rates.
- Grasp Quality and Limitations: In-situ grasp failures remain when surface placement, clearance, collisions, slippage, or articulation obstacles constrain execution.The pipeline was revised to bias contacts toward fingertip centers, with fingertip-edge contacts preferred for small or thin objects.
4 Benchmark
MolmoSpaces-Bench evaluates policies across eight base tasks spanning navigation, manipulation, and mobile manipulation in diverse, verified environments. It also supports controlled variants and LLM-generated long-horizon tasks for systematic evaluation.
- MolmoSpaces-Bench spans eight base tasks across navigation, manipulation, and mobile manipulation, with explicit diversity requirements and solvability verification.
- Tasks: The benchmark defines navigate-to, pick, four pick-and-place variants, open, close, and open-door tasks with task-specific success conditions.
- Embodiments: Manipulation tasks use a Franka FR3 in the DROID configuration, while navigation and open-door tasks use the Rainbow RB-Y1.
- Policies: The benchmark evaluates generalist π models, task-specific CAP models, and simulation-trained or dual-system navigation policies under their respective setups.
- Controlled and long-horizon evaluation: Controlled benchmark variants vary vocabulary, initial robot configurations, and clutter, while an LLM-based generator creates feasible long-horizon task sequences.
5 Experiments
Experiments evaluate zero-shot policies across diverse unseen scenes and compare simulation with real-world performance. They find strong pick-task sim-to-real correlation and expose sensitivities to prompts, robot initialization, camera occlusion, grasping strategy, and task horizon.
- 5.1 Policy evaluation: Zero-shot evaluation compares π, CAP, RING, and DualVLN policies across unseen environments and objects without task-specific fine-tuning.
- 5.2 Sim-to-real correlation: 0.96 Pearson and 0.98 Spearman correlations link MolmoSpaces-Bench pick success with 752 RoboArena pick tasks.
- 5.3 Distributional evaluations: More frequent DROID prompt phrasing brings π0 within 1% of π0.5, compared with a 14% gap for less frequent phrasing.
- 5.3 Distributional evaluations: Perturbing π0.5’s initial joint positions reduces performance, while lighting changes have little effect; wrist-camera occlusion reduces success to 2%.
- 5.3 Distributional evaluations: π0.5 favors top-down grasps and performs better on top-opening objects, whereas CAP favors side grasps and performs better on bottles and apples.
- 5.4 Controlled policy comparison: π models use a 300-step horizon versus 50 for CAP, and successful π episodes average 2.65 grasp-ungrasp transitions for π0.5 and 4.63 for π0.
6 Conclusion
The paper presents MolmoSpaces and MolmoSpaces-Bench as an open foundation for scalable, diverse robot-policy evaluation. It argues that simulation benchmarks can support progress toward robust real-world performance despite simulation’s inherent imperfections.
- MolmoSpaces combines large-scale simulation environments, diverse object assets, grasp datasets, and benchmarks covering base skills and LLM-assisted long-horizon tasks.
- Controlled difficulty and perturbations enable detailed analysis and principled comparison of generalist robot policies.
- Simulation remains imperfect, but well-designed simulation benchmarks provide a foundation for evaluating policies and guiding progress toward robust real-world performance.
A Object Model Dataset Details
The object-model dataset combines converted AI2-THOR and curated Objaverse assets with physical, semantic, geometric, and grasp annotations. Multi-stage filtering and placement alignment target reliable use in simulation and procedural scene generation.
- The dataset includes converted AI2-THOR objects and curated Objaverse objects in simulator-compatible formats, including articulable household assets.
- Physical realism is improved by validating masses and densities and manually tuning articulated joint properties with a simulated Franka FR3.
- Collider construction balances physical fidelity and stability by using convex decomposition for manipulable objects and primitive colliders for selected thin or receptacle objects.
- Additional annotation identifies object counts, excess geometry, missing geometry, texture quality, and receptacle parts to filter unreliable models.
- Filtering enforces metadata completeness, valid synsets, plausible scale, texture quality, cross-renderer fidelity, and processed file-size constraints.
- Heuristic alignment maps Objaverse synsets to compatible ProcTHOR categories using room, placement, scale, affordance, and semantic constraints.
- All objects provide scale, mass, semantic metadata, convex meshes, grasps, and canonical coordinates for loading into simulation scenes.
B MolmoSpaces-Scenes-MultiType Generation Details
MolmoSpaces-Scenes-MultiType generation combines constrained object placement, persona-conditioned scene styling, and automated asset and synset annotation. Its tooling also supports large-scale grasp evaluation across diverse generated scenes.
- Grid layout placement: A grid constraint extends the DFS-based floor object solver to jointly place repetitive object sets in structured layouts for non-residential scenes.The solver enumerates feasible row-by-column grid shapes and accommodates an iteratively decreasing object set.
- Grid layout placement: Grid configurations are scored by object-constraint satisfaction while softly discouraging door-to-door obstruction, and the highest-scoring placement is retained before DFS resumes.The grid footprint is placed within the current room, with positions and rotations selected greedily.
- Style accents with persona descriptions: Adding filtered persona descriptions to scene specifications produces stylistic changes and a wider variety of object types in LLM-generated scenes.Persona descriptions were filtered from PersonaHub with GPT-4.1-mini and then added through the binding structure used for roughly 110k MSMultiType scenes.
- Asset annotation and filtering: GPT-based asset filtering identifies multi-object 3D models and assesses missing or excessive geometry for Objaverse assets.The filtering prompt targets models such as kitchens, dining sets, or object assortments and distinguishes geometry defects from missing textures.
- Grasp evaluation: MolmoSpaces reports the highest number of object instances among the large-scale grasp datasets compared in Table 8.The comparison concerns large-scale grasp datasets.
- Synset placement options: GPT-generated synset placement options encode Boolean judgments about likely object properties in JSON, using associated types, descriptions, and scale ranges.These options support placement decisions for batches of synsets used to label objects.
- Persona selection: Persona selection uses GPT-4.1-mini to identify identity statements whose associated interests may make indoor scenes visually memorable through prominent objects.The task focuses on interest-relevant objects that stand out even in cluttered environments.
- Grasp evaluation: Figure 24 reports average grasp success rates by category for objects sampled from MSCrafted and MSProcObj scenes.The figure summarizes category-level grasp performance across a random sample of scenes from the MolmoSpaces-Scene dataset.