Source-linked AI summary
NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation
Yumeng He, Yichen Song, Xiaotian Yang, Weijia Zhang, Zanwei Zhou, Junru Gong, Xiaokang Yang, Yunbo Wang
TL;DR
NeoWorld-Pro tackles the difficulty of converting monocular observations into physically grounded, interactive simulation scenes. It synthesizes executable programs for geometry, articulation, and physical properties, then refines them with physics-in-the-loop feedback; experiments report improvements over open-loop baselines across reconstruction and downstream manipulation tasks.
Problem
Existing monocular image-to-URDF methods lack scene-level modeling, generalization beyond curated assets or category priors, and corrective feedback for physical inconsistencies.
Method
NeoWorld-Pro uses MLLM-driven procedural program synthesis followed by object-level and scene-level physics-in-the-loop refinement.
Results
NeoWorld-Pro significantly outperforms existing open-loop methods across asset reconstruction, articulation prediction, scene assembly, and manipulation task execution.
Takeaways & Limitations
The framework produces simulation-ready assets with accurate geometry, articulation, and spatial relationships for robot learning and physical interaction.
Takeaways & Limitations
NeoWorld-Pro is limited to rigid and articulated objects under quasi-static physics, and evaluation covers scenes with fewer than 10 objects.
Abstract
from arXiv · showhide
The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.
1 Introduction
NeoWorld-Pro addresses the lack of scalable, physically grounded scene reconstruction by translating monocular images into executable, editable programs and refining them through physics feedback. It targets simulation-ready geometry, articulation, physical properties, and inter-object relationships for embodied AI.
- Embodied AI requires interactive digital twins with accurate geometry, kinematics, physical properties, and simulation-ready inter-object relationships.
- Existing methods commonly focus on single-asset reconstruction, depend on curated assets or category priors, and operate open-loop without corrective physical feedback.
- NeoWorld-Pro reformulates monocular interactive scene construction as MLLM-driven procedural program synthesis with executable definitions of geometry, articulation, physical properties, and layout.
- Physics-in-the-loop execution feeds collision, instability, penetration, scale, and articulation feedback back into generated code for object- and scene-level refinement.
- NeoWorld-Pro is evaluated on asset reconstruction, articulation prediction, scene assembly, and manipulation, outperforming existing open-loop methods in producing robot-learning environments.
- The benchmark contains 90 articulated object categories and 30 USD-format scenes for validating the framework.
2 Related Work
Prior articulated-asset methods rely on structured inputs, retrieval or templates, category priors, or language models used mainly as auxiliary or interpretive components. NeoWorld-Pro instead combines direct procedural generation with physics-based refinement for scene-level reconstruction.
- Existing articulated-asset research predominantly uses static or dynamic point clouds and meshes as structured 3D inputs.
- Retrieval- and assembly-based methods are constrained by asset-library coverage and canonical biases, while category-prior methods remain limited by training-distribution coverage.
- Simulation and robotic-task evaluations are typically post-hoc, leaving articulation, contact, and inter-object spatial failures uncorrected during generation.
- Earlier LLM and MLLM approaches often provide supervision, annotations, augmentation, or interpretive priors while downstream modules construct the final assets.
- More recent methods directly generate structured articulated specifications, but remain largely constrained by limitations identified in prior single-image asset-generation approaches.
3 Method
NeoWorld-Pro builds simulation-ready scenes through MLLM-generated executable programs and two-level physics-in-the-loop refinement. The pipeline parses scene structure, programs articulated assets, and optimizes scene layouts using simulation and semantic feedback.
- Framework overview: NeoWorld-Pro represents a monocular scene as articulated URDF assets assembled in a USD stage, jointly inferring geometry, kinematics, and spatial relationships.
- Framework overview: MLLMs synthesize executable Blender Python programs that are compiled into URDF and USD assets, providing an executable, interpretable, and editable representation.
- Pipeline stages: The pipeline uses scene parsing, procedural asset programming, and object-level physics refinement in a generate-critic-refine workflow.
- Pipeline stages: Scene parsing predicts hierarchy, 2D and 3D boxes, and support or occlusion attributes so relational information constrains localization and layout.
- Pipeline stages: Procedural programming sequentially generates geometry, articulation, and URDF export code, with runtime debugging and rendered-output critic refinement.
- Object-level refinement: Object-level simulations use free-fall and force perturbation to expose mass, collision, penetration, disassembly, mobility, and articulation errors for targeted refinement.
- Scene-level refinement: The scene-level loop optimizes relative scale, orientation, and spatial offsets with CEM while excluding vertical perturbations to preserve scene-tree snapping constraints.
- Scene-level refinement: The hybrid reward combines semantic similarity with penetration and pose-drift penalties measured from rendered candidates and physics rollouts.
4 Experiments
Experiments evaluate NeoWorld-Pro across object reconstruction, scene assembly, physical validity, real-world generalization, and manipulation. The results indicate stronger geometric, articulatory, and scene-level consistency than open-loop alternatives.
- Benchmark: The benchmark contains 90 articulated object categories assembled into 30 USD-format multi-object scenes with occlusions and downstream tasks.
- Object-level reconstruction: NeoWorld-Pro achieves the most accurate qualitative recovery of articulated joints and highly accurate geometric reconstructions in single-object image-to-URDF evaluation.
- Object-level reconstruction: Across appearance, geometry, and articulation metrics, NeoWorld-Pro delivers strong overall performance, including best LPIPS and CLIP, highest F-scores, and improved joint inference.
- Scene-level reconstruction: On synthetic scenes, NeoWorld-Pro better preserves objects, relative scale, orientation, and spatial relationships while supporting stable and physically meaningful manipulation behaviors.
- Real-world generalization: On real-world images with clutter, illumination variation, and partial occlusion, reconstructed primary objects and spatial relationships remain usable, with movable components retaining functionality.
- Ablation studies: Ablations show that semantic scoring captures arrangement and affordances, while simulator feedback corrects penetration, unstable stacking, scale, and articulation errors.
5 Conclusion and Limitation
NeoWorld-Pro combines executable program synthesis with simulation feedback to produce simulation-ready reconstructed assets. Its current scope is limited to rigid or articulated objects, quasi-static physics, and scenes containing fewer than 10 objects.
- Conclusion: NeoWorld-Pro produces simulation-ready assets with accurate geometry, articulation, and spatial relationships through MLLM-driven programs and object- and scene-level feedback.
- Conclusion: The evaluation reports improvements over open-loop baselines in appearance fidelity, geometric accuracy, articulation prediction, and downstream manipulation tasks.
- Limitations: The framework does not yet model highly deformable or continuous media, richer material dynamics, or scenes with 10 or more objects.
A Scene Task Configurations
The scene-level task set tests reconstructed environments across varied object configurations and manipulation goals. Tasks assess whether scenes are visually plausible and physically executable in simulation.
- Task coverage: The task set includes placement, assembly, and interaction with articulated parts across diverse object configurations and manipulation goals.
- Execution criteria: Tasks require reasonable relative scale and pose, limited interpenetration, and task-relevant affordances for downstream manipulation.
B Scene-Level Refinement Implementation Details
Scene-level refinement initializes object scale and pose distributions from parsed geometry, then uses normalized semantic and physical rewards to optimize layouts. Transform conventions and scale-adaptive perturbations are designed for stable CEM refinement.
- Initialization: Each object URDF is deterministically rescaled to its parsed 3D bounding box before CEM refinement, establishing absolute scale from the input scene.
- CEM distribution: CEM initializes a diagonal Gaussian over planar translation, yaw, and log-scale, with perturbations adapted to object size.
- CEM distribution: The implementation uses 15° yaw noise and σ_log s = 0.1, approximately corresponding to a 10% multiplicative scale perturbation.
- Transform application: Candidate transforms apply scale, yaw rotation, and translation in that order to match intended USD transform semantics.
- Reward design: Reward terms are normalized before weighting, while the semantic score evaluates depth ordering, relative layout, and related criteria from canonical-camera renderings.