PaperlayerView exact revision on arXiv

Source-linked AI summary

NeoWorld-Pro: Programming Interactive Scenes from Monocular Images for Embodied Simulation

Yumeng He, Yichen Song, Xiaotian Yang, Weijia Zhang, Zanwei Zhou, Junru Gong, Xiaokang Yang, Yunbo Wang

arXiv:2608.24212v1cs.CV

TL;DR

NeoWorld-Pro tackles the difficulty of converting monocular observations into physically grounded, interactive simulation scenes. It synthesizes executable programs for geometry, articulation, and physical properties, then refines them with physics-in-the-loop feedback; experiments report improvements over open-loop baselines across reconstruction and downstream manipulation tasks.

  • Problem

    Existing monocular image-to-URDF methods lack scene-level modeling, generalization beyond curated assets or category priors, and corrective feedback for physical inconsistencies.

    Source: §1 · p. 1

  • Method

    NeoWorld-Pro uses MLLM-driven procedural program synthesis followed by object-level and scene-level physics-in-the-loop refinement.

    Source: §1 · p. 2

  • Results

    NeoWorld-Pro significantly outperforms existing open-loop methods across asset reconstruction, articulation prediction, scene assembly, and manipulation task execution.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The framework produces simulation-ready assets with accurate geometry, articulation, and spatial relationships for robot learning and physical interaction.

    Source: §5 · p. 11

    NeoWorld-Pro is limited to rigid and articulated objects under quasi-static physics, and evaluation covers scenes with fewer than 10 objects.

    Source: §5 · p. 11

Abstract

from arXiv · show

The advancement of Embodied AI necessitates high-quality simulation assets that faithfully mirror the real world. However, transforming raw visual observations into simulation-ready scenes remains challenging due to the lack of physical grounding and scene-level interactivity in current image-to-URDF methods. We propose NeoWorld-Pro, a framework that reformulates monocular scene reconstruction as procedural programming for interactive 3D environments. Leveraging the zero-shot reasoning and code synthesis capabilities of MLLMs, NeoWorld-Pro converts a single RGB image into executable programs specifying object geometry, articulation, and physical properties. A physics-in-the-loop mechanism then iteratively refines the generated programs by validating their execution in a physics engine, enforcing physically plausible articulations, valid object compositions and interactions, and accurate spatial relationships. Experiments show that NeoWorld-Pro outperforms open-loop and prior monocular reconstruction methods, while enabling complex downstream tasks such as stable stacking and fine-grained manipulation.

1 Introduction

NeoWorld-Pro addresses the lack of scalable, physically grounded scene reconstruction by translating monocular images into executable, editable programs and refining them through physics feedback. It targets simulation-ready geometry, articulation, physical properties, and inter-object relationships for embodied AI.

  • Embodied AI requires interactive digital twins with accurate geometry, kinematics, physical properties, and simulation-ready inter-object relationships.Source: §1 · p. 1
  • Existing methods commonly focus on single-asset reconstruction, depend on curated assets or category priors, and operate open-loop without corrective physical feedback.Source: §1 · p. 1
  • NeoWorld-Pro reformulates monocular interactive scene construction as MLLM-driven procedural program synthesis with executable definitions of geometry, articulation, physical properties, and layout.Source: §1 · p. 2
  • Physics-in-the-loop execution feeds collision, instability, penetration, scale, and articulation feedback back into generated code for object- and scene-level refinement.Source: §1 · p. 2
  • NeoWorld-Pro is evaluated on asset reconstruction, articulation prediction, scene assembly, and manipulation, outperforming existing open-loop methods in producing robot-learning environments.Source: §1 · p. 2
  • The benchmark contains 90 articulated object categories and 30 USD-format scenes for validating the framework.Source: §1 · p. 3

2 Related Work

Prior articulated-asset methods rely on structured inputs, retrieval or templates, category priors, or language models used mainly as auxiliary or interpretive components. NeoWorld-Pro instead combines direct procedural generation with physics-based refinement for scene-level reconstruction.

  • Existing articulated-asset research predominantly uses static or dynamic point clouds and meshes as structured 3D inputs.Source: §2.1 · p. 3
  • Retrieval- and assembly-based methods are constrained by asset-library coverage and canonical biases, while category-prior methods remain limited by training-distribution coverage.Source: §2.1 · p. 3
  • Simulation and robotic-task evaluations are typically post-hoc, leaving articulation, contact, and inter-object spatial failures uncorrected during generation.Source: §2.1 · p. 3
  • Earlier LLM and MLLM approaches often provide supervision, annotations, augmentation, or interpretive priors while downstream modules construct the final assets.Source: §2.2 · p. 3
  • More recent methods directly generate structured articulated specifications, but remain largely constrained by limitations identified in prior single-image asset-generation approaches.Source: §2.2 · p. 3

3 Method

NeoWorld-Pro builds simulation-ready scenes through MLLM-generated executable programs and two-level physics-in-the-loop refinement. The pipeline parses scene structure, programs articulated assets, and optimizes scene layouts using simulation and semantic feedback.

Framework overview

  • NeoWorld-Pro represents a monocular scene as articulated URDF assets assembled in a USD stage, jointly inferring geometry, kinematics, and spatial relationships.Source: §3.1 · p. 4
  • MLLMs synthesize executable Blender Python programs that are compiled into URDF and USD assets, providing an executable, interpretable, and editable representation.Source: §3.1 · p. 4

Pipeline stages

  • The pipeline uses scene parsing, procedural asset programming, and object-level physics refinement in a generate-critic-refine workflow.Source: §3.2 · p. 5
  • Scene parsing predicts hierarchy, 2D and 3D boxes, and support or occlusion attributes so relational information constrains localization and layout.Source: §3.2 · p. 5
  • Procedural programming sequentially generates geometry, articulation, and URDF export code, with runtime debugging and rendered-output critic refinement.Source: §3.2 · p. 5

Object-level refinement

  • Object-level simulations use free-fall and force perturbation to expose mass, collision, penetration, disassembly, mobility, and articulation errors for targeted refinement.Source: §3.2 · p. 5

Scene-level refinement

  • The scene-level loop optimizes relative scale, orientation, and spatial offsets with CEM while excluding vertical perturbations to preserve scene-tree snapping constraints.Source: §3.1 · p. 5, §3.3 · p. 6
  • The hybrid reward combines semantic similarity with penetration and pose-drift penalties measured from rendered candidates and physics rollouts.Source: §3.3 · p. 7

4 Experiments

Experiments evaluate NeoWorld-Pro across object reconstruction, scene assembly, physical validity, real-world generalization, and manipulation. The results indicate stronger geometric, articulatory, and scene-level consistency than open-loop alternatives.

Benchmark

  • The benchmark contains 90 articulated object categories assembled into 30 USD-format multi-object scenes with occlusions and downstream tasks.Source: §4.1 · p. 7

Object-level reconstruction

  • NeoWorld-Pro achieves the most accurate qualitative recovery of articulated joints and highly accurate geometric reconstructions in single-object image-to-URDF evaluation.Source: §4.2 · p. 7
  • Across appearance, geometry, and articulation metrics, NeoWorld-Pro delivers strong overall performance, including best LPIPS and CLIP, highest F-scores, and improved joint inference.Source: §4.2 · pp. 8–9

Scene-level reconstruction

  • On synthetic scenes, NeoWorld-Pro better preserves objects, relative scale, orientation, and spatial relationships while supporting stable and physically meaningful manipulation behaviors.Source: Figure 4 · p. 8, §4.3 · p. 9

Real-world generalization

  • On real-world images with clutter, illumination variation, and partial occlusion, reconstructed primary objects and spatial relationships remain usable, with movable components retaining functionality.Source: §4.3 · p. 9, Figure 5 · p. 9

Ablation studies

  • Ablations show that semantic scoring captures arrangement and affordances, while simulator feedback corrects penetration, unstable stacking, scale, and articulation errors.Source: §4.5 · pp. 10–11

5 Conclusion and Limitation

NeoWorld-Pro combines executable program synthesis with simulation feedback to produce simulation-ready reconstructed assets. Its current scope is limited to rigid or articulated objects, quasi-static physics, and scenes containing fewer than 10 objects.

Conclusion

  • NeoWorld-Pro produces simulation-ready assets with accurate geometry, articulation, and spatial relationships through MLLM-driven programs and object- and scene-level feedback.Source: §5 · p. 11
  • The evaluation reports improvements over open-loop baselines in appearance fidelity, geometric accuracy, articulation prediction, and downstream manipulation tasks.Source: §5 · p. 11

Limitations

  • The framework does not yet model highly deformable or continuous media, richer material dynamics, or scenes with 10 or more objects.Source: §5 · p. 11

A Scene Task Configurations

The scene-level task set tests reconstructed environments across varied object configurations and manipulation goals. Tasks assess whether scenes are visually plausible and physically executable in simulation.

Task coverage

  • The task set includes placement, assembly, and interaction with articulated parts across diverse object configurations and manipulation goals.Source: A Scene Task Configurations · p. 14

Execution criteria

B Scene-Level Refinement Implementation Details

Scene-level refinement initializes object scale and pose distributions from parsed geometry, then uses normalized semantic and physical rewards to optimize layouts. Transform conventions and scale-adaptive perturbations are designed for stable CEM refinement.

Initialization

  • Each object URDF is deterministically rescaled to its parsed 3D bounding box before CEM refinement, establishing absolute scale from the input scene.Source: B Scene-Level Refinement… · p. 14

CEM distribution

Transform application

Reward design

  • Reward terms are normalized before weighting, while the semantic score evaluates depth ordering, relative layout, and related criteria from canonical-camera renderings.Source: B Scene-Level Refinement… · p. 15

Related papers

Loading 2608.24212v1…