Source-linked AI summary
Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
Zihan Wang, Seungjun Lee, Yinghao Xu, Gim Hee Lee
TL;DR
Embodied navigation lacks scalable environments that combine visual fidelity, physical grounding, and interactive structure. Image2Sim builds such environments from posed visual captures by separating feed-forward 3D scene construction from generative rendering and automating action and instruction synthesis. Models trained in these environments show strong benchmark performance, cross-simulator transfer, and real-world zero-shot generalization, while the system remains limited in richer physical interactions and may introduce linguistic mismatch.
Problem
Embodied-navigation progress is constrained by the lack of scalable, high-fidelity, physically grounded, interactive 3D environments.
Method
Image2Sim combines feed-forward 3D feature-Gaussian reconstruction, geometry-aware one-step panoramic RGB-D rendering, collision-aware motion simulation, and automated instruction generation.
Results
Image2Sim produces over 10M navigation samples across 20K interactive scenes, and models trained in these environments achieve strong benchmark performance with cross-simulator and real-world zero-shot transfer.
Takeaways & Limitations
Scalable neural simulation is presented as a practical foundation for scaling embodied navigation beyond manually modeled scenes and annotated trajectories.
Takeaways & Limitations
The simulator does not yet support richer contact dynamics, movable objects, or human–robot interactions, and VLM annotation may introduce linguistic bias or semantic mismatch.
Abstract
from arXiv · showhide
Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.
1 Introduction
Image2Sim addresses the embodied-navigation data bottleneck by separating geometric scene anchoring from photorealistic synthesis and extending simulation into automated vision-action-instruction generation. It scales this framework to thousands of scenes and millions of samples while supporting benchmark gains and transfer beyond its training simulator.
- Motivation: Existing embodied-navigation data sources trade visual fidelity against scalability, physical grounding, or interactive consistency.Real-world scans are visually strong but expensive to scale, while procedural and generative alternatives have realism or interaction limitations.
- Image2Sim Framework: Image2Sim decouples 3D spatial anchoring from photorealistic observation synthesis through feed-forward feature-Gaussian construction and geometry-aware one-step panoramic RGB-D rendering.The construction stage produces an explicit geometric and semantic representation, while rendering synthesizes observations from sparse Gaussian projections.
- Image2Sim Framework: Approximately 40 FPS panoramic RGB-D synthesis on a single RTX 4090 supports geometrically grounded, high-fidelity, computationally efficient environments.The simulator is designed for real-time neural simulation rather than only offline scene reconstruction.
- Scalable Embodied Data Engine: Over 10M navigation training samples across 20K interactive scenes are produced by integrating object-centric target mining, physics-aware trajectories, and VLM-based instruction annotation.The resulting data engine jointly constructs vision, action, and language supervision without manual trajectory or instruction annotation.
- Empirical Evidence for Embodied Scaling: Models trained exclusively in Image2Sim achieve strong cross-simulator performance and effective real-world zero-shot transfer, with consistent unsaturated gains as generated data increases.These findings provide empirical evidence for scaling embodied navigation with neural simulation.
2 Related Work
Embodied-navigation simulation spans realistic scanned environments and scalable synthetic environments, but each paradigm has important limitations. Neural scene methods improve visual fidelity yet commonly require costly per-scene optimization, while navigation datasets remain constrained by annotation and environment scale.
- Simulation Environments for Embodied Navigation: Real-scanned environments provide geometric realism, real-world layout statistics, and physical grounding but are expensive and labor-intensive to create at scale.High-quality scanning and digital-twin post-processing limit scene scale and diversity.
- Simulation Environments for Embodied Navigation: Synthetic and procedural environments improve scalability but can introduce gaps from unrealistic assets, layouts, and rendering statistics.This creates a contrast between easier scaling and fidelity to real-world navigation conditions.
- Neural Scene Representations and World Models: Optimization-based neural scene representations achieve high-fidelity rendering but require costly per-scene optimization, limiting their use as large-scale embodied data engines.Embodied simulators built on such neural scenes inherit this scalability limitation.
- Scaling Vision-Language Navigation: Human-annotated vision-language-navigation datasets provide high-quality supervision but remain constrained by annotation cost and the limited number of underlying 3D environments.Later approaches synthesize instructions, goals, or trajectories to reduce annotation bottlenecks.
3 Method
Image2Sim builds executable neural simulators from posed RGB-D captures by combining feed-forward 3D feature-Gaussian construction, geometry-aware panoramic rendering, and physically grounded motion simulation.
- System formulation: Image2Sim converts posed RGB-D observations into a persistent 3D feature-Gaussian scene, traversable voxel graph, and pose-conditioned panoramic RGB-D renderer.The simulator supports physically grounded atomic actions, collision-aware trajectory sampling, and rendering along feasible paths.
- Feed-Forward 3D Scene Construction: Feed-forward lifting uses semantic and geometric features to predict Gaussians whose depth-unprojected centers anchor the representation to metric scene geometry.Each Gaussian stores center, anisotropic scale, rotation, opacity, RGB values, and compact semantic features; voxel downsampling avoids per-scene optimization.
- Geometry-Aware One-Step Pixel Flow Rendering: The renderer combines Gaussian-splatted RGB-D and semantic projections with conditional pixel-flow completion to preserve observed geometry while filling sparse-capture holes.An alpha-gated source state keeps high-opacity regions near projected 3D evidence and gives larger perturbations to low-opacity regions for generative completion.
- Training the Pixel-Flow Renderer: A UNet-style geometry-conditioned model with a transformer bottleneck predicts one-step transport from noisy projected source states to target panoramic RGB-D observations.MeanFlow estimates average velocity for one-step generation, while opacity-adaptive weighting concentrates gradients on low-confidence regions.
- Training the Pixel-Flow Renderer: Momentum-based self-distillation improves single-step completion by training a student to match multi-scale decoder features from an EMA teacher conditioned on privileged ground-truth RGB-D state.Normalized layer-wise weights emphasize deeper semantic decoder layers.
- Motion Simulation and Trajectory Generation: Motion simulation voxelizes the reconstructed scene and combines semantic labels, obstacle geometry, occupancy queries, and agent-footprint clearance to form a traversable connectivity graph.GPU-parallel ray marching rejects invalid motions, detects penetration, and supports physically plausible sliding along obstacle boundaries.
4 Experiments
Experiments evaluate Image2Sim's rendering quality, cross-simulator navigation transfer, scaling behavior, ablations, and real-world performance. Results show robust rendering under noisy sparse inputs, strong zero-shot navigation generalization, gains from larger generated datasets, and improved physical-world success.
- Experimental Setup: 19,936 scenes from real-world and synthetic datasets form the training environments, using panoramic and pinhole inputs for neural simulation.The renderer outputs 512×1024 panoramic RGB-D images; training excludes evaluation and test scenes.
- Novel-View Rendering: 45.6 FPS with 17.43 PSNR and 0.470 SSIM demonstrates robust panoramic rendering on sparse, high-noise LiDAR depth.This substantially outperforms Gaussian-only AnySplat in rendering quality while preserving real-time operation.
- Cross-Simulator Zero-Shot Navigation: Image2Nav achieves new state-of-the-art results across R2R-CE, RxR-CE, and REVERIE-CE when trained in Image2Sim and evaluated zero-shot in Habitat.The cross-simulator evaluation tests whether generated data transfers beyond the training environment rather than only improving in-domain performance.
- Scaling Law: 10M training samples produce consistent unsaturated gains, with SR increasing from 46.1 to 66.3 and SPL from 41.3 to 61.5.Scaling jointly expands interactive scenes, feasible trajectories, rendered observations, and generated instructions rather than only augmenting language over fixed environments.
- Ablation Study: Ablations show that pixel flow, projected semantic features, alpha-gated fusion, and momentum-based self-distillation each support rendering quality and stability.Removing these components degrades completion, weak-region artifact suppression, or preservation of reliable projected geometry.
- Real-World Navigation: Image2Nav improves real-world path-following success from 8/20 to 11/20 and goal-oriented success from 5/20 to 9/20 over the strongest baselines.The evaluation uses 20 trials per instruction type on a Hello Robot Stretch 3 in a home-like environment.
5 Conclusion
Image2Sim converts passive visual captures into scalable, physically grounded, interactive environments and synthesizes vision-language-action data without manual scene modeling or trajectory annotation. Experiments report strong benchmark performance and improved cross-simulator and real-world generalization, while limitations remain in renderer capacity, physical interaction scope, and instruction annotation.
- Conclusion: Image2Sim combines feature-Gaussian reconstruction, panoramic RGB-D generation, collision-aware motion simulation, and automated instruction generation.Together, these components convert passive visual captures into interactive navigation environments and vision-language-action data.
- Conclusion: Navigation models trained with Image2Sim achieve strong benchmark performance and improved cross-simulator and real-world generalization.The paper presents neural simulation as a practical foundation for scaling embodied navigation.
- Limitations: The compact renderer trades off model capacity and generation ability to support real-time interaction, low computational cost, and online DAgger/RL training.This limitation reflects the design constraints of the compact renderer.
- Limitations: The simulator mainly supports navigation-level physical validity rather than richer contact dynamics, movable objects, or human–robot interactions.These interaction capabilities are identified as unsupported scope areas.
- Limitations: VLM-based instruction annotation enables scale but may introduce linguistic bias or occasional semantic mismatch.The limitation concerns the reliability and neutrality of automatically generated language instructions.
A Data Analysis and Statistics
This section analyzes the datasets used to build Image2Sim’s interactive environments and the distributions of its generated navigation data.
- A Data Analysis and Statistics: The section provides detailed statistics for the datasets used to build Image2Sim’s interactive environments.The analysis concerns the underlying data used for environment construction.
- A Data Analysis and Statistics: The section analyzes the distributions of the navigation data generated by Image2Sim.The distributions form a separate focus of the data analysis.
- A Data Analysis and Statistics: The analysis covers both interactive-environment construction data and generated navigation data.This section therefore combines dataset statistics with generated-data distribution analysis.
A.1 Environment Dataset Statistics
Image2Sim aggregates diverse real-world and synthetic datasets into nearly 20K scenes with extensive panoramic, pinhole-image, and navigable-area coverage, while excluding validation and test scenes from training.
- A.1 Environment Dataset Statistics: 19,936 scenes provide the foundation for Image2Sim’s interactive 3D environments.The scenes are aggregated from diverse real-world and synthetic datasets.
- A.1 Environment Dataset Statistics: The environment collection contains over 617K panoramas, 1.16M pinhole images, and nearly 1.2M m2 of navigable space.These statistics summarize the collection’s visual coverage and navigable area.
- A.1 Environment Dataset Statistics: Twenty-nine validation and test scenes are excluded from the 90 Matterport3D scenes during navigation training to prevent data leakage.The exclusion applies specifically to the Matterport3D subset.
- A.1 Environment Dataset Statistics: Table 6 summarizes scene, panoramic-image, pinhole-image, and navigable-area statistics for the 3D environments.The table provides detailed statistics for the scenes used in Image2Sim.
A.2 Generated Navigation Data Statistics
Image2Sim generates over 10 million diverse vision-language-action samples across balanced instruction styles, with varied instruction lengths and physical trajectories spanning local approaches to long-horizon exploration.
- A.2 Generated Navigation Data Statistics: Over 10 million vision-language-action training samples are synthesized from the constructed scenes.The generated instructions cover diverse text lengths and styles to mitigate instruction bias.
- A.2 Generated Navigation Data Statistics: R2R-style path-following, REVERIE-style object-oriented, and Demand-style human-centric instructions contribute 33.42%, 33.49%, and 33.09% of samples, respectively.The three instruction categories are distributed approximately evenly.
- A.2 Generated Navigation Data Statistics: Generated instructions range from 1 to 581 words, with a mean of 38.1 words and a median of 24.0 words.The distribution combines concise commands with detailed navigational guidance.
- A.2 Generated Navigation Data Statistics: Trajectory lengths have a mean of 8.53 meters, a median of 8.50 meters, and a standard deviation of 3.86 meters.The physical trajectories range from 0.37 meters to 85.72 meters.
- A.2 Generated Navigation Data Statistics: The trajectory distributions include discrete-action step counts alongside physical trajectory lengths.Figure 8 presents the distribution of trajectory steps, while Figure 7 presents trajectory lengths in meters.
B Clarifications on Experimental Setup and Performance Analysis
The experimental setup standardizes inputs across heterogeneous rendering baselines, enabling a fair comparison of their novel-view rendering performance.
- Table 1 experiments use standardized input preprocessing to ensure fair comparison among rendering baselines with heterogeneous input requirements.The preprocessing procedure is designed to equalize input conditions before evaluating the baselines.
- DiT360 and SE3DS condition on base images rendered by Gaussian splatting and output native panoramic RGB images.
- AnySplat receives six perspectively projected pinhole RGB-D images, each with a 90◦ field of view, because it requires pinhole inputs.
B.2 Source of Navigation Performance Gains (Table 2 and Table 3)
The analysis attributes Image2Nav’s navigation gains primarily to the scale and diversity of Image2Sim-generated data rather than architectural complexity. Training on standard human-annotated datasets alone performs substantially worse, while 10M synthetic samples support competitive generalization.
- Image2Sim-generated data, rather than architectural inductive biases or complex navigation modules, fundamentally accounts for Image2Nav’s substantial performance improvements.
- Image2Nav trained exclusively on R2R and RxR performs significantly lower than almost all competing Table 2 baselines.
- 10M synthetic training samples drive Image2Nav’s competitive edge and superior cross-simulator generalization, while the architecture provides high training efficiency.
B.3 Hyperparameters and Loss Weights
Image2Sim uses a 400K-iteration AdamW training schedule with staged curriculum learning that shifts emphasis from geometry and flow distillation to high-frequency refinement. The real-world evaluation uses a Stretch 3 robot with streaming RGB-D input and remote inference.
- Hyperparameters: Image2Sim is trained with AdamW at an initial learning rate of 1 × 10−4, weight decay of 1 × 10−4, and cosine decay to 1 × 10−6 over 400K iterations.Training uses Distributed Data Parallel with per-GPU batch size 2.
- Curriculum: A three-stage curriculum stabilizes one-step rendering and distillation by dynamically changing loss priorities across training.
- Curriculum: Stage 1 grounds the 3D representation, Stage 2 introduces EMA-guided flow distillation, and Stage 3 freezes major modules while refining decoder high-frequency details.The perceptual-loss weight increases from 1.0 to 2.0 and then 4.0 across the stages.
- Real-world setup: The real-world evaluation uses a Hello Robot Stretch 3 with a head-mounted Intel RealSense D435i RGB-D camera and remote RTX 4090 inference.The robot communicates with the workstation over a local-area network in a home-like environment.
C Detailed Architecture of Image2Sim
Image2Sim decouples persistent 3D spatial anchoring from photorealistic generative completion through feed-forward scene encoding, projection and semantic alignment, and MeanFlow rendering. Its geometry-aware motion engine extracts navigable space, builds collision-aware voxel graphs, and produces executable trajectories.
- Architecture: The modular architecture has three stages: Scene Encoding, Projection and Semantic Alignment, and MeanFlow Rendering.The framework explicitly separates 3D spatial anchoring from photorealistic generative completion.
- Scene Encoding: A feed-forward first stage constructs a persistent 3D Gaussian scene from visual captures without per-scene optimization.RGB, metric depth, and camera ray directions support format-agnostic inputs, including panoramic and pinhole views.
- Scene Encoding: Dual semantic and detail streams fuse visual, geometric, and ray-based context into 16-dimensional scene features and 8-dimensional 3D Gaussian attributes.The semantic stream uses frozen DINOv3 features, while the detail stream processes RGB, depth, normals, and ray maps.
- Rendering: Given a target pose, PanoAnisotropicSplatting produces feature, RGB, depth, and alpha projections that condition semantic alignment and the RenderingDecoder.The decoder synthesizes photorealistic panoramic RGB-D observations with a continuous-time MeanFlow formulation.
- Rendering: Alpha-gated skip connections use multi-scale opacity masks to prevent noisy Gaussian projections from corrupting generated regions.A zero-initialized gate with bias 2.0 gives an initial value of σ(2.0) ≈0.88, favoring projections early in training.
- Motion simulation: The motion engine cleans segmented navigable point clouds, voxelizes them at 0.05m, and constructs collision-free connectivity graphs using clearance and line-of-sight checks.Collision-aware costs favor safer corridors, and Dijkstra’s algorithm plans paths through the graph.