Source-linked AI summary
WorldAgents: Can Foundation Image Models be Agents for 3D World Models?
Ziya Erkoç, Angela Dai, Matthias Nießner
TL;DR
The paper investigates whether 2D foundation image models inherently possess 3D world model capabilities, addressing the challenge of producing coherent 3D-consistent environments. It introduces a VLM-directed multi-agent pipeline with image synthesis and two-stage verification, and finds that the approach extracts implicit 3D knowledge to produce robust, expansive scenes supporting novel-view rendering.
Problem
The paper asks whether 2D foundation image models trained on 2D projections implicitly learn 3D spatial structures, potentially reducing reliance on scarce explicit 3D training data.
Method
The method combines a VLM director, a 2D image generator, and a VLM-backed verifier that checks generated views in both 2D image and 3D reconstruction spaces.
Results
The agentic approach produces robust, 3D-consistent scene reconstructions that support rendering arbitrary novel views, demonstrating that 2D foundation models encapsulate a grasp of 3D worlds.
Takeaways & Limitations
Careful multi-agent orchestration can exploit implicit 3D knowledge in 2D foundation models for expansive, realistic, navigable world synthesis.
Takeaways & Limitations
The method is framed around image diffusion models, and extending it to video diffusion models is identified as future work because longer sequences can accumulate geometric drift and multi-view inconsistencies.
Abstract
from arXiv · showhide
Given the remarkable ability of 2D foundation image models to generate high-fidelity outputs, we investigate a fundamental question: do 2D foundation image models inherently possess 3D world model capabilities? To answer this, we systematically evaluate multiple state-of-the-art image generation models and Vision-Language Models (VLMs) on the task of 3D world synthesis. To harness and benchmark their potential implicit 3D capability, we propose an agentic framing to facilitate 3D world generation. Our approach employs a multi-agent architecture: a VLM-based director that formulates prompts to guide image synthesis, a generator that synthesizes new image views, and a VLM-backed two-step verifier that evaluates and selectively curates generated frames from both 2D image and 3D reconstruction space. Crucially, we demonstrate that our agentic approach provides coherent and robust 3D reconstruction, producing output scenes that can be explored by rendering novel views. Through extensive experiments across various foundation models, we demonstrate that 2D models do indeed encapsulate a grasp of 3D worlds. By exploiting this understanding, our method successfully synthesizes expansive, realistic, and 3D-consistent worlds.
1 Introduction
The paper asks whether 2D foundation models implicitly encode 3D world knowledge and proposes a multi-agent framework to exploit that capability for consistent 3D world generation.
- Motivation: 3D-consistent environment synthesis remains challenging because existing methods face scarce high-quality 3D data or costly multi-view consistency constraints.The paper positions 2D models as a potential alternative source of 3D knowledge.
- Motivation: 2D foundation models may implicitly learn spatial structures and physical rules because their training images are 2D projections of the 3D world.This motivates investigating whether they possess inherent 3D world model capabilities and could reduce reliance on explicit 3D datasets.
- Method: The proposed method frames 3D scene generation as a collaborative process involving a VLM director, image generator, and two-step verifier.The architecture orchestrates 2D foundation models rather than relying on single-pass prompting.
- Method: The VLM director formulates prompts, the image generator sequentially inpaints geometrically aligned views, and the verifier selectively evaluates generated frames.The verifier checks semantic and structural coherence in 2D image space, then geometric alignment in 3D reconstruction space.
- Results: The agentic approach produces robust 3D reconstructions that support rendering arbitrary novel views across generated environments.The experiments are presented as evidence that 2D foundation models encapsulate useful 3D world understanding.
2 Related Works
Related work spans image- and depth-based scene generation, panorama and video approaches, image foundation models, and VLM-driven 3D reconstruction. The paper distinguishes its approach by using VLM agents to orchestrate direct frame generation for navigable worlds.
- Scene generation: Prior methods generate 3D worlds from text or input views using image or video models with 3D-based, often camera-controlled, conditioning.Panorama generation and layered image generation are representative approaches.
- Scene generation: WonderWorld and Text2Room use handcrafted prompts with image- and depth-inpainting to synthesize new regions for 3D scenes.The paper contrasts this prompt design with its VLM-based orchestration.
- Image foundation models: Image foundation models have also supported downstream 3D reconstruction through Score Distillation Sampling and related conditioning or personalization methods.These models provide the generative foundation for several 3D-oriented techniques.
- Agent-driven generation: VLM agents have demonstrated semantic scene understanding and the ability to manipulate code representations for image-to-3D reconstruction.VIGA is identified as the closest related agent-based method.
- Agent-driven generation: Unlike VIGA’s proxy-code approach for static reconstruction, this paper directly generates image frames to synthesize interactive, navigable 3D worlds from text prompts.Its frame-by-frame generation is inspired by VLM reasoning and advances in 2D foundation models.
3 Method
The method orchestrates a Director, Generator, and Verifier to iteratively synthesize and curate views before reconstructing a navigable 3D Gaussian Splatting scene. It combines VLM-guided semantic planning, sequential inpainting from re-rendered views, and binary checks for semantic and geometric consistency.
- 3 Method: Three specialized agents collaboratively generate 3D scenes: a VLM Director plans views, a 2D Generator synthesizes them, and a Verifier maintains consistency.The Director analyzes verified history and proposes prompts; the Generator uses image-conditioned synthesis; the Verifier compares each candidate with accepted views.
- 3 Method: Verified views, camera poses, and prompts are accumulated into a world state and finally reconstructed as a 3D Gaussian Splatting representation for navigation and exploration.The final representation is produced after the iterative Director–Generator–Verifier process terminates.
- 3 Method: The process explores left- and rightward viewpoints with fixed rotations and random perturbations, stopping when coverage is sufficient or the image budget is reached.The Director can signal that the scene is comprehensively covered; otherwise, the trajectory continues through additional candidate views.
- 3 Method: The Verifier acts as a binary gate that evaluates each candidate against the established world in both 2D semantic space and 3D reconstruction space.Rejected candidates are discarded and the Generator resamples; accepted frames update the global state, enforcing multi-view constraints.
- 3 Method: The Director maps the current world state and global prompt to view-specific text prompts that guide semantic scene expansion.It uses verified views and previous prompts to select meaningful next viewpoints, while the initial frame is generated directly from the input description.
- 3 Method: The Generator reconstructs verified history into 3D Gaussians, renders a target viewpoint, and uses sequential inpainting to fill disoccluded regions.Explicit reprojection preserves geometric alignment in overlapping regions, while the 2D foundation model completes missing content under the Director’s localized prompt.
4 Experiments
Experiments compare WorldAgents with state-of-the-art baselines and varied image-model/VLM combinations, then assess how its components affect 3D scene quality. Results show stronger rendering fidelity, scene complexity, prompt alignment, and coherence, while ablations identify verifier, director, and inpainting contributions.
- Comparisons with State of the Art: WorldAgents outperforms Text2Room and WorldExplorer in rendering fidelity and scene complexity, producing denser, more structurally realistic scenes with fewer artifacts.Baselines show sparse scenes, structural artifacts, and blurred object boundaries, whereas the method achieves better standard metrics.
- Image Model and VLM Analysis: Flux.2 [Pro] with GPT-4.1 achieves the most favourable performance among evaluated image-model and VLM combinations.Other combinations remain generally plausible but exhibit geometric inconsistencies, incomplete views, weaker inpainting, or less accurate directives and verifications.
- Ablations: Integrating the generator, verifier, director, and inpainting components resolves blur, incompleteness, and window misalignments in the ablation scene.The full configuration synthesizes a coherent scene after partial configurations retain distinct artifacts or inconsistencies.
- Ablations: The verifier filters geometrically inconsistent generations, removes catastrophic failures, and reduces blur by avoiding inconsistent frames.Its role is to prevent erroneous views from corrupting the global reconstruction geometry.
- Ablations: The director dynamically creates context-aware prompts that improve scene completeness, structural diversity, and overall fidelity over static prompting.Static descriptions with random camera poses can cause semantic redundancy and duplicate objects, limiting object diversity.
- Ablations: 3D-aware sequential inpainting anchors novel views to existing geometry, while omitting it degrades prompt alignment and scene coherence.The process renders an intermediate 3D representation from a new camera pose and inpaints unobserved regions using localized prompts.
5 Conclusion
The paper concludes that a multi-agent framework can extract implicit 3D knowledge from 2D foundation models, producing expansive, robust, 3D-consistent scenes that support novel-view rendering.
- The framework combines a VLM director, image generator, and two-step verifier operating across 2D and 3D spaces.
- The evaluations demonstrate that carefully framing the generation process can extract implicit 3D knowledge from 2D foundation models.
- The approach yields expansive, robust, and 3D-consistent scene reconstructions that support novel-view rendering.
6 Appendix
The appendix documents implementation choices, additional qualitative results, and system prompts governing scene expansion, spatial continuity, and strict 3D-consistency verification.
- Appendix overview: The supplementary document covers implementation details, extended experimental results, and system prompts for the VLM agents.It organizes these materials into Sections 6.1, 6.2, and 6.3.
- Implementation details: The method synthesizes missing regions by re-rendering target areas with AnySplat and supplying black-masked novel views to image models without native mask inputs.This embeds the spatial mask directly into the RGB input for inpainting.
- Extended results: Additional results include indoor environments, crystal caves with underground lakes, and a medieval stone crypt specified through text prompts.
- System prompts: The Director Agent expands a single room by identifying blind spots, preserving architectural boundaries, and inventing matching furniture and decor for newly revealed areas.
- System prompts: Spatial continuity uses approximately equal known and unknown regions plus an existing-object edge as a visual anchor for each new view.
- System prompts: The verifier rejects frames with camera, redundancy, outpainting, tie-point, or other 3D-consistency failures before reconstruction.Its red-flag criteria include mismatched outpainting, abrupt seams, and geometry popping.