Source-linked AI summary
WonderWorld: Interactive 3D Scene Generation from a Single Image
Hong-Xing Yu, Haoyi Duan, Charles Herrmann, William T. Freeman, Jiajun Wu
TL;DR
Interactive 3D scene generation is constrained by slow multi-view synthesis, depth alignment, and representation optimization. WonderWorld uses FLAGS and guided depth diffusion to generate connected, diverse scenes in less than 10 seconds on a single A6000 GPU, while remaining an interactive prototyping method because its geometry and object detail are limited.
Problem
Existing 3D scene-generation approaches are too slow for interactive, user-controlled construction because they generate multiple views and optimize scene representations.
Method
WonderWorld generates connected scenes from a single view using Fast Layered Gaussian Surfels and guided depth diffusion for geometry alignment.
Results
Less than 10 seconds on a single GPU enables WonderWorld to generate connected and diverse 3D scenes interactively.
Takeaways & Limitations
WonderWorld enables users to interactively create and explore diverse connected worlds for virtual reality, gaming, and creative design.
Takeaways & Limitations
Generated scenes contain only frontal-facing surfaces and detailed objects such as trees may produce holes or floaters, so WonderWorld is intended for interactive prototyping.
Abstract
from arXiv · showhide
We present WonderWorld, a novel framework for interactive 3D scene generation that enables users to interactively specify scene contents and layout and see the created scenes in low latency. The major challenge lies in achieving fast generation of 3D scenes. Existing scene generation approaches fall short of speed as they often require (1) progressively generating many views and depth maps, and (2) time-consuming optimization of the scene geometry representations. We introduce the Fast Layered Gaussian Surfels (FLAGS) as our scene representation and an algorithm to generate it from a single view. Our approach does not need multiple views, and it leverages a geometry-based initialization that significantly reduces optimization time. Another challenge is generating coherent geometry that allows all scenes to be connected. We introduce the guided depth diffusion that allows partial conditioning of depth estimation. WonderWorld generates connected and diverse 3D scenes in less than 10 seconds on a single A6000 GPU, enabling real-time user interaction and exploration. We demonstrate the potential of WonderWorld for user-driven content creation and exploration in virtual environments. We release full code and software for reproducibility. Project website: https://kovenyu.com/WonderWorld/.
1. Introduction
WonderWorld targets interactive 3D scene generation by replacing slow offline pipelines with low-latency, user-controlled generation of connected worlds. It combines FLAGS, single-view generation, and guided depth diffusion to accelerate creation and reduce seams between scenes.
- Motivation: Existing methods are offline, often taking tens of minutes to hours before returning fixed 3D scenes or videos.This limits iterative control over scene contents and layouts in applications such as game development.
- Motivation: Progressively generating dense multi-view images and optimizing scene representations are the main speed bottlenecks.Depth-map inaccuracies also create geometric distortion and seams at scene boundaries.
- Framework: WonderWorld accepts a starting image plus online camera and content controls, producing coherently connected 3D scenes that form a larger world.The framework is designed around interactive user-driven scene construction.
- Core techniques: FLAGS provides a fast layered Gaussian-surfel representation and a single-view generation algorithm, while guided depth diffusion improves alignment with existing geometry.Together, these techniques address generation speed and boundary distortion.
- Scope and demonstrations: WonderWorld supports interactive creation of diverse, connected scenes and demonstrates applications across nature, city, and campus environments.The paper evaluates user-driven generation and exploration across varied scene types.
2. Related Work
Prior work constructs novel views or extends scenes, but typically remains limited in viewpoint range, domain, consistency, or speed. WonderWorld positions FLAGS as a geometry-initialized representation for faster optimization and interactive connected-world generation.
- Novel view generation: Single-image novel-view methods use layered depth images, radiance fields, multi-plane images, or point features but support only small viewpoint changes.These methods build a single static representation that does not extend beyond the input image.
- 3D world generation: Extended-scene methods generate larger viewpoint changes or connected scenes, including perpetual videos, triplanes, and cloud representations.The related approaches span image stitching, nature-specialized generation, and expandable scene representations.
- 3D world generation: WonderJourney also generates diverse scene sequences but runs offline and requires tens of minutes per scene because it synthesizes dense views.Other large-scale approaches focus primarily on cities, while many scene-generation methods target a single local area.
- Fast 3D scene representations: Existing approaches are promising but do not guarantee 3D consistency and remain too slow for interactive use.WonderWorld uses surfels with principled geometry-based initialization to speed scene-representation optimization.
3. Approach
WonderWorld generates connected 3D scenes interactively from a single image and user controls. It combines layered Gaussian surfels, geometry-based initialization, and guided depth diffusion to support fast generation and coherent scene connections.
- Framework overview: WonderWorld generates diverse, coherently connected 3D scenes from an initial image, camera movements, and text prompts.Users control where to generate through camera movement and what to generate through prompts.
- Framework overview: The outer loop alternates between generating a scene image and generating FLAGS from that image.New scenes may extrapolate existing scenes or remain standalone for later connection.
- Fast Layered Gaussian Surfels (FLAGS): FLAGS represents each scene with foreground, background, and sky radiance-field layers composed of surfels.Each surfel carries position, orientation, scale, opacity, and color parameters; unlike traditional surfels, it carries a small radiance field.
- Fast Layered Gaussian Surfels (FLAGS): Single-view layer generation uses text-guided diffusion and inpainting to produce masked foreground, background, and sky layer images for surfel construction.The background is inpainted beneath the foreground, while valid pixels and masks determine the layer content used for surfel generation.
- Fast Layered Gaussian Surfels (FLAGS): Geometry-based initialization aligns surfels with image pixels and sets their scales from depth, focal length, and surface orientation.The sampling interval is TN = d/(f cos θ), and the resulting initialization provides visible-surface coverage before optimization.
- Guided Depth Diffusion: Guided depth diffusion conditions depth estimation on the scene image and partially visible existing depth to produce smoother geometry extrapolation.The method modifies an off-the-shelf latent depth diffusion denoiser with a guidance term based on visible depth and its mask.
4. Experiments
Experiments compare WonderWorld with representative scene-generation baselines using standardized camera paths, qualitative judgments, semantic metrics, and speed measurements. Results indicate faster generation, stronger novel-view quality, and improved coherence, while ablations examine key design choices.
- Baselines: WonderJourney, LucidDreamer, and Text2Room serve as representative baselines using point clouds, 3DGS, and meshes, respectively.Their official implementations are used for comparison.
- Evaluation setup: 28 scenes across four real and synthetic test examples are evaluated using a fixed panoramic camera path shared across methods.The examples span city, campus, nature, and fantasy scenes.
- Qualitative and quantitative comparison: WonderWorld is overwhelmingly preferred in human 2AFC comparisons and significantly outperforms other approaches in CLIP score and CLIP consistency.These evaluations target semantic alignment and novel-view consistency.
- Scene quality and diversity: Baseline methods show limited extrapolation or severe boundary distortion, while WonderWorld produces coherent large-scale 3D scenes and supports diverse contents and styles.Text2Room is reported not to generalize to outdoor scenes, while WonderWorld supports varied scene types and user-selected styles.
- Ablations: Ablations separately study geometry-based initialization, the layered design, and guided depth diffusion under the same baseline-comparison protocol.Quantitative ablation results are reported in Table 4, with corresponding novel-view examples in Figures 6–8.
5. Conclusion
WonderWorld is presented as a fast system for interactively generating and exploring large, diverse 3D scenes. Its scope is bounded by limited view synthesis and difficulty modeling detailed objects, motivating its role as a prototyping method.
- Conclusion: WonderWorld is introduced as a system for interactive 3D scene generation featuring fast generation of large, diverse scenes.Users can generate and explore requested parts of a scene interactively.
- Limitations: Frontal-facing surfaces limit view synthesis to areas around the camera because object backsides are not generated.The authors suggest separately generating objects as a future direction.
- Conclusion: Ablations evaluate geometry-based initialization, the layered design, and guided depth diffusion for novel-view rendering.The study includes rendered novel views and quantitative results in Table 4.
- Limitations: Detailed objects such as trees can leave holes or floaters when viewpoints change, so WonderWorld is positioned as an interactive 3D world prototyping method rather than a full end-to-end solution.Slower, higher-fidelity models are proposed for later refinement.
A. Overview
The supplementary material documents the algorithms, guided depth diffusion, implementation details, and additional experiments, and points readers to videos and interactive viewing examples.
- Supplementary contents: The supplementary material contains WonderWorld algorithms, guided depth diffusion details, further implementation details, and additional experiment results.It also compiles video results and interactive viewing examples on the project website.
B. Algorithms
The supplementary material summarizes WonderWorld’s control loop and the procedures for generating FLAGS.
- Algorithms: Algorithm 1 summarizes the WonderWorld control loop, while Algorithms 2 and 3 summarize FLAGS generation.
C. Details on Guided Depth Diffusion
Guided depth diffusion is designed to align newly estimated depth with existing scene geometry while reducing guidance overhead and accommodating targeted depth constraints.
- Accelerated depth guidance implementation: Guided depth diffusion applies guidance only during the last 8 of 30 Euler denoising steps to reduce runtime latency.The guidance weights make the guidance-signal norm proportional to the predicted-update norm.
- Relation to other guidance methods: Unlike semantic diffusion guidance, guided depth diffusion targets consistent depth estimation aligned with existing geometry.The method controls depth alignment rather than the semantic contents of generated images.
- Tackling ground plane distortion: The formulation supports depth constraints such as correcting curved ground planes caused by inaccurate camera intrinsics and depth estimates.
D. Further Experiment Details
The supplementary details describe the data, prompting, control loop, layered FLAGS generation, depth-guided optimization, rendering, and evaluation protocols used by WonderWorld.
- Input data: Experiments use both real photographs and synthetic stylized images, with multiple real-photo examples documented across the project website and figures.
- Language-model prompting: GPT-4 generates structured scene descriptions, while GPT-4V captions the initial image and supplies a style prompt retained throughout generation.The structured description includes a scene name, three entities, and a background prompt.
- Implementation settings: All generated scene images are 512 × 512 pixels, with fx = fy = 960 pixels used as the camera focal length.Estimated depth is post-processed using an efficient SAM-based procedure.
- Interactive control loop: The control loop renders the current FLAGS world continuously while user camera updates and asynchronous scene generation proceed in parallel.New scene images are outpainted from empty rendered pixels before FLAGS layers are generated and added to the world.
- FLAGS generation: FLAGS generation creates sky, background, and foreground layers, progressively freezing each optimized layer into the existing scene set.Layer optimization uses guided depth diffusion, normal estimation, pixel unprojection, scale computation, initialization, and subsequent optimization.
- Human study: The human preference study recruits 204 participants per comparison and presents randomized image pairs across 12 questions.The evaluation compares WonderWorld with three baselines using examples from Figures 5 and 15.
E. Additional Results
Additional results examine automated and qualitative generation, including baseline comparisons, depth-alignment ablations, multiple worlds from one image, and style variation within a world.
- Additional comparisons: Additional baseline comparisons are reported in Figure 15, alongside qualitative examples in Figures 10, 11, and 12.The automated examples use panoramic camera paths and LLM-generated scene names.
- Depth-alignment ablation: Guided depth diffusion provides much better depth alignment than both no alignment treatment and the heuristic Shift+Scale method.Alignment is measured with scale-invariant root mean square error between estimated depth and visible existing depth.
- Runtime analysis: A time analysis for generating a single extrapolated scene is provided in Table 5.
- Qualitative examples: Each generated world in the qualitative examples consists of 9 scenes whose text prompts are generated by an LLM.
- Diversity and style control: WonderWorld generates different virtual worlds from the same input image and supports different styles within one generated world.Shown styles include Minecraft, painting, and Lego.