Source-linked AI summary
WonderJourney: Going from Anywhere to Everywhere
Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T. Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, Charles Herrmann
TL;DR
Existing perpetual view-generation work is limited in scene scope, motivating generation of long journeys through diverse, coherently connected 3D scenes from arbitrary image- or text-specified locations. WonderJourney combines LLM scene descriptions, text-driven point-cloud generation, and VLM validation in a modular pipeline. The paper reports compelling diverse journeys across scene types and styles, with strong user preference over both baselines on all four evaluated axes.
Problem
Prior perpetual view-generation methods focus on a single type of scene, whereas the paper studies long sequences of diverse, coherently connected 3D scenes from arbitrary image- or language-specified starting points.
Method
WonderJourney uses an LLM for next-scene descriptions, a text-driven visual module for colored point-cloud scenes, and a VLM to verify and regenerate undesired results.
Results
WonderJourney produces compelling, diverse journeys across scene types and styles and is strongly preferred over InfiniteNature-Zero and SceneScape on all four user-study axes.
Takeaways & Limitations
The framework enables users to journey through diverse, coherently connected 3D scenes starting from their own image- or text-described locations.
Abstract
from arXiv · showhide
We introduce WonderJourney, a modularized framework for perpetual 3D scene generation. Unlike prior work on view generation that focuses on a single type of scenes, we start at any user-provided location (by a text description or an image) and generate a journey through a long sequence of diverse yet coherently connected 3D scenes. We leverage an LLM to generate textual descriptions of the scenes in this journey, a text-driven point cloud generation pipeline to make a compelling and coherent sequence of 3D scenes, and a large VLM to verify the generated scenes. We show compelling, diverse visual results across various scene types and styles, forming imaginary "wonderjourneys". Project website: https://kovenyu.com/WonderJourney/
1. Introduction
WonderJourney frames perpetual 3D scene generation as creating long journeys of diverse, coherently connected scenes from an arbitrary image or language-specified starting point. Its modular solution combines language reasoning, visual 3D generation, and visual validation to address semantic, spatial, and geometric challenges.
- Problem: Perpetual 3D scene generation creates diverse 3D scenes from an arbitrary image or language description, connected along a long-range camera trajectory.The resulting sequence can render as a fly-through video through varied plausible places.
- Challenges: The task requires deciding which objects belong in each scene, where to place them, and how successive scenes connect geometrically.These requirements involve semantic understanding, visual common sense, and geometric effects such as occlusion, disocclusion, and parallax.
- Approach: WonderJourney uses an LLM for scene descriptions, a text-driven visual module for connected colored point clouds, and a VLM to verify generations.The LLM supplies semantic and commonsense reasoning, while the vision module supplies visual and geometric understanding.
- Contributions: The framework decomposes perpetual 3D scene generation into scene description, coherent visual generation, and visual verification modules.Its modular design supports using pretrained models and triggering regeneration when undesired visual effects are detected.
- Results: WonderJourney produces interesting and varied journeys in a user study comparing it with SceneScape and InfiniteNature-Zero.The study is presented alongside visual results across diverse scenes and styles.
2. Related Work
Prior perpetual-generation methods typically remain within one scene domain or a single scene, while related video approaches connect separate clips with hard cuts. WonderJourney is presented as a modular, training-free workflow spanning scene description, visual generation, and validation.
- WonderJourney: The framework is described as training-free, allowing its modules to benefit from future advances in vision and language models.This property is stated in the framework figure caption.
- Perpetual view generation: Perpetual view-generation methods synthesize navigation through images or autoregressively generate views, but commonly target nature scenes or a single cave-like scene.Examples include InfiniteNature-Zero for nature scenes and SceneScape for a single text-driven cave-like scene.
- 3D scene generation: Text-to-3D research commonly generates objects or single background-containing scenes rather than long sequences of connected scenes.Existing representations include NeRFs and textured 3D meshes.
- Text-guided video generation: Text-guided video methods can create story-following clips, but separate scenes appear as different clips with hard cuts.Perpetual 3D scene generation instead targets coherently connected scene sequences.
3. Approach
WonderJourney generates successive 3D scenes by combining autoregressive textual scene descriptions with a text-guided visual module that maintains geometric and semantic continuity. Its modular pipeline renders, outpaints, lifts images into point clouds, refines depth, and addresses geometric artifacts before extending the journey.
- Visual scene generation: The design uses modular parametric models and operations, allowing individual components to benefit from advances in the corresponding research topics.The visual scene generation module is illustrated as a sequence of model-based and rendering operations.
- Scene description generation: The framework predicts each next-scene description from the task specification and the memory of past and current scene descriptions.The description generator is an LLM, and the memory is updated by adding the predicted scene description.
- Scene description generation: Each scene description contains a consistent style, scene objects, and a concise background caption, with lexical filtering retaining nouns and adjectives.The paper reports more coherent connections when natural-language descriptions are filtered rather than directly structured by the LLM.
- Visual scene generation: The visual generator conditions on the current scene image and next-scene description to produce a colored point cloud representing the next 3D scene.It combines learning-free operations, including rendering and perspective unprojection, with pretrained vision models.
- Visual scene generation: Depth and geometry corrections address sky-boundary errors, distant-pixel unreliability, disparity-based refinement, depth misalignment, and disocclusion inconsistencies.The pipeline removes points along sky boundaries, applies a far background depth cutoff, aligns new depths, and re-renders to detect inconsistent pixels.
- Visual scene generation: The module renders a partial future view, outpaints it with the next-scene description, and lifts the resulting image into 3D using estimated and refined depth.The outpainting equation is I_i+1 = g_outpaint(I_hat_i+1, S_i+1), and the new points are obtained by unprojecting outpainted pixels.
4. Experiments
Experiments evaluate WonderJourney through qualitative demonstrations, baseline comparisons, and human preference judgments across four dimensions. The results show diverse, coherently connected journeys and stronger preference than both baselines.
- Dataset and baselines: The evaluation uses a newly assembled mixture of self-taken, copyright-free, and generated photos because perpetual 3D scene generation lacks an existing dataset.DALL-E 3 performs text-to-image pairing, and InfiniteNature-Zero and SceneScape serve as image-based and text-based baselines.
- Qualitative demonstration: WonderJourney generates diverse, coherently connected scenes from varied input images and can reach different destinations across runs.The qualitative demonstrations cover different scenes and styles, including diverse journeys from the same input.
- Human preference evaluation: Human preference is measured on diversity, visual quality, scene complexity, and overall interestingness using videos generated with each approach's camera setup.InfiniteNature-Zero is compared only on photorealistic nature inputs, while SceneScape is compared using three styles.
- Human preference evaluation: WonderJourney is strongly preferred over both baselines on all four evaluated axes.Table 1 reports human preference across diversity, visual quality, scene complexity, and overall interestingness.
- Human preference evaluation: InfiniteNature-Zero remains limited to nature scenes, whereas SceneScape tends toward cave-like outputs and limited object generation, while WonderJourney connects more varied scenes and objects.These baseline characteristics are identified as factors that might contribute to WonderJourney's greater user preference.
5. Conclusion
The paper concludes that WonderJourney generates long sequences of diverse, coherently connected 3D scenes from user-provided locations. It presents these journeys across varied scene types and styles as user-directed adventures.
- Conclusion: WonderJourney generates long sequences of diverse yet coherently connected 3D scenes starting at any user-provided location.The conclusion characterizes the system as supporting generated journeys through user-provided adventures.
- Supplementary material: The supplementary material includes additional qualitative results, longer journeys, controlled text-guided journeys, implementation details, ablations, and model details.The listed supplementary sections cover journeys of up to 30 scenes, user-provided descriptions, rendering and depth processing, ablations, and LLM/VLM details.
B. Additional Results
Additional results provide further examples of WonderJourney going from anywhere and going to everywhere.
- Additional results: Figures 7 and 8 show additional results for going from anywhere and going to everywhere, respectively.These figures extend the qualitative demonstrations beyond the main examples.
C. Longer “Wonderjourneys”
Longer wonderjourneys show that extending the sequence can include more diverse scenes while maintaining high visual quality.
- Longer wonderjourneys: Longer wonderjourneys allow more diverse scenes while retaining high visual quality.The examples in Figure 9 illustrate these longer journeys.
D. Controlled “Wonderjourneys”
WonderJourney can use user-provided text, including poems, haiku, and story abstracts, to control the generated wonderjourney.
- User-provided poems, haiku, and story abstracts can replace LLM-generated scene descriptions to control wonderjourneys.
- Controlled examples include classical Chinese poems, haiku, “Jabberwocky,” and literary story abstracts.
E. Details on Visual Scene Generation
The visual scene generation system processes depth, renders colored point clouds, and moves through scenes with designed camera paths.
- Depth processing: Depth processing corrects sky, distant-pixel, and near-object depth issues for MiDaS v3.1.Sky depth is set to 0.025, the background far plane F = 0.0015, and a depth shift of 0.0001 is added.
- Rendering: The renderer rasterizes normalized-device-coordinate points and composites up to 8 z-buffer points with disparity-based softmax weights.Higher disparity values assign greater weights to nearer points, while occluded points can blend at object boundaries for anti-aliasing.
- Qualitative results: The framework produces additional qualitative journeys from real photos and diverse journeys from the same input image.
- Camera paths: Camera paths use straight-line motion or rotation, with interpolation, height perturbations, and acceleration to smooth generated videos.Straight-line movement uses a backward translation of 0.0005; rotation uses 0.45 radians with translation of 0.0001.
- Qualitative results: Long wonderjourneys contain 30 generated scenes, although the displayed results show 15 scenes.
F. Ablation Study
Ablations show that sufficient outpainting space, VLM border validation, and depth processing are important for generating coherent scenes and preserving requested content.
- Outpainting white space: Insufficient outpainting space prevents new objects such as houses from appearing, even with identical descriptions and final camera locations.The authors attribute this empirically to outpainting models avoiding partial objects near image borders.
- Visual validation: VLM visual validation detects unwanted borders and triggers regeneration, avoiding disruptions that can propagate into subsequent scenes.
- Depth processing: Without the proposed depth processing, rendered partial images show strong distortions in object boundaries, skies, and distant regions.Examples include distortions affecting a tower, bird, and sky.
- LLM and VLM details: The framework uses GPT-4 to generate scene descriptions from a memory of past and current descriptions containing style, objects, and background.The prompt asks for sequentially interconnected scenes and entities adapted to fit them.
- LLM and VLM details: GPT-4V validates images by checking their four borders for thin photographic or painting frames.The validator is instructed to answer yes when uncertain, favoring rejection of potentially disruptive borders.
- LLM and VLM details: When GPT-4V fails for practical reasons, center-cropping a 560 × 560 image is a less effective fallback that can create partial foreground objects and floaters.
H. Details on Human Preference Evaluation
The human preference evaluation recruited participants through Prolific and presented an anonymized survey using Google Forms.
- Study setup: Participants were recruited through Prolific for the human preference evaluation.
- Study setup: The survey was presented through Google Forms and anonymized for both participants and the host.The paper provides anonymous survey links for comparisons with InfiniteNature-Zero and SceneScape.