Wonderland: Navigating 3D Scenes from a Single Image

1. Introduction

• NeRF and 3D Gaussian Splatting approaches require dense multi-view images and time-consuming per-scene optimization, limiting scalability and flexibility.Source: §1 · p. 2
• Single- or sparse-view methods using image diffusion can produce distorted occluded geometry and blurry backgrounds, while sparse-point-cloud video methods remain narrow in scope.These methods also struggle to generate regions invisible from input views.Source: §1 · p. 2
• Transformer feed-forward reconstruction methods face memory demands from tokens for overlapping high-resolution views and are mostly limited to object-level or narrow-scope scenes.Source: §1 · p. 2
• Wonderland regresses 3D Gaussian Splatting scenes from video latents in a feed-forward manner, aiming to generate high-fidelity, wide-scope scenes from a single image.The paper attributes the latent space’s 3D awareness to video diffusion priors that capture spatial relationships across views.Source: §1 · p. 2
• Video diffusion latents provide multi-view spatial relationships, while Wonderland’s dual-branch camera conditioning controls trajectories and LaLRM lifts the latents into 3D.Source: §1 · p. 2

3D Scene Generation

• Most prior 3D scene methods generate novel views from sparse inputs and then optimize a 3D representation per scene, with warping and inpainting often introducing distortions and artifacts.Source: §2 · p. 2
• Image diffusion camera conditioning can leave backgrounds blurry, while video diffusion methods using global point clouds depend on point-cloud quality and remain limited to narrow-scoped scenes.Source: §2 · p. 3
• Wonderland integrates explicit camera control into video diffusion and reconstructs scenes directly from video latents in a feed-forward manner.Operating in latent space allows many views while keeping computation efficient.Source: §2 · p. 3

Sparse-View Reconstruction

• Feed-forward transformer methods regress 3D representations from sparse views, but computational overhead and view-count limits often confine them to narrower tasks such as object reconstruction.Source: §2 · p. 3

3. Method

• Wonderland first generates wide-scope, camera-guided video latents, then lifts them to 3D Gaussian Splatting with LaLRM in a feed-forward, memory-efficient process.Source: §3 · p. 3

3.1. Preliminaries

• Video diffusion models operate on compressed latents, with the diffusion transformer predicting noise from a noisy latent and conditional signal.The source video is encoded into a latent representation before diffusion processing.Source: §3.1 · pp. 3–4

3.2. Camera-Guided Video Latent Generation

• Dual-branch camera conditioning adapts video diffusion to precise camera trajectories and static-scene generation without changing the pretrained model’s original weights.The ControlNet-inspired branch integrates camera conditions, while the LoRA-inspired branch supports efficient fine-tuning.Source: §3.2 · p. 4

3.3. Latent Large Reconstruction Model

• The video latent compresses input while retaining perceptual detail, and its web-scale pretraining supports out-of-domain scene generalization.These properties allow LaLRM to reconstruct high-quality, wide-scope, generic scenes under memory constraints.Source: §3.3 · p. 5
• LaLRM regresses pixel-aligned Gaussian primitives from video latents and camera poses, using rendered views and a weighted image-reconstruction loss for supervision.Source: §3.3 · p. 5
• Progressive training begins with lower-resolution benchmark videos and seen/unseen-view supervision, then adds higher resolutions and generated out-of-domain videos.Generated video frames supervise reconstructions from their latents and conditioned camera poses.Source: §3.3 · p. 5

4.1. Camera-Guided Video Generation

• On RealEstate10K, DL3DV, and Tanks and Temples, our method outperforms video-generation baselines across visual-quality, camera-precision, and visual-similarity metrics.The evaluation reports FID/FVD, Rerr/Terr, and LPIPS/PSNR/SSIM.Source: Table 1 · p. 6, §4.1 · p. 7

4.2. 3D Scene Generation

• Our method outperforms all 3D scene-generation baselines by a large margin across multiple datasets, producing high-fidelity, geometrically consistent scenes from single views.Table 2 reports the quantitative comparison; qualitative results include detail preservation in visible and unseen regions.Source: §4.2 · p. 8, Table 2 · p. 8
• For in-the-wild scenes, our method maintains realism and 3D consistency at wide scope, while ViewCrafter is limited in range and WonderJourney is blurry and artifact-prone.WonderJourney comparisons are limited to zooming away from the input image.Source: §4.2 · p. 8, Figure 5 · p. 8
• At viewpoints near the input, our rendering quality is similar to Cat3D and better than ZeroNVS; farther away, our scenes retain sharper details while Cat3D blurs, especially in backgrounds.The comparison uses Mip-NeRF scenes rendered at the input view and about 120° rotation.Source: §4.2 · p. 8, Figure 6 · p. 9
• Around 5 minutes, our scene generation is 3.2× faster than Cat3D, 1.2× faster than ViewCrafter, and 36× faster than ZeroNVS.Latency is reported end-to-end on a single NVIDIA A100 GPU.Source: §4.2 · pp. 8–9

4.3. Analysis of the Architectural Design

• LaLRM with a frozen 3DVAE encoder performs best among the tested RGB and latent reconstruction designs; jointly finetuning the encoder degrades performance.The designs were evaluated on RE10K and Tanks test samples.Source: §4.3 · p. 9, Table 3 · p. 9

5. Conclusions

• Wonderland uses camera-conditioned video diffusion and compact video latents for feed-forward 3D Gaussian Splatting reconstruction, addressing multi-view data and per-scene optimization requirements.Its dual-branch camera conditioning supports precise pose control and static novel-view synthesis.Source: §5 · p. 9

• Table A1 lists the mathematical notation used in the paper.Source: Table A1 · p. 15

A.1. Effect of LoRA on Static Scene Generation and Camera Controllability

• LoRA is a lightweight, cost-effective fine-tuning approach compatible with pretrained models that leaves their original weights unchanged.The paper also notes minimal computational demands and reduced risk of overfitting to customized datasets.Source: §A.1 · p. 16
• LoRA fine-tuning on static-scene datasets yields more varied generations, especially with humans and animals, without compromising visual quality.The comparison used 20 in-the-wild prompts without pose control.Source: §A.1 · p. 16
• Removing LoRA from the main branch causes a noticeable drop in visual quality and camera-guidance precision.The ablation compares “Dual w/o LoraModule” with “Dual-branch” on RealEstate10K clips.Source: §A.1 · p. 16

A.2. Analysis of the ControlNet-Branch Design

• The full model uses a trainable copy of the first 21 of the video model’s 42 transformer blocks for its ControlNet branch.The branch-only evaluation informed this design choice.Source: §A.2 · p. 16
• Weight copying substantially improves all metrics, particularly visual quality, across the tested ControlNet-branch configurations.The variants include a 21-block branch without weight copying and weight-copied branches with 1, 10, or 30 blocks.Source: §A.2 · p. 16
• Increasing the number of weight-copied blocks strengthens camera-pose guidance, while a single block provides weak controllability.The 21-block branch achieves pose controllability similar to the 30-block branch while maintaining high visual quality.Source: §A.2 · p. 16

B.1. Fine-Tuning With the In-the-Wild Dataset

• LoRA fine-tuning on static-scene datasets produces video generations with significantly more static scenes than the source model.Figure A1 compares the source model with the model fine-tuned using LoRA modules.Source: Figure A1 · p. 17
• LaLRM fine-tuned with in-the-wild data shows noticeably better rendering quality than LaLRM–, which excludes that data.The comparison includes benchmark datasets and 20 disjoint in-the-wild image prompts; Table A3 covers novel-view synthesis from a single-view condition.Source: §B.1 · pp. 16–17, Table A3 · p. 17, Figure A2 · p. 17

C. Additional Discussion about Related Work

• Prior scene-generation methods commonly generate novel views first and then optimize a 3D representation per scene.Early approaches use depth-based warping and diffusion inpainting, which can introduce artifacts and inconsistencies in occluded regions.Source: C. Additional Discussion about Related… · p. 17
• Image-diffusion approaches can yield blurry backgrounds or inconsistencies across views, while video-diffusion approaches can depend on point-cloud quality and remain narrow in scope.The discussion also notes that prior methods lack explicit pose control during generation.Source: C. Additional Discussion about Related… · pp. 17–18
• Wonderland integrates explicit camera control into video diffusion and reconstructs scenes from video latents, eliminating per-scene optimization.The paper presents this as a way to align generation and reconstruction and bridge image and 3D spaces through video latents.Source: C. Additional Discussion about Related… · p. 18

D. Implementation Details

• Table A1 lists the mathematical notation used in the paper.Source: D. Implementation Details · p. 18

D.1. Additional Architectural Details

• The framework combines a Camera-Guided Video Diffusion Model and LaLRM to generate 3D scenes conditioned on single images.The architecture aligns generation and reconstruction through the video latent space, bridging image and 3D spaces.Source: §D.1 · p. 18
• Camera encoders project camera embeddings into tokens that are concatenated or added element-wise to visual tokens before transformer processing.The camera-guidance architecture includes both a ControlNet branch and a LoRA branch.Source: §D.1 · p. 18
• LaLRM tokenizes video latents and Plücker camera embeddings, merges their tokens, and processes them with transformer blocks to regress 3D Gaussian features.The latent input may come from generated video during inference or source clips during training.Source: §D.1 · p. 18
• A latent decoding module produces a 12-channel Gaussian feature map with pixel-level correspondence to the source video in RGB space.The module uses a 3D deconvolution layer with upsampling strides specified for the latent and camera-token design.Source: §D.1 · p. 18

D.2. Training and Evaluation Details

• The video diffusion backbone generates 49 frames at 480 × 720 resolution, with a ControlNet branch initialized from 21 blocks and camera-LoRA rank 256.The model is trained for 40K steps with batch size 24 and learning rate 2 × 10^-5.Source: §D.2 · p. 18
• Video-generation evaluation uses FID and FVD for visual quality and temporal coherence, and rotation and translation errors for camera-guidance precision.Generated-video camera poses are obtained using COLMAP; the evaluation compares specified frame subsets across methods.Source: §D.2 · pp. 18, 20
• The camera-guidance architecture merges the conditional image with noisy video latents and integrates camera guidance through LoRA and ControlNet branches.The figure description notes omitted text tokens, diffusion-time and positional embeddings, and reshaping operations for simplicity.Source: Figure A3 · p. 19
• LaLRM uses 24 transformer blocks with hidden dimension 1,024 and a 3D deconvolution layer for latent decoding.The figure describes LaLRM as directly regressing 3D Gaussian splattings from video latents and camera embeddings.Source: §D.2 · p. 20, Figure A4 · p. 20
• Training progresses from 49 × 240 × 360 clips to 49 × 480 × 720 clips, with 48 supervision views split between seen and unseen views.The high-resolution stage uses 20K generated videos with image prompts to improve out-of-domain generalization; the stages run for 200K and 100K iterations, respectively.Source: §D.2 · p. 20
• Reconstruction evaluation renders the constructed 3DGS and assesses similarity on frames from the first 14 camera poses.The authors report empirically better results using COLMAP-derived poses from generated videos than using conditional camera poses directly.Source: §D.2 · p. 20

E. Limitations and Future Work

• Most inference compute is consumed by video generation, although the authors suggest parallel inference or more efficient denoising as possible mitigations.Source: E. Limitations and Future Work · p. 21
• The approach focuses on static scenes; occasional motion in generated videos hampers reconstruction, while extending the pipeline to dynamic scenes and 4D content is future work.Source: E. Limitations and Future Work · p. 21
• The authors state that addressing these limitations could improve the framework for broader applications and enhance performance.Source: E. Limitations and Future Work · p. 21