Source-linked AI summary
Extend3D: Town-Scale 3D Generation
Seungwoo Yoon, Jinmo Kim, Jaesik Park
TL;DR
Large-scale 3D scene generation is limited by object-centric models and fixed latent sizes. Extend3D expands the latent space, couples overlapping patches, and uses point-cloud initialization, SDEdit, and 3D-aware latent optimization; experiments report better geometry, appearance, completeness, and image faithfulness than prior methods, with some completion, memory, and street-image limitations.
Problem
Object-centric 3D models and fixed-size latent spaces make large, detailed scene generation challenging.
Method
Extend3D expands a pretrained latent space, jointly denoises overlapping patches, initializes structure from monocular-depth point clouds, and optimizes latents with SDEdit and 3D-aware objectives.
Results
Extend3D outperforms prior methods in human-preference and quantitative experiments across geometry, appearance, completeness, and image faithfulness.
Takeaways & Limitations
The training-free pipeline enables scalable, generalizable image-guided 3D scene generation using an object-centric model.
Takeaways & Limitations
Occluded-region completion can remain incomplete, SLAT optimization requires substantial memory for large scenes, and performance is limited on street-level images.
Abstract
from arXiv · showhide
In this paper, we propose Extend3D, a training-free pipeline for 3D scene generation from a single image, built upon an object-centric 3D generative model. To overcome the limitations of fixed-size latent spaces in object-centric models for representing wide scenes, we extend the latent space in the $x$ and $y$ directions. Then, by dividing the extended latent space into overlapping patches, we apply the object-centric 3D generative model to each patch and couple them at each time step. Since patch-wise 3D generation with image conditioning requires strict spatial alignment between image and latent patches, we initialize the scene using a point cloud prior from a monocular depth estimator and iteratively refine occluded regions through SDEdit. We discovered that treating the incompleteness of 3D structure as noise during 3D refinement enables 3D completion via a concept, which we term under-noising. Furthermore, to address the sub-optimality of object-centric models for sub-scene generation, we optimize the extended latent during denoising, ensuring that the denoising trajectories remain consistent with the sub-scene dynamics. To this end, we introduce 3D-aware optimization objectives for improved geometric structure and texture fidelity. We demonstrate that our method yields better results than prior methods, as evidenced by human preference and quantitative experiments.
1. Introduction
Extend3D addresses the difficulty of generating large, detailed 3D scenes with object-centric models by expanding their latent space and coordinating overlapping patch-wise generation. It uses image- and point-cloud-based initialization and optimization, and reports improved geometry, appearance, completeness, and image faithfulness.
- Motivation: Large-scale 3D scene generation remains difficult because datasets and models are predominantly object-centric, while fixed latent sizes limit detail.Prior outpainting methods can also degrade consistency and create visible seams in large scenes.
- Method: Extend3D expands a pretrained object-centric model’s latent space and generates overlapping patches simultaneously for scalable, detailed scene synthesis.Neighboring patches influence one another to improve 3D reconstruction and avoid the limitations of separate outpainting.
- Method: The pipeline initializes structure from a monocular-depth point cloud, refines occluded regions with under-noised SDEdit, and optimizes latents using 3D-aware objectives.These priors address image–3D alignment and the object-centrality of pretrained models.
- Results: Human-preference, quantitative, and ablation experiments show improved geometry, appearance, completeness, and image faithfulness, with overlapping flow, initialization, and optimization identified as crucial.The method is also reported as scalable and generalizable.
2. Related Work
Related work includes object-centric latent-flow models, trained scene-generation systems, and training-free pipelines. Extend3D differs by simultaneously coupling overlapping sub-scenes and optimizing latent trajectories with scene priors.
- Object-centric 3D generation: Modern 3D latent-flow models generate high-quality objects but use fixed-size voxel or set-based latents and object-centric training data.These choices limit direct representation of large scenes with multiple objects and backgrounds.
- Scene-generation models: Trained scene-generation methods use cropped sub-scenes, multiscale diffusion, latent trees, or autoregressive representations, but limited datasets reduce category coverage and detailed image conditioning.Extend3D instead targets general scenes with detailed image conditioning without additional training.
- Training-free generation: Training-free methods exploit object-centric models because 3D scene datasets are scarce, while sequential or separate sub-scene generation can produce inconsistencies and visible seams.SynCity attaches tiles sequentially, and 3DTown completes scenes patch by patch.
- Extend3D: Extend3D simultaneously generates overlapping sub-scenes with interacting denoising paths and optimizes the latent at each step to reduce geometric errors and path deviation.Small transitions between patches help capture local information during joint generation.
- Challenges: Naively extending latent generation can produce vanishing floors, repeated objects, noisy patches, and poor scene fidelity, motivating structural priors.The reported failure modes arise from the model’s image-conditioning, 3D, and object-centrality dynamics.
- Priors and refinement: Extend3D applies SDEdit to refine initialized structure, introduces under-noising for 3D completion, and optimizes intermediate latents with scene-specific losses.The optimization assumes priors provide ground-truth 3D structure and texture information.
3. Preliminaries
The preliminaries describe latent-flow generation for voxelized or set-based 3D representations and the SDEdit procedure used to edit an initialized latent. Extend3D organizes these components into sparse-structure and structured-latent generation stages.
- Latent flow models: Latent-flow models represent 3D space with fixed-size voxelized or set-based latents within a confined region, and the framework can apply to either representation.The paper illustrates the approach with Trellis.
- Trellis pipeline: Trellis encodes an image condition, first generates occupied sparse-structure coordinates, and then denoises a structured latent for 3D decoding.The overall pipeline has sparse structure generation followed by structured latent generation.
- Extend3D pipeline: Extend3D applies SDEdit and overlapping patch-wise denoising across both pipeline stages, with priors used to initialize structure and optimize vector fields.The figure identifies iterative SDEdit in initialization and prior-based optimization in both stages.
- Structured latents: SLAT pairs sparse coordinates with latent features and evolves them through a condition-dependent vector field before decoding to 3D Gaussians, radiance fields, or meshes.The notation uses Z_t and v for the relevant latent and vector-field quantities.
- SDEdit: SDEdit noises a guide latent to an intermediate time and then denoises it under an editing condition, preserving guidance while moving toward the intended distribution.Increasing t_start lengthens the denoising path and enlarges conditioning and generative effects.
4. Method
Extend3D expands an object-centric 3D latent, generates overlapping patches with coupled denoising, and uses depth-based initialization, under-noised SDEdit, and 3D-aware optimization to produce aligned, complete scenes.
- Extend3D expands a pre-trained object-centric 3D latent in the x and y directions to represent larger, more detailed scenes.
- Overlapping Patch-wise Flow: Overlapping latent patches are generated simultaneously with matched image patches, while merged vector fields couple their denoising dynamics through shared regions.Patch vector fields are inverse-mapped and averaged in overlaps; the resulting extended vector field drives the latent evolution.
- Initialize with Prior: A monocular-depth point cloud initializes the scene because direct denoising from Gaussian noise produces fragmented outputs and poor image–latent alignment.The point cloud is voxelized into an occupancy grid before refinement.
- Initialize with Prior: Under-noising makes the latent denoised more aggressively than it was noised, allowing the model to interpret missing or occluded structure as noise and fill it.SDEdit is applied iteratively because one pass can leave holes in the scene.
- Optimize with Prior: 3D-aware latent optimization preserves initialized voxels and refines rendered scene textures and patch boundaries during denoising.The sparse-structure loss prevents desired voxels from disappearing, while the rendering loss improves texture details and makes seams invisible.
5. Experiments
Experiments show that Extend3D scales to wide and diverse scenes while improving image faithfulness, geometry, appearance, completeness, and patch consistency over prior methods. Ablations identify overlapping patch-wise flow, prior initialization, under-noising, and latent optimization as important, with a geometry–texture trade-off for SLAT optimization.
- 5.1. Human Preference: Extend3D outperformed prior methods in human preference across geometry, faithfulness, appearance, and completeness.The study used 10 participants, 14 images, and 3D scenes, comparing Extend3D with Trellis, Hunyuan3D-2.1, and EvoScene.
- 5.2. Quantitative Results: Extend3D achieved the best LPIPS, SSIM, and PSNR scores on 100 wide-scene input images, indicating stronger structural and textural faithfulness.On 45 UrbanScene3D image–mesh pairs, it also surpassed previous methods on Chamfer Distance and F-score at threshold 0.05.
- 5.2. Quantitative Results: Extend3D surpassed SynCity in text compatibility, quality, and patch-wise consistency.The comparison used CLIP, HPSv3, and Intra-LPIPS, with Intra-LPIPS measuring LPIPS between patches within a scene.
- 5.3. Qualitative Results: The extended latent produced a 36× larger result than the original latent space while representing landmarks, small buildings, towns, food tables, studies, and indoor rooms.Compared with SynCity and EvoScene, the method produced scenes without patch boundaries, less distorted geometry, and more detailed textures.
- 5.4. Ablation Study: Increasing the division factor improved local detail because coupled overlapping patches mutually rectified errors and captured finer structure.With d = 2, local structure was distorted, whereas d = 4 avoided that distortion.
- 5.4. Ablation Study: Prior initialization was essential, under-noising naturally completed occluded regions, and prior-based optimization refined seams, distortion, structure, and texture.SLAT optimization improved texture but sometimes degraded geometry, revealing a geometry–texture trade-off; quantitative ablations were consistent with these observations.
6. Conclusion
Extend3D is a training-free pipeline for image-guided 3D scene generation that combines extended latent generation with patchwise flow, initialization, and optimization schemes. The authors report notable improvements in image-guided 3D scene generation, while identifying incomplete occlusion completion, memory demands, and limited street-level performance as limitations.
- Extend3D enables scalable 3D scene generation by extending the latent space of a pretrained object-centric model.
- Overlapping patchwise flow, initialization, optimization, iterative SDEdit, under-noising, and 3D-aware objectives improve image-guided 3D scene generation.
- Occluded-region completion is sometimes incomplete, SLAT optimization requires considerable memory, and street-level images show limited performance.
A.1. Algorithms
The algorithm takes an input image, estimates monocular depth, and uses the resulting point cloud for subsequent scene processing.
- MoGe2 estimates a point cloud P from the input image I before scene processing.
A.2. Image Patchification
Extend3D patchifies the input image according to point-cloud coordinates in the extended latent space, preserving spatial correspondence between image regions and 3D patches.
- Point-cloud coordinates extracted from monocular depth determine which extended-latent patch contains each image pixel.
- Each image patch collects pixels mapped to its 3D patch, blacks out other pixels, and crops the result to a square.
A.3. Dilated Sampling
The appendix describes patch construction, coupled sampling, computational trade-offs, iterative SDEdit behavior, and additional qualitative comparisons. These details show how Extend3D balances local refinement, scalability, scene completion, and resource demands.
- Dilated Sampling: Dilated sampling divides the extended latent into patches, samples pillars, and attaches them while preserving relative positions.It is used only for sparse structure generation because applying it to structured latent generation empirically worsens texture.
- Dilated Sampling: PatchWise and Dilated dynamics are blended with a time-dependent weight γ_t, using α = 5 in the experiments.Dilated sampling is restricted to sparse structure generation because it empirically worsens texture for structured latent generation.
- Computational Cost: Extend3D is heavier than object-centric methods but requires substantially less memory and time than other scene-level pipelines.
- Computational Cost: 28GB peak GPU memory is required for a = b = 2 and d = 4 with SLAT optimization, versus 14GB without it.For a = b = 6 and d = 4, peak memory is 61GB without SLAT optimization, while the authors report promising results without that optimization.
- Computational Cost: Inference complexity is O(abd2niter), creating a trade-off between output quality and inference time as scene detail and completeness increase.
- Ablation on Iterative SDEdit: A single under-noised SDEdit step noticeably improves geometry, while additional steps can degrade quality and trade geometric detail against scene completion.