Source-linked AI summary
SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion
Paul Engstler, Iro Laina, Christian Rupprecht, Andrea Vedaldi
TL;DR
Existing 3D generators primarily produce single objects, while scene-scale generation lacks sufficient 3D scene data for training. SynCity 3000 fine-tunes a 3D generator for convolutional inference using synthetic scene-like data and converts generated 2D templates into large 3D scenes. It produces coherent scenes of arbitrary structure and complexity, outperforming SynCity and other reconstruction methods qualitatively and quantitatively, although realistic replication of real-world scenes remains limited.
Problem
Existing 3D generators are usually limited to single objects, while the scarcity of large-scale 3D scene data complicates training generators for entire scenes.
Method
SynCity 3000 generates arbitrary-complexity 2D scene templates, then uses a TRELLIS-based 3D generator fine-tuned on synthetic scene-like data for convolutional inference into 3D Gaussian Splats.
Results
SynCity 3000 outperforms SynCity and other reconstruction methods qualitatively and quantitatively across scenes of different sizes and complexities, with an average multi-view plausibility rating of 3.57/5.
Takeaways & Limitations
The framework supports large-scale 3D scene generation with arbitrary structure and complexity while preserving semantic and visual coherence through joint scene-template generation.
Takeaways & Limitations
Because dimetric scenes and Objaverse-XL fine-tuning may amplify cartoonish biases, the method has limited ability to replicate real-world scenes realistically.
Abstract
from arXiv · showhide
We present SynCity 3000, a framework for generating 3D scenes that are globally coherent while enabling fine-grained layout control. Building on the ability of current image-to-3D generators to produce complex 3D assets from a single image, we extend this capability to the scale of entire scenes by adapting the generator to be applicable as a convolutional operator. We achieve this by fine-tuning the model on scene-like data generated by a new synthetic data engine, which we propose to address the scarcity of 3D scene data for training. The convolutional generator is then applied to a dimetric image of the entire scene, generated from the user prompt, resulting in 3D scenes of arbitrary size and complexity. Across diverse prompts and layouts, SynCity 3000 produces large, coherent, and detailed scenes, addressing the shortcomings of prior approaches to 3D scene generation.
1 Introduction
SynCity 3000 extends 3D generation from individual assets to large-scale scenes by first creating a coherent 2D template and then converting it into 3D. Its sliding-window 2D and 3D pipeline supports arbitrary scene structure while preserving layout control.
- Motivation: SynCity 3000 targets large-scale 3D scene generation, addressing prior methods’ restriction to single objects and SynCity’s visible grid-like structure.The framework is designed for applications including movies, games, and simulations.
- Approach: The framework first generates a high-resolution 2D scene template defining appearance and layout, then converts it into a corresponding 3D scene automatically.The two stages are independent, so template-like images created by graphic artists can also serve as inputs to the second stage.
- Approach: Overlapping latent windows and averaged conditioning contributions let the 2D generator create scenes of arbitrary extent while incorporating optional fine-grained layout constraints.The windows are not forced to resemble square tiles, and their outputs blend with surrounding regions.
- Approach: The 3D stage first generates a coarse voxelized world, enriches each voxel with visual and semantic features, and decodes the result into detailed 3D Gaussian Splats.Partially overlapping windows support scenes of arbitrary size, while fine-tuning adapts the 3D generator to sliding-window operation.
- Results: Across scenes with different sizes and complexities, SynCity 3000 is reported to outperform SynCity and other reconstruction methods qualitatively and quantitatively.The comparison is presented in Figures 6 and 7 and Table 1.
2 Related work
Prior scene-generation methods use image projection, asset assembly, or direct scene synthesis, but face artifacts, discontinuities, limited resolution, or demanding data requirements. SynCity 3000 addresses these issues with coherent overlapping-window diffusion and a fine-tuned convolutional 3D generator trained using synthetic scene-like data.
- Prior approaches: Image-based methods project generated images into 3D using depth estimation and spatial-fusion heuristics, while asset-based methods populate layouts with retrieved or generated objects.These families represent scenes as meshes, Neural Radiance Fields, or 3D Gaussian Splats, and asset layouts may come from images, language models, or diffusion models.
- Prior approaches: Explicit scene generators avoid some geometric-consistency issues but require large-scale, domain-specific training data and may have other generation limitations.LT3SD uses patch-by-patch coarse-to-fine synthesis, whereas X^3 uses hierarchical sparse voxel grids.
- Prior approaches: SynCity generates and fuses 3D tiles sequentially, producing globally themed scenes whose lack of overall coherence appears as a busy, grid-like structure.Its training-free use of TRELLIS leverages object-centric 3D datasets.
- Prior approaches: 3DTown follows an input scene image but suffers from geometric artifacts, holes, and limited resolution, while NuiScene exhibits chunk discontinuities and is constrained by a small dataset of 43 scenes.NuiScene also requires SceneTex for texture generation.
- SynCity 3000: SynCity 3000 strictly generalizes SynCity through an arbitrarily complex scene template, convolutional inference for a fine-tuned 3D model, and a synthetic engine that generates scene-like training data.The synthetic engine is intended to bypass the shortage of large-scale scene datasets and scale beyond NuiScene’s training data.
3 Method
SynCity 3000 converts text and optional local layout prompts into a 2D dimetric template, then transforms that template into a 3D Gaussian Splat scene through convolutional, overlapping-window diffusion. Independent control of template and latent resolutions supports arbitrary scene size and detail.
- Inputs and outputs: The method accepts global theme and style prompts, optionally augmented with local prompts associated with specific spatial windows.The output is a full 3D scene represented as Gaussian Splats and renderable from arbitrary viewpoints.
- Pipeline: Its two independent stages generate a whole-scene 2D template and then a corresponding 3D model.This separation allows the second stage to use any suitable template-like image.
- 2D template generation: The 2D generator is prompted in inpainting mode to produce a dimetric scene image, overcoming the off-the-shelf model’s lack of direct suitability for 3D reconstruction.A supporting base image and masking strategy guide the dimetric template.
- 2D template generation: Stage 1 uses latent diffusion on overlapping windows, averaging their estimated noise at every denoising step to jointly update a large latent canvas.Window-specific prompts provide local content, while the global style prompt conditions every window.
- 3D generation: Stage 2 adapts TRELLIS for convolutional inference: one generator produces a coarse voxel structure, another samples voxel-attached features encoding shape and appearance, and decoders produce Gaussian Splats.The scene is processed through corresponding overlapping latent and image windows.
- Fine-tuning: Fine-tuning uses objectives with and without surrounding context, applies masked conditioning, and restricts the mean-squared-error loss to the core region.The no-context objective mirrors TRELLIS’s original task.
- 3D generation: TRELLIS reconstructs a coarse occupancy grid into a lower-resolution latent grid, but its original denoiser must be modified because it is not convolutional for much wider scenes.Overlapping sub-grids are denoised in parallel and averaged to update the larger latent grid.
- Resolution control: Template and latent resolutions can be chosen independently, with 3D latent resolution set in integer multiples of core resolution to control scene size and detail.This establishes direct correspondence between template pixels and latent space.
4 Dataset
SynCity 3000 introduces a synthetic dataset engine for scene-scale training data, using randomized terrains and Objaverse-XL objects to create voxelized, feature-enriched scene crops.
- Dataset construction: The engine generates complete 3D scenes by placing randomly selected Objaverse-XL objects on randomly generated surfaces.The process is designed to produce scene-like data for fine-tuning scene-scale 3D generation models.
- Dataset construction: Scenes are rendered with an orthographic camera in dimetric projection to match the generated 2D templates.This aligns the synthetic training data with the representation used during scene-template generation.
- Processing and representation: The rendered core is paired with a 3D crop containing surrounding context, which is converted into a voxel grid.The crop includes the central core and additional contextual voxels for training.
- Processing and representation: Voxel grids are infused with DINOv2 features and encoded into sparse structures and structured latents using TRELLIS encoders.These representations provide the training targets for the fine-tuned 3D generation models.
- Dataset construction: The engine also generates 32 random top views, side views, and a bottom view for each scene.The technical details specify VF := 32 + 5 views in total.
5 Experiments
Experiments evaluate SynCity 3000 on layout control, template faithfulness, geometric reconstruction, and overall scene quality against prior methods. Results favor the proposed convolutional approach, particularly for flexible layouts and larger scenes.
- Experimental setup: 320k synthetic samples support fine-tuning, while experiments evaluate layout control, template faithfulness, reconstruction quality, and scene preference.The study combines user preferences with quantitative and qualitative comparisons.
- Layout control: Users unanimously prefer SynCity 3000’s flexible layout control over SynCity’s rigid, low-resolution grid structure.SynCity’s grid is effective for naturally grid-planned scenes but limits open layouts and structures spanning multiple tiles.
- Overall scene quality: SynCity 3000 produces richer, more vibrant, and more coherent scenes than SynCity under the same prompts.The comparison attributes the difference to the organic structure of the generated worlds and their visual and semantic coherence.
- Scene-scale reconstruction: SynCity 3000 maintains more consistent performance as scene size increases, while TRELLIS deteriorates with larger scenes.Geometric quality is evaluated on synthetic scenes at two scales using multiple metrics.
- Visual plausibility: Users rate the multi-view visual plausibility of generated scenes at an average of 3.57 on a 1-to-5 scale.The paper notes that this property is difficult to judge directly without ground-truth data.
- Ablations: Fine-tuning the TRELLIS models is central to convolutional inference, whereas off-the-shelf models suffer significant performance degradation.Without context, quality deteriorates slightly, and large strides without prediction averaging produce inferior reconstructions.
6 Conclusion
SynCity 3000 generates large-scale 3D scenes from text by jointly creating coherent templates and converting them through fine-tuned convolutional 3D inference.
- Conclusion: SynCity 3000 generates large-scale 3D scenes from scratch using shifted-window templates and a fine-tuned two-stage 3D generative model.The final scenes are represented as 3D Gaussian Splats.
- Conclusion: Synthetic scene-like data enables fine-tuning for convolutional inference while preserving high-quality predictions.The approach avoids brittle heuristics used by previous scene-generation methods.
- Conclusion: Joint diffusion of the complete scene template is used to maintain semantic and visual coherence.This directly addresses the stated limitations of SynCity.
Diffusion (Supplementary Materials)
The supplementary diffusion section is not represented by substantive diffusion passages in the supplied excerpts.
- Diffusion: No substantive diffusion methodology, result, or analysis is provided in the supplied passages for this section.
A Additional technical details
The pipeline uses separate template-generation and 3D-generation stages, with shifted-window processing and implementation choices for scalable inference. Additional details cover layout prompting, latent decoding, color correction, sampling, and runtime.
- Prompting and layout: LLM-generated prompts can specify an overall scene concept, stylistic theme, and spatial constraints for template generation.The instructions describe small worlds ranging from towns to park regions and allow detailed regional constraints.
- Prompting and layout: Template generation uses relative patch units, and these instructions apply only to the 2D template stage.The layout representation is separated from pixel coordinates and is not used directly in 3D generation.
- Latent encoding and decoding: The structured latent VAE and 3D Gaussian Splat decoder use shifted-window processing because they are transformer-based, while the sparse structure VAE is already fully convolutional.This enables the non-convolutional components to process scenes of varying size.
- Color correction: Color statistics are transferred from the template to the Gaussian Splats in L*a*b* space to compensate for reduced lightness in reconstructions.The correction transfers per-channel means and standard deviations but does not simplify the underlying color prediction problem.
- Inference: Inference uses guidance scale 7.5 with 50 steps for sparse structure and scale 3.0 with 12 steps for structured latents.The sparse-structure setting improves results but slows inference; both stages together take about 30 minutes for an average scene.
- Inference: Runtime and peak memory rise with template size: 31m/45.7G, 82m/48.4G, and 179m/55.8G for the three reported resolutions.The scaling is described as roughly quadratic, consistent with convolutional processing.
B Additional experimental details
The experiments evaluate reconstruction faithfulness and user-facing scene preference using varied generated templates and balanced scene selections. Failed reconstructions are excluded from the faithfulness comparison.
- Faithfulness evaluation: Faithfulness evaluation uses templates spanning six themes, including fortresses, mountains, solarpunk cities, suburban towns, theme parks, and university campuses.Seven seeds were used per theme, producing 42 templates before exclusions.
- Faithfulness evaluation: After TripoSG and Hunyuan3D-2.1 failed on seven scenes, the comparison retained 35 templates.The remaining templates were used to compare reconstruction faithfulness across methods.
- User study: The user study compares three scenes per prior work and balances grid-like and non-grid-like structures to reduce selection bias.The comparison also considered each method's capabilities and the availability of code and data.
C Limitations
SynCity 3000 improves prior approaches but retains limitations involving perspective, texture fidelity, interactivity, structural duplication, and realistic appearance.
- Forced perspective: The convolutional inference setup requires templates to present scenes in a dimetric perspective.Future work could seek an approach that avoids forcing a particular perspective.
- Texture detail: Averaging overlapping shifted-window outputs slightly reduces texture fidelity compared with the original TRELLIS model.The reduction follows from averaging structured latent predictions across windows.
- Degree of control: Users must wait for the template to be produced before judging whether it meets their needs.The paper suggests more interactive or real-time approaches as future work.
- Structure duplication: Tall or large structures under severe occlusion can be duplicated or partially broken because tiled templates must align reconstructed geometry.This failure mode is tied to the convolutional tiling design.
- Cartoonish appearance: Training biases from Objaverse-XL and FLUX contribute to a cartoonish appearance and limit realistic reconstruction of real-world scenes.The paper attributes these biases to stylized objects and the dimetric, video-game-like nature of scenes.
D Additional comparisons
Additional comparisons show that prior image-conditioned models often struggle with scene reconstruction, while SynCity 3000 more faithfully preserves details and scales to arbitrary scene sizes.
- Qualitative comparisons: TripoSG and Hunyuan3D-2.1 struggle to reconstruct scenes resembling the templates, with inaccurate colors and unreliable geometry that may contain holes.These qualitative failures accompany the reported faithfulness comparison.
- Qualitative comparisons: TRELLIS produces convincing reconstructions but has fuzzy details, limited output resolution, and occasional severe deviations from the template.SynCity 3000 more reliably preserves details such as car positions, orientations, and shapes.
- Comparison with prior models: SynCity 3000 achieves more faithful detailed reconstructions and scales to arbitrary scene sizes, whereas object-centric models remain limited in output size.The comparison attributes the size difference to the models' object-centric design.
E Additional qualitative results
SynCity 3000 produces detailed, complete scenes across varied sizes and viewpoints, while its indoor-scene plausibility remains limited by FLUX biases.
- SynCity 3000 generates scenes with arbitrary complexity, while increasing scene size improves detail fidelity through higher template resolution.The scenes in Fig. 4 use 3136×1568-pixel templates; TRELLIS loses detail at lower resolution.
- The method can generate scenes from templates not produced by its own pipeline, including samples demonstrated with the fine-tuned TRELLIS model.The synthetic dataset engine produces diverse samples independent of the scene-template generation process.
- Indoor-scene quality and plausibility are limited by biases in FLUX, despite the fine-tuning setup being agnostic to scene design.The limitation is illustrated by the indoor scenes in Fig. 3.
- Additional viewpoints show fully realized SynCity 3000 scenes without the holes observed in prior work.
- The dataset engine provides diverse rendered 3D samples used as conditioning images for qualitative comparisons.The samples are randomly selected and rendered with the black background removed for visualization.