Source-linked AI summary
SynCity: Training-Free Generation of 3D Worlds
Paul Engstler, Aleksandar Shtedritski, Iro Laina, Christian Rupprecht, Andrea Vedaldi
TL;DR
Generating large 3D worlds is difficult because 3D generative methods are mostly object-focused, while image-based scene methods struggle to maintain coherent 3D structure. SynCity combines pretrained 2D and 3D generators in a tile-based pipeline to produce detailed, coherent worlds with tile-level control.
Problem
3D generative methods are mostly limited to individual objects, while image-based approaches struggle to maintain coherent 3D structure across large scenes.
Source: §1 · p. 2Method
Results
SynCity produces diverse, high-quality, detailed 3D worlds that are stitched into coherent scenes and can be freely explored.
Source: §5 · p. 8Takeaways & Limitations
Worlds can scale to arbitrary grid sizes and support either a whole-world prompt or tile-specific instructions while maintaining thematic consistency.
Source: §5 · p. 8The method retains a tile structure, with square unit-basis tiles; future work could relax this using shifted and scaled tiles.
Source: §3.2 · p. 4, §5 · p. 9
Abstract
from arXiv · showhide
We address the challenge of generating 3D worlds from textual descriptions. We propose SynCity, a training- and optimization-free approach, which leverages the geometric precision of pre-trained 3D generative models and the artistic versatility of 2D image generators to create large, high-quality 3D spaces. While most 3D generative models are object-centric and cannot generate large-scale worlds, we show how 3D and 2D generators can be combined to generate ever-expanding scenes. Through a tile-based approach, we allow fine-grained control over the layout and the appearance of scenes. The world is generated tile-by-tile, and each new tile is generated within its world-context and then fused with the scene. SynCity generates compelling and immersive scenes that are rich in detail and diversity.
1. Introduction
SynCity combines 3D generative models with 2D image generators to build large 3D worlds tile by tile, using scene context and 3D blending to join tiles.
1. Introduction
- 3D scene creation remains laborious, and manual production of non-artistic content can consume human resources, talent, and creativity.Generative models may help automate these tasks.Source: §1 · pp. 1–2
- Image-based scene expansion can produce artistic, prompt-responsive scenes but struggles to maintain coherent 3D structure across large spaces.The cited discussion says users generally cannot walk more than a few steps into such scenes.Source: §1 · p. 2
- 3D scene-generation methods can produce large coherent spaces, but training them for scenes limits quality and diversity and forfeits strengths of pretrained 2D generators.Those 2D generators bring artistic quality and complex-prompt interpretation from pretraining on billions of images.Source: §1 · p. 2
- SynCity uses TRELLIS to generate 3D tiles from isometric image prompts, extending an object-level model's ability to reconstruct local compositions into world building.The paper notes that TRELLIS can reconstruct fairly complex local compositions of multiple objects.Source: §1 · p. 2
- SynCity stabilizes tile framing with Flux prompting, then uses neighboring-tile context for 2D appearance consistency and 3D blending for geometric consistency.Each new tile inpaints a missing region in an isometric scene view, and neighboring 3D representations are blended with the 3D generative model.Source: §1 · p. 2
2. Related Work
Related work covers novel-view synthesis, image-projection and procedural scene generation, and direct 3D scene generation. SynCity positions itself as leveraging object generators for scene generation.
Novel view synthesis for scenes
- Novel-view synthesis methods improve scene consistency and camera control, but semantic drift and object permanence remain challenges when transferring generated views into explicit 3D.Examples include SynSin [58], ZeroNVS, and GenWarp [45]; explicit representations such as NeRF [35] or Gaussians [18] [19] require resolving geometric conflicts.Source: §2 · p. 2
Image projection-based scene generation
- Image-projection methods expand scenes with 2D generators and fuse views in 3D, yet many produce geometric artifacts, small scenes, or limited movement.Text2Room [17] supports free exploration within indoor mesh bounds; WonderJourney builds separate representations between keyframes, while WonderWorld [65] builds a single scene with limited extent.Source: §2 · p. 3
Procedural scene generation
3D scene generation
- Direct 3D scene-generation methods support layout control or large-scale environments, but can restrict object variety, texturing, or scene type.BlockFusion [59] requires domain-specific 3D training data and produces untextured meshes; LT3SD [33] is trained for indoor scenes.Source: §2 · pp. 3–4
- Pretrained 3D object generators can produce diverse, high-fidelity objects, and the authors state that prior work had not leveraged them for scene generation.The cited object-generation work includes [26] [28] [31] [40] [41] [47] [53] [61] [70].Source: §2 · p. 4
3. Method
SynCity builds a 3D world tile by tile from an initial text prompt, using 2D generation for contextual tile images and 3D generation to reconstruct and blend them. Its method combines prompt engineering with off-the-shelf generators, without retraining.
3. Method
- The world is generated progressively on a square tile grid, with each tile reconstructed from an isometric image and the 3D tiles aligned and blended into a world.The pipeline expands the world prompt, generates contextual tile images, reconstructs tile geometry, then aligns and blends the reconstructions.Source: §3 · p. 4
3.1. Prompting the Language Model
- Tile-specific prompts and a shared world-style prompt can be written manually or generated by an LLM from a seed description.The prompts describe individual tiles and the overall visual style.Source: §3.1 · p. 4
3.2. Prompting the 2D Generator
- For regular 3D tiles, the method assumes a square unit basis viewed isometrically and uses a gray square base and cube-shaped mask to frame image generation without fine-tuning.The tile-specific and world-level prompts are combined, and the base image and mask constrain the inpainting output.Source: §3.2 · p. 4
- For later tiles, the rendered existing world provides image context, while trimming tall geometry avoids occluding the target tile and supports ground continuity.The mask is adjusted to leave already generated tiles uncovered.Source: §3.2 · p. 5
3.3. Prompting the 3D Generator
- To reconstruct a tile, the method extracts its foreground from the contextual image, adds a larger slab as a detectable 3D base, then validates and normalizes the reconstruction.Post-processing crops the added base, rescales the tile to unit size, and reorients it to match the 2D prompt.Source: §3.3 · pp. 5–6
3.4. 3D Blending
- Naive latent interpolation produces poor reconstructions; occupancy-guided, multi-view denoising upsamples the latents, while adjacent tiles are blended by denoising their stitched boundary region.The boundary is first predicted by 2D inpainting, then the 3D latent denoiser updates only the joined middle region while keeping the rest fixed.Source: §3.4 · pp. 6–7
4. Experiments
Experiments examine tile prompting and stitching choices, while qualitative results show generated worlds that can be explored along consistent, non-trivial trajectories.
4. Experiments
- The experiments use ChatGPT o3-mini-high to generate text prompts and Flux ControlNet as the 2D inpainter.Source: §4 · p. 7
4.1. Ablations
- Neither precise nor abstract prompts effectively steer Flux to generate a single large-scale scene with the desired layout control.Source: §4.1 · p. 7
- Removing neighboring-tile context and sampling tiles independently produces inconsistent relative object scales.Source: §4.1 · p. 7
- Without rebasing, TRELLIS tiles are on average neither perfectly square nor bounded by a faithfully reconstructed solid border.The reported geometry metrics include base area, squareness, and faithfully reconstructed border voxels.Source: §4.1 · p. 7, Table 2 · p. 7
- The proposed upsampling method outperforms naive interpolation across LPIPS, FID, KID, and SSIM; multiple conditioning frames further improve performance.The comparison samples 10 random views from each of 200 TRELLIS tiles.Source: Table 3 · p. 7, §4.1 · p. 8
- Without 3D blending, tile boundaries show discontinuities; after blending, previously obvious seams are well-blended.Source: §4.1 · p. 8, Figure 12 · p. 8
4.2. Qualitative Results
- Generated-world trajectories are consistent without semantic drift, and the spaces are large enough for non-trivial navigation.A skybox is added for visual effect.Source: §4.2 · p. 8, Figure 11 · p. 8
5. Conclusion
SynCity builds diverse, detailed 3D worlds by generating tiles progressively and stitching them into coherent scenes. It uses pretrained generators without retraining, while identifying tile structure and alignment as areas for future work.
- SynCity autoregressively generates tiles on a grid, using local context and 3D inpainting to stitch coherent scenes that can scale to arbitrary grid sizes.Tile-specific instructions allow control of individual tiles while maintaining overall thematic consistency.Source: §5 · p. 8
- The method demonstrates effective off-the-shelf generation using pretrained language, 2D, and 3D generators with carefully designed prompting, requiring no component retraining; available 3D scene-scale data may enable fine-tuning for further improvements.Source: §5 · p. 8
- Future work could relax the tile structure with random tile shifts and scales, using coarse-to-fine modeling to retain global coherence and fine-grained local detail.Source: §5 · p. 9
A.1. Language Model Prompting Details
The appendix describes LLM-assisted creation of tile prompts, using a global world description and tile-specific instructions arranged in JSON.
- A prompt can be written manually or generated by an LLM such as ChatGPT from a seed prompt.Source: §A.1 · p. 12
- The LLM is asked to produce expressive prompts for each tile in a grid, alongside a global prompt for context, lighting, time of day, and art style.Source: §A.1 · p. 12
- A JSON file provides the template for the tile instructions.Figure 13 shows an example file for a 2 × 2 world.Source: §A.1 · p. 12, Figure 13 · p. 12
A.2. 2D Prompting Details
A special leftmost-column case can lack context, so the method temporarily supplies a neighboring tile during inpainting; the appendix also illustrates tile-geometry validation.
- Tiles (0, y) with y > 0 form a special case handled separately from tiles generated using existing context, because their context has to be bootstrapped.Source: §A.2 · p. 12
- For those tiles, the previous tile is temporarily copied to position (−1, y) to provide scale and appearance context, then removed after inpainting.Source: §A.2 · p. 12
- The geometry-validation figure depicts an occupancy grid, a 2D slice showing object extent at a height, and an example 3D tile-base template.Source: Figure 14 · p. 12
A.3. 3D Geometry Validation Details
SynCity validates generated tiles using TRELLIS’s occupancy volume, first checking the tile footprint and then verifying that the added base was reconstructed.
A.3. 3D Geometry Validation Details
- TRELLIS’s 64³ binary occupancy volume captures rough tile geometry for geometric validation.It is produced in TRELLIS’s first stage, before the 3D Gaussian mixture.Source: §A.3 · p. 12
- A tile is rejected when its footprint area is below (R/2)^2 or its width-to-height ratio is below 1.The footprint is computed from active voxels at the base height; the checks enforce sufficient area and a square shape.Source: §A.3 · p. 12
- An adaptive base template is formed at the slice with the largest footprint, and tiles are rejected when less than 95% of that template is occupied.The template is constructed to match the input tile, and the added base is checked for faithful 3D reconstruction.Source: §A.3 · pp. 12–13
A.4. 3DGS Post-Processing Details
Post-processing crops each generated tile to its footprint, aligns its ground surface, and resolves its orientation before tiles are combined.
A.4. 3DGS Post-Processing Details
- Color changes across vertical slices identify four cuts around the tile, whose footprint is then cropped, recentered, and resized to standard tile dimensions.The cuts are determined by comparing each slice’s average color with the leftmost slice.Source: §A.4 · p. 13
- The average height of the tile’s four corners estimates its surface level so neighboring tiles can be vertically aligned.Source: §A.4 · p. 13
- Four possible 90-degree rotations are rendered, and the orientation with the lowest LPIPS loss against the input image is selected.The paper attributes the limited rotation ambiguity likely to TRELLIS’s axis-aligned synthetic-object training data.Source: §A.4 · p. 13
A.5. Ablation Details
Direct whole-scene generation with TRELLIS produces visually appealing structures but does not provide sufficient detail or reliable layout control. Additional results show tile-level control can produce diverse scenes and patterned transitions, alongside limitations involving cross-tile structures, heuristics, and appearance consistency.
A.5. Ablation Details
- Direct TRELLIS scene generation has limited detail and scalability, weak editability, and mostly ignores layout instructions.The figure contrasts generic conditioning prompts with explicit layout prompts; neither makes the approach readily extendable or controllable.Source: §A.5 · p. 13, Figure 15 · p. 13
- Adding the 2D prompting design and Flux ControlNet inpainting does not improve quality, and layout instructions are still mostly ignored.The authors report that direct Flux prompting for large-scale scenes was unsuccessful, motivating the tile-based approach.Source: §A.5 · p. 13, Figure 16 · p. 14
A.6. Additional Qualitative Results
- Tile-level control supports diverse scenes and patterned changes, including seasonal transitions and shifts between city-like and rural environments.Source: §A.6 · p. 14, Figure 20 · p. 18
A.7. Limitations
- Tiles remain individual units; structures spanning multiple tiles require harmonious cooperation between Flux and TRELLIS.Source: §A.7 · p. 14
- Heuristics for estimating ground height and removing the added base include fallback mechanisms but are not infallible.Source: §A.7 · p. 14
- TRELLIS may only vaguely follow the conditioning image’s appearance, particularly its color, so tile transitions may not look perfectly smooth.Source: §A.7 · p. 14