PaperlayerView exact revision on arXiv

Source-linked AI summary

SynCity 3000: Bootstrapping Scene-Scale 3D Diffusion

Paul Engstler, Iro Laina, Christian Rupprecht, Andrea Vedaldi

arXiv:2607.05392v1cs.CV

TL;DR

Scene-scale 3D generation is constrained by scarce scene data, while recent 3D generators usually produce single objects. SynCity 3000 uses synthetic scene-like data to fine-tune convolutional 3D generation from scene templates of arbitrary structure and complexity. It outperforms SynCity and other reconstruction methods qualitatively and quantitatively across scenes of varied sizes and complexities.

  • Problem

    Scene-scale 3D generation faces scarce scene data, while recent 3D generators usually produce single objects.

    Source: §1 · p. 1, §4 · p. 9

  • Method

    SynCity 3000 creates scene templates of arbitrary structure and complexity, then uses convolutional inference with a fine-tuned 3D model trained on synthetic scene-like data.

    Source: §1 · p. 1, §2 · p. 4

  • Results

    SynCity 3000 outperforms SynCity and other reconstruction methods qualitatively and quantitatively across scenes of varied sizes and complexities.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that generating the complete scene as a joint template ensures semantic and visual coherence, directly addressing SynCity’s limitations.

    Source: §6 · p. 14

    Dimetric scenes often look reminiscent of video games and cartoonish, and the method has limited ability to realistically replicate real-world scenes.

    Source: C Limitations · p. 25

Abstract

from arXiv · show

We present SynCity 3000, a framework for generating 3D scenes that are globally coherent while enabling fine-grained layout control. Building on the ability of current image-to-3D generators to produce complex 3D assets from a single image, we extend this capability to the scale of entire scenes by adapting the generator to be applicable as a convolutional operator. We achieve this by fine-tuning the model on scene-like data generated by a new synthetic data engine, which we propose to address the scarcity of 3D scene data for training. The convolutional generator is then applied to a dimetric image of the entire scene, generated from the user prompt, resulting in 3D scenes of arbitrary size and complexity. Across diverse prompts and layouts, SynCity 3000 produces large, coherent, and detailed scenes, addressing the shortcomings of prior approaches to 3D scene generation.

1 Introduction

SynCity 3000 generates large 3D worlds by creating a scene-wide 2D template and converting it into 3D, without requiring a fixed tile grid. It supports arbitrary scene structures and complexity through optional layout constraints and sliding-window generation.

1 Introduction

  • Earlier SynCity avoided training a scene generator from scratch by generating semi-independent grid tiles, but the grid remained visible in its outputs.SynCity 3000 is introduced to overcome this fixed-grid limitation.Source: §1 · p. 1
  • SynCity 3000 automatically converts a scene-wide 2D appearance-and-layout template into a 3D scene with arbitrary structure and complexity.The two stages are independent, so template-like images—including artist-created images—can be used as input.Source: §1 · pp. 1–2
  • Overlapping latent-space windows let the image generator create scenes of any extent, while optional layout constraints provide fine-grained control and blend into their surroundings.The method averages constraint contributions during image diffusion, inspired by MultiDiffusion.Source: §1 · p. 2
  • The 3D stage voxelizes the world, enriches each voxel with visual and semantic features, and decodes the result into detailed 3D Gaussian Splats.The generator is fine-tuned on procedural synthetic data to operate convolutionally over overlapping sliding windows, enabling arbitrary scene sizes.Source: §1 · p. 2
  • Across varied scene sizes and complexities, SynCity 3000 outperforms SynCity and other reconstruction methods qualitatively and quantitatively.The introduction points to Figs. 6 and 7 and Tab. 1 for these results.Source: §1 · p. 2

2 Related work

Prior 3D scene-generation methods use image projection, asset placement, or direct scene synthesis, but face geometric artifacts, limited controllability, dataset dependence, or coherence and generalization problems. SynCity 3000 addresses these limits with convolutional 3D generation fine-tuned using synthetic scene-like data.

Image-based scene generation

  • Image-based methods turn generated images into meshes, Neural Radiance Fields, or 3D Gaussian Splats, but occlusions cause artifacts and holes away from generation viewpoints.Methods commonly project images using monocular depth estimates, while others use panoramas or posed images.Source: §2 · p. 3

Asset-based scene generation

  • Asset-based methods place retrieved or generated 3D assets into image-, language-model-, or diffusion-derived layouts, but often produce monotonous, unrealistic scenes with collisions and indoor-scene restrictions.Layouts may be image-based, generated by large language models or trained diffusion models, with assets retrieved or generated using object-centric models.Source: §2 · p. 3

Explicit scene generation

  • Explicit synthesis includes LT3SD, X 3, BlockFusion, and SceneCraft, but requires domain-specific data or struggles with conditioning; focuses on semantic completion, not text or image interfaces.LT3SD uses unconditioned patch-by-patch coarse-to-fine diffusion, X 3 uses hierarchical sparse voxel grids, BlockFusion extends meshes autoregressively, and SceneCraft generates scene renderings.Source: §2 · p. 3

SynCity

  • SynCity generates and fuses prompted tiles using TRELLIS, producing themed scenes whose sequential generation yields a busy, grid-like appearance and weak overall coherence.TRELLIS generates high-quality 3D objects from object-centric datasets.Source: §2 · p. 3

3DTown

  • 3DTown converts scene images into Gaussian Splats using depth-based priors and region completion, but generated scenes retain geometric artifacts, holes, and limited resolution.It partitions the scene into overlapping regions and masks unknown parts during 3D generation.Source: §2 · pp. 3–4

NuiScene

  • NuiScene enables unbounded chunk-based outpainting from 43 Objaverse scenes, but chunk discontinuities, separately generated textures, and limited training data constrain quality and generalization.Textures are generated with SceneTex, and chunk encodings use learned VecSet.Source: §2 · p. 4

Positioning

  • SynCity 3000 combines a controllable scene template with convolutional inference fine-tuned on synthetic scene-like data, avoiding brittle heuristics and the shortage of broad scene datasets.The approach is described as strictly generalizing SynCity and scaling far beyond NuiScene’s training dataset.Source: §2 · p. 4

3 Method

SynCity 3000 turns global and optional local scene prompts into a 3D Gaussian Splat scene through separate 2D-template and 3D-generation stages. Its windowed denoising and convolutional 3D models support coherent generation across larger scenes.

3 Method

  • The method accepts global theme and style prompts, optionally adds local prompts tied to scene windows, and outputs 3D Gaussian Splats renderable from arbitrary viewpoints.Source: §3 · p. 4

3.1 2D template generation

  • Overlapping windows denoised jointly pass information across the latent canvas, while dimetric inpainting on a supporting base creates a 2D template with global coherence.Windows are conditioned on the global style and either a local prompt or the overall theme prompt.Source: §3.1 · p. 5

3.2 3D generation

  • TRELLIS is fine-tuned for convolutional inference so it can transform the complete scene template at once, rather than reconstructing separate tiles.Its generators produce a coarse voxel structure and per-voxel structured latents, which are decoded into 3D Gaussian Splats.Source: §3.2 · p. 6

Convolutional sparse structure generation.

  • Overlapping latent windows use matching image crops and averaged denoising updates; masked conditioning and surrounding 3D context focus core predictions while preserving spatial context.Core positions retain the original positional encoding, while context positions extend beyond the core; alternating context and no-context objectives with loss on the core support convolutional adaptation and the original TRELLIS task.Source: Convolutional sparse structure… · pp. 7–8

Convolutional structured latents generation.

  • The sparse structured-latent generator uses the same convolutional scheme because its tokens are sparse rather than dense, while otherwise resembling the sparse-structure model.Source: Convolutional structured latents… · p. 9

Scene inference.

  • Choosing 3D latent resolution in core-resolution multiples establishes template-to-latent correspondence and independently controls scene size and detail.Windowed inference denoises core and context but retains core predictions, averaging overlapping contributions at each timestep.Source: Scene inference · p. 9

4 Dataset

SynCity introduces a synthetic scene-data engine to address the scarcity of 3D scene data for fine-tuning scene-scale generators. It assembles randomized Objaverse scenes and exports them in a dimetric, voxel-based format aligned with the training scheme.

4 Dataset

  • To address the scarcity of scene-scale 3D data, SynCity introduces a synthetic dataset engine that generates complete scenes for its scene-scale diffusion scheme.The engine is intended to support fine-tuning TRELLIS for large-scale scenes.Source: §4 · p. 9
  • The engine places Objaverse-XL objects on randomly generated surfaces, following the random-shape-and-texture bootstrapping approach of LRM-Zero.The generated scenes are rendered in dimetric projection to match the 2D templates.Source: §4 · p. 10, Figure 5 · p. 10
  • Random terrains, colors, object counts, placements, scales, and lighting produce varied scenes with few restrictions on content.The engine places randomly selected Objaverse objects on variable-size surfaces, keeping objects away from the surface margins and adding overhead shadows.Source: §4 · p. 10
  • Dimetric renders are paired with centered voxel crops containing the core and context, then augmented with projected DINOv2 features to produce sparse structures and structured latents.The crop uses a cube of resolution M_context × M_context × M voxels; additional perspective views provide the DINOv2 features.Source: §4 · p. 10, Figure 5 · p. 10
  • The Blender engine uses 32 random top views, one view from each side, and one bottom view for its additional perspectives.This yields VF := 32 + 5 views.Source: §4 · p. 10

5 Experiments

SynCity 3000 is evaluated for layout control, template faithfulness, geometric reconstruction quality, and overall scene quality using user studies and quantitative and qualitative comparisons. The experiments also test how fine-tuning, context, and inference stride affect reconstruction.

Experimental details

  • The models were fine-tuned on 320k generated samples with batch size 1, and scene inference used a 0.5-patch stride on 1344 × 672 px templates.The sparse structure and structured latent models were trained for 260k and 660k steps, respectively, on 2× NVIDIA RTX A6000 GPUs; each template’s core patch was 896 × 448 px.Source: §5 · p. 11

Comparisons

  • Users unanimously preferred SynCity 3000’s flexible layout control over SynCity’s rigid, low-resolution grids, which constrain open scenes and structures spanning tiles.The comparison used the same theme prompts and LLM-generated instructions; the convolutional approach alleviates this tile-based limitation.Source: §5 · pp. 11–12, Figure 6 · p. 12
  • On 35 LLM-generated templates, object-centric TRELLIS, TripoSG, and Hunyuan3D-2.1 produced insufficient-resolution scenes that could lack detail or deviate from the template, especially at larger scales.Reconstructions were rendered with an orthographic perspective matching the template and evaluated with image-similarity metrics; the large-scene comparison illustrates the detail difference.Source: §5 · p. 12, Figure 7 · p. 14
  • Across two scales of 25 synthetic scenes each, SynCity 3000 stayed consistent and ahead in most geometric-quality metrics, while TRELLIS deteriorated as scene size increased.Evaluation used Chamfer Distance, F-score, LPIPS, and PSNR; LPIPS and PSNR were measured on 32 random views.Source: §5 · p. 12, Table 3 · p. 13
  • For general generation preference, users chose between scenes in forced binary choices using prompts similar to those generated by NuiScene and 3DTown; SynCity 3000 scenes were overall preferred.Source: §5 · p. 13

Ablations

  • Fine-tuning enables convolutional inference; removing context mildly reduces quality, reduced context also hurts, and smaller strides improve predictions by averaging tiles and reducing duplicated structures.Off-the-shelf models perform significantly worse, partly because the sparse structure model infers a canonical front-facing orientation that is ill-defined for directionless objects; smaller strides also resolve ambiguities from template masking.Source: §5 · pp. 13–14

6 Conclusion

SynCity 3000 generates large-scale 3D scenes from scratch by converting complex layouts into 2D templates and applying a fine-tuned two-stage 3D generative model through convolutional inference. Joint scene generation promotes semantic and visual coherence while preserving high-quality predictions without brittle scene-generation heuristics.

6 Conclusion

  • SynCity 3000 generates large-scale 3D scenes from scratch by converting complex layouts into 2D templates and producing 3D Gaussian Splats through convolutional inference.The method uses shifted windows to form the templates and a fine-tuned two-stage 3D generative model to convert them.Source: §6 · p. 14
  • A synthetic dataset engine creates random scene-like 3D structures used to fine-tune the generative model.Source: §6 · p. 14
  • Generating the complete scene as a template in a joint diffusion process ensures semantic and visual coherence while fine-tuning preserves high-quality predictions and removes brittle heuristics.The authors describe these as addressing limitations of SynCity and previous scene-generation methods, respectively.Source: §6 · p. 14

A Additional technical details

The appendix details how SynCity 3000 builds layouts and templates, scales latent decoding to arbitrarily sized scenes, and corrects color statistics. It also reports inference settings and runtimes for template generation and 3D generation.

LLM prompting

  • An LLM can draft scene and theme prompts plus up to three region constraints in relative patch units; these instructions guide only 2D template generation.The prompt asks for varied world styles, keeps specified scene parameters fixed, and avoids fog, steam, clouds, and very small objects.Source: A Additional technical details · p. 21

Latent encoding & decoding

  • The sparse-structure VAE is fully convolutional, while shifted windows let the transformer-based structured-latent VAE and 3DGS decoder handle arbitrary scene sizes.For structured-latent decoding, small-stride shifted windows with averaging worked best; each voxel’s Gaussians are predicted once and fused into the larger scene representation.Source: A Additional technical details · p. 21

Color correction

  • To address reduced reconstruction lightness, the method transfers each channel’s mean and standard deviation from the template to Gaussians in L*a*b* space.This is a visual enhancement, not a simplification: the method must still predict colors correctly in the less-illuminated color space.Source: A Additional technical details · pp. 22–23

Template generation details

  • Template generation sizes base image and mask extents to each window, uniformly weights overlapping windows, and lets unconstrained windows condition the black latent background.The black background is removed for visualization in the paper.Source: A Additional technical details · p. 23

Inference

  • Template generation uses Flux and ControlNet at guidance 3.5 for 24 steps per window; sparse-structure and structured-latent stages use guidance/steps 7.5/50 and 3.0/12.On one NVIDIA RTX A6000, template generation took about 3–5 minutes and the two 3D stages took about 30 minutes for an average scene; fewer sparse-structure steps can be used with minor qualitative degradation.Source: A Additional technical details · p. 23
  • Runtime/peak memory were 31m/45.7G, 82m/48.4G, and 179m/55.8G at template sizes 1344 × 672, 2240 × 1120, and 3136 × 1568 px, scaling roughly quadratically.These measurements used a single NVIDIA H100 GPU.Source: A Additional technical details · p. 23
  • For human-in-the-loop creativity, template-generation speed matters more; 3D generation only transforms templates into Gaussian splats, with smaller quality degradation than prior methods (Tab. 1).The 3D-generation process makes no further modifications to the template.Source: A Additional technical details · p. 23

B Additional experimental details

The section describes evaluation setups for reconstruction faithfulness and overall scene preference, including template selection and choices made to enable fair comparisons.

B Additional experimental details

  • After seven failures by TripoSG and Hunyuan3D-2.1, reconstruction-faithfulness evaluation compared methods on 35 of 42 templates spanning six themes and seven seeds per theme.Prompts were randomly generated with an LLM, and users could select their preferred reconstruction method; Fig. 5 shows template examples.Source: B Additional experimental details · pp. 23–24
  • The overall-preference user study compared three scenes per prior work, balancing grid-like and non-grid-like SynCity scenes while accounting for each method’s capabilities and available code and data.For NuiScene, SynCity prompts were chosen to produce similar fortress, suburban-city, and pagoda scenes, and textures were shown despite not being part of NuiScene; three teaser scenes from 3DTown were converted into templates because its code was unpublished.Source: B Additional experimental details · p. 24

C Limitations

SynCity 3000 improves on prior work but retains limitations in perspective, texture fidelity, user interaction, structural reconstruction, and realistic appearance.

Forced perspective

  • Convolutional inference requires scenes to be presented in a dimetric perspective, and future work could remove this constraint.Source: C Limitations · p. 24

Texture detail

  • Averaging overlapping shifted-window results slightly reduces texture fidelity compared with the original TRELLIS model.Source: C Limitations · p. 24

Degree of control

  • Users must wait for a generated template before judging it, motivating more interactive or real-time approaches that allow direct intervention.Source: C Limitations · p. 24

Structure duplication

  • Tiling can duplicate or partially break tall or large structures under severe occlusion when the fine-tuned stage 1 model fails to align geometry.This limitation follows from the method’s convolutional design and tiled template.Source: C Limitations · p. 24

Cartoonish appearance

  • SynCity scenes have a cartoonish look and limited ability to realistically reproduce real-world scenes, with training biases presumed to amplify the effect.The authors suspect biases from stylized Objaverse-XL objects and FLUX, and say dimetric scenes may carry video-game associations.Source: C Limitations · pp. 24–25

D Additional comparisons

Qualitative comparisons show that SynCity 3000 reconstructs scene details more faithfully than recent image-conditioned 3D models and scales to arbitrary scene sizes. TRELLIS produces more convincing reconstructions than TripoSG and Hunyuan3D-2.1, but remains limited in resolution and detail fidelity.

D Additional comparisons

  • TripoSG [36] and Hunyuan3D-2.1 [56] often fail to resemble the scene template, with poor color accuracy, unreliable geometry, and possible holes.Source: D Additional comparisons · p. 25
  • TRELLIS [62] reconstructions look significantly more convincing overall, but compared with the authors’ method, details can appear fuzzy or blurry and may deviate severely, especially in detail.Source: D Additional comparisons · p. 25
  • SynCity 3000 more faithfully reconstructs scene details and scales to arbitrary scene sizes, unlike object-centric models limited in output size.The comparison notes that SynCity 3000 preserves car positions, orientation, and shape where TRELLIS deviates from the template.Source: D Additional comparisons · p. 25

E Additional qualitative results

Additional qualitative results show that SynCity 3000 can produce fully realized scenes, gain detail from larger templates, and reconstruct scenes from templates made outside its pipeline. They also expose limitations in indoor-scene plausibility caused by FLUX template biases.

Scene size

  • A 3136×1568-pixel template increases scene “resolution” and detail fidelity, while TRELLIS scenes lose details due to lower resolution.Scene size is set in layout instructions and affects template size and, in turn, 3D scene size.Source: E Additional qualitative results · p. 25, Figure 4 · p. 28

Templates not generated by our pipeline

  • SynCity 3000 can reconstruct scenes from templates not generated by its pipeline, including a template generated by ChatGPT Images.The synthetic dataset engine creates diverse samples not tied to the pipeline’s scene-template generation.Source: E Additional qualitative results · p. 25, Figure 2 · p. 26

Indoor scenes

Additional views

Dataset samples

Related papers

Loading 2607.05392v1…