PaperlayerView exact revision on arXiv

Source-linked AI summary

DistScene: Object-to-Scene Distillation for 3D Scene Generation

Kunming Luo, Hongyu Yan, Ken Deng, Chengcheng Zhou, Tianyu Liu, Haipeng Li, Haibin Huang, Xuelong Li, Ping Tan

arXiv:2610.06960v1cs.CV

TL;DR

Single-image compositional 3D generation must recover objects, their placement, and geometry, while existing approaches can propagate errors or omit environmental geometry. DistScene jointly generates the environment and objects, refines objects with scene context, and uses automatically composed synthetic scenes for training; the paper reports improved scene coherence and fine-grained object detail over existing approaches.

  • Problem

    Compositional 3D scene generation must jointly recover object decomposition, spatial placement, and geometry from one image, while cascaded methods can propagate errors and feed-forward methods often omit environmental geometry.

    Source: §1 · pp. 1–2

  • Method

    DistScene jointly generates environment and object components in a shared scene frame, refines each object locally with scene context, and distills training supervision from composed and rendered synthetic scenes.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    DistScene produces more coherent scenes than existing approaches while preserving fine-grained local object details across indoor and outdoor settings.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    The explicit environment component provides a common geometric context for organizing object placement and spatial relationships.

    Source: §1 · p. 2, §6 · p. 9

    The outdoor training set is smaller than the indoor set because of limited computational resources.

    Source: §A.5 · p. 18

Abstract

from arXiv · show

We present DistScene, a framework for single-image compositional 3D scene generation by jointly modeling the environment and individual objects. Unlike existing methods that represent scenes primarily as collections of objects, we model the environment as an explicit scene component to provide geometric context for object placement. Specifically, we introduce Scene-Frame Generation, which jointly generates separate environment and object components in a shared coordinate frame, allowing their geometry and relative placement to be learned together. Then we introduce Object-Centric Refinement to refine each object in a local frame with scene context. Finally, we develop Object-to-Scene Distillation to transfer pretrained object-generation priors to scene generation through automatically composed and rendered synthetic scenes. Evaluations on indoor and outdoor benchmarks demonstrate improved scene-level spatial coherence over the evaluated baselines. Project page: https://coolbeam.github.io/DistScene/

1 INTRODUCTION

Single-image compositional 3D scene generation must jointly recover object decomposition, placement, and geometry, yet existing approaches either propagate errors across stages or omit explicit environmental geometry. DistScene jointly generates environment and objects, refines objects locally, and distills training supervision from a pretrained object generator.

1 INTRODUCTION

  • Single-image scene generation must jointly resolve object decomposition, spatial placement, and 3D geometry, while cascaded and object-only approaches leave errors or environmental constraints unresolved.Cascaded pipelines can propagate intermediate errors; feed-forward methods often omit the environment, making globally coherent layouts difficult.Source: §1 · pp. 1–2
  • Scene-Frame Generation jointly produces separate environment and object components in a shared frame, using the environment as spatial context for object placement, scale, and relationships.Source: §1 · p. 2
  • Object-Centric Refinement addresses lost object detail by refining each object in a normalized local space conditioned on scene context, then returning it to its scene location.The stated goal is to improve local geometry while preserving scene layout and component relationships.Source: §1 · p. 2
  • Object-to-Scene Distillation automatically constructs scene supervision from a pretrained object generator, addressing the need for decomposed environment and object training data.The introduction reports more coherent scenes than existing approaches while preserving fine-grained local object details.Source: §1 · p. 2

2 RELATED WORK

Prior compositional scene-generation work includes component-aligned reconstruction pipelines and feed-forward generation of multiple objects and their layouts. DistScene differs by explicitly generating the environment alongside objects and using object-generation priors beyond their usual canonical-space setting.

Compositional 3D Scene Generation

  • Component-aligned reconstruction methods combine scene decomposition, placement or pose estimation, and geometry estimation, while feed-forward methods generate multiple instances and their arrangements in one learned process.CAST and SAM3D exemplify the reconstruction direction; MIDI and SceneGen exemplify feed-forward generation.Source: §2 · p. 2
  • Feed-forward methods remain primarily object-centric, whereas DistScene jointly generates the environment and objects in a shared scene frame and refines objects while preserving scene placement.The related-work discussion identifies missing environmental geometry as a limitation on grounding object organization.Source: §2 · p. 3

3D Object Generation

  • Object generators provide geometry and appearance priors but typically do not encode an asset’s position, scale, or relations to an environment; DistScene extends these priors to shared scene-space generation.Source: §2 · p. 3

3 PRELIMINARY: SPARSE-VOXEL-BASED 3D GENERATION

The preliminary section describes a sparse-voxel image-to-3D generation paradigm: an asset is encoded into a latent representation, generated conditionally on an image, and decoded. DistScene extends this single-asset paradigm to a complete compositional scene.

3 PRELIMINARY: SPARSE-VOXEL-BASED 3D GENERATION

  • Sparse-voxel generation represents assets as active grid voxels carrying geometry and material features, compressed by a sparse VAE into a compact latent space.Source: §3 · p. 3
  • A flow-matching generator conditioned on an image synthesizes assets in stages: sparse occupancy, geometry latents, then appearance latents aligned with geometry.Source: §3 · p. 3
  • The single-asset pipeline encodes and decodes sparse latents around image-conditioned generation, and DistScene extends this paradigm to complete compositional scenes.The passage names E and D as the sparse VAE encoder and decoder, G as the image-conditioned generator, and I as the input image.Source: §3 · p. 3

4 METHOD

DistScene generates a decomposable scene from one image in a unified sparse-voxel space, jointly synthesizing objects and environment before refining objects with scene context.

4.1 OBJECT-TO-SCENE DISTILLATION

  • Object-to-Scene Distillation builds scalable scene supervision without existing scene datasets or human annotation by composing generated meshes and environments, adjusting plausible layouts, and rendering views with all objects visible.Layout adjustment checks environment contacts and inter-object collisions; only views where all objects are visible are retained as conditioning images.Source: §4.1 · p. 4

4.2 SCENE-FRAME GENERATION

  • Scene-Frame Generation jointly generates environment and object components in a shared sparse-voxel frame, using the environment latent to anchor object positions and encode spatial interactions.The components are generated together without external segmentation or reconstructed scene geometry at inference.Source: §4.2 · pp. 4–5

4.3 SCENE-AWARE OBJECT-CENTRIC REFINEMENT

  • Object-Centric Refinement re-centers each object in canonical space and conditions higher-resolution recovery on the image, scene latents, and object-region marker before reinsertion at its original position.Scene conditioning helps preserve spatial and semantic correspondence, while independent refinement leaves the global layout unchanged.Source: §4.3 · pp. 5–6

4.4 TRAINING OBJECTIVE AND NETWORK ARCHITECTURE

  • Training adapts a pretrained sparse DiT with LoRA and separately flow-supervises object and environment latents; refinement learns detail recovery from self-generated coarse-to-fine pairs without external high-resolution data.The LoRA adapters keep the pretrained backbone frozen while retaining its generative prior.Source: §4.4 · p. 6

5 EXPERIMENTS

DistScene is evaluated on indoor and outdoor scene-generation benchmarks, with ablations testing environment generation, distilled training data, object refinement, and resolution. The reported results favor DistScene on the main comparisons and show distinct gains from its components.

5.1 IMPLEMENTATION DETAILS

  • The main experiments use 1024-resolution scene-frame generation and refinement; synthetic training scenes are generated at 512 resolution, and the adapted model supports 1024-resolution inference.Indoor evaluation uses MIDI-test and Gen3DSR-test; outdoor evaluation uses UrbanScene3D.Source: §5.1 · p. 7

5.2 MAIN COMPARISON

  • On UrbanScene3D, DistScene achieves CD-L1 0.0772 and F-score 0.722; on MIDI-test, scene CD falls from 0.1295 to 0.0877 and scene F-score rises from 65.08 to 71.59 versus 3D-Fixer.The outdoor values are the best among evaluated methods; the MIDI-test changes correspond to a 32.2% relative CD improvement and a 6.51-point F-score increase.Source: §5.2 · p. 7, Table 1 · p. 7, Table 2 · p. 7
  • On Gen3DSR-test, DistScene records scene CD 0.0958 and F-score 79.68%, versus 0.1027 and 77.97% for 3D-Fixer; it also leads the user study across all three criteria.Qualitative comparisons report better preservation of relative layout and support relationships, including on diverse text-to-image inputs.Source: §5.2 · pp. 7–8, Table 2 · p. 7, Table 3 · p. 8

5.3 ABLATION STUDIES

  • Adding the environment raises MIDI-test object F-score from 29.98 to 40.98 and bounding-box IoU from 0.1798 to 0.2544, while Gen3DSR-test scene CD falls from 0.1367 to 0.1096.Gen3DSR-test scene F-score rises from 62.04 to 73.21; MIDI-test scene CD is nearly unchanged while scene F-score rises from 67.94 to 68.86.Source: §5.3 · p. 8, Figure 6 · p. 9
  • With the same object-only architecture, distilled data improves scene-level metrics over MIDI training and yields larger gains on Gen3DSR-test, where scene CD falls from 0.1936 to 0.1367 and F-score rises from 46.68 to 62.04.Both test sets are out of domain for the synthetic data; MIDI training retains an advantage on MIDI-test object metrics and bounding-box IoU, while the comparisons indicate transfer beyond the synthetic training distribution.Source: §5.3 · pp. 8–9
  • Object-Centric Refinement lowers MIDI-test object CD from 0.2037 to 0.1529 and raises object F-score from 40.98 to 51.64; removing scene-frame context performs markedly worse across metrics.Without scene context, refinement has difficulty associating the target component with its corresponding object in the input image.Source: Table 4 · p. 8, §5.3 · p. 9
  • Increasing resolution consistently improves reconstruction quality at additional runtime and modest memory cost; the 512-resolution refined variant uses 18.0 GB peak memory.It requires 5.7 seconds for scene generation and 3.3 seconds per refined object, while the full 1024-resolution model achieves the best accuracy with 21.8 GB peak memory.Source: §5.3 · p. 9, Table 5 · p. 9

6 CONCLUSION

DistScene makes the environment an explicit component of compositional 3D scene generation. It combines shared-frame generation, local object refinement, and distilled supervision, with experiments reporting superior results on indoor and outdoor benchmarks.

6 CONCLUSION

  • DistScene explicitly models the environment alongside objects, providing shared geometric context for their spatial arrangement.The environment and objects are generated together in a shared scene frame.Source: §6 · p. 9
  • Object-Centric Refinement restores fine object geometry in local coordinate frames, while Object-to-Scene Distillation supplies aligned training supervision.Source: §6 · p. 9
  • Experiments on indoor and outdoor benchmarks show superior results to existing methods.Source: §6 · p. 9

A SUPPLEMENTARY MATERIAL

The supplement outlines additional architecture and data-curation details, comparisons, limitations, and qualitative results. It also points readers to accompanying videos.

A SUPPLEMENTARY MATERIAL

A.1 NETWORK ARCHITECTURE

DistScene uses sparse voxel supports and structured latents to generate environment and object geometry jointly in a shared scene frame, then refines each object in a normalized local frame using scene context.

A.1 NETWORK ARCHITECTURE

  • Sparse voxels provide a common spatial substrate: a voxel support marks active coordinates and coarse component extent, while a Structured Latent assigns learned features to active voxels.Shape SLATs encode geometry; texture SLATs encode appearance and material information conditioned on geometry.Source: §A.1 · p. 14
  • A flow transformer jointly generates structure latents for the environment and objects, which decode into separate sparse voxel supports in the shared scene frame.Learnable type embeddings distinguish component slots, and each support retains component identity.Source: §A.1 · p. 14
  • A sparse flow transformer generates geometry features on active voxels across components, enabling their decoded meshes to be assembled directly while remaining separate.Source: §A.1 · p. 14, Figure 7 · p. 15
  • For refinement, each target mesh is translated and uniformly scaled into a normalized local frame, then voxelized to increase effective spatial resolution while preserving orientation.The refinement transformer conditions on the image and fixed scene-frame geometry latents; component embeddings preserve correspondence.Source: §A.1 · p. 14, Figure 7 · p. 15
  • Only the target object's local latent is regenerated; the decoded refined mesh is returned to its recorded scene placement and scale, leaving scene-frame latents unchanged.Figure 7 uses snowflakes for frozen pretrained transformer weights and flames for trainable LoRA adapters.Source: §A.1 · pp. 14–15, Figure 7 · p. 15

A.2 TRAINING DATA CURATION

Object-to-Scene Distillation builds scene supervision by composing generated objects with generated environments, then rendering and re-encoding accepted scenes. The resulting examples retain separable scene components and rendered conditioning views.

A.2 TRAINING DATA CURATION

  • Objects are generated as canonical assets, sampled for each environment, and placed with sampled scales and translations in the environment coordinate system.Component identities, canonical latents, and scene-space transforms are retained.Source: §A.2 · p. 15
  • Compositions pass geometry-based checks for support, object intersections, and collisions with non-floor environment surfaces.Floating or penetrating placements are discarded; unresolved object intersections are also rejected.Source: §A.2 · p. 15
  • Accepted scenes are rendered from sampled viewpoints and re-encoded into shared scene-frame targets, while canonical object latents remain refinement targets.The retained targets include the original canonical shape and texture SLATs.Source: §A.2 · p. 15
  • Examples span indoor and outdoor scenes, with separately identifiable components and multiple rendered conditioning views.The visualized samples include diverse object compositions and spatial layouts.Source: §A.2 · p. 15, Figure 8 · p. 16

A.3 TRAINING AND INFERENCE DETAILS

The training and inference procedures adapt pretrained generation modules and jointly generate scene components before refining objects locally. Evaluation inputs are rendered to provide the surrounding environment required by the model.

A.3 TRAINING AND INFERENCE DETAILS

  • Trainable generation modules use rank-32 LoRA while pretrained backbones remain frozen, with 20,000 iterations on 512-resolution targets.The main experiments run the same modules at 1024 resolution.Source: §A.3 · p. 16
  • Training scenes are capped at 30 components, with empty object latents padding sequences to a fixed component count.Inference uses 20 slots by default and discards slots decoded as empty supports.Source: §A.3 · p. 16
  • Figure 9 compares four combinations of scene-frame generation and object refinement at 512 and 1024 resolution using matched scene views and geometry crops.Source: Figure 9 · p. 17
  • Training uses component-aligned scene supervision for scene-frame generation and object refinement; inference jointly generates the environment and objects before local refinement.Source: §A.3 · p. 17
  • MIDI-test scenes are re-rendered with floors, walls, and objects to supply complete-scene inputs, while the original geometry and evaluation protocol remain unchanged.Aligned object masks and depth maps are also rendered for methods requiring those inputs.Source: §A.3 · p. 17

A.4 RESOLUTION COMPARISON

The qualitative comparisons report coherent scene layouts and detailed object geometry, with higher-resolution refinement resolving structures more clearly in the shown examples. The failure cases identify indoor ceiling omissions and outdoor geometry loss as limitations tied to data and representation constraints.

A.4 RESOLUTION COMPARISON

  • Refinement at 1024 resolution produces more clearly resolved structures than the shown alternatives, even when overall arrangements are similar.The examples highlight thin plant structures and local shape details on outdoor objects and vehicle surfaces.Source: §A.4 · p. 17

A.5 FAILURE CASES AND LIMITATIONS

  • Indoor reconstructions may omit ceilings because the generated environment assets used for training generally lack ceilings.The current setup has difficulty obtaining reliable fully enclosed empty rooms from the pretrained generator.Source: §A.5 · p. 17
  • Outdoor scenes may contain holes or lose fine geometry when their large spatial extent spreads limited scene-frame voxels over a wide region.The outdoor training set is also smaller than the indoor set because of computational-resource limits.Source: §A.5 · pp. 17–18, Figure 10 · p. 20

A.6 ADDITIONAL QUALITATIVE RESULTS

  • DistScene produces globally coherent layouts and detailed environment and object geometry, while the comparisons identify shortcomings in competing methods' context or component representations.Extend3D may have incomplete geometry; TRELLIS.2 can fuse objects into one representation, while 3D-Fixer and SAM3D may lack environmental context or complete geometry.Source: §A.6 · pp. 19–20, Figure 11 · p. 21, Figure 12 · p. 22

Related papers

Loading 2610.06960v1…