Source-linked AI summary
I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
Lu Ling, Yunhao Ge, Yichen Sheng, Aniket Bera
TL;DR
Limited scene-dataset coverage constrains generalization in interactive 3D scene generation. I-Scene reprograms a pre-trained 3D instance model as a scene-level spatial learner and uses a view-centric representation. It achieves strong spatial reasoning from randomly composed scenes and generalizes to unseen layouts.
Problem
Learning-based interactive scene generators rely on limited, curated scene datasets, whose restricted coverage constrains generalization to new layouts.
Source: §1 · pp. 1–2Method
I-Scene reprograms a pre-trained 3D instance prior for scene-level spatial learning and uses view-centric scene space to preserve layout cues.
Source: §1 · p. 2Results
Trained on randomly composed scenes, I-Scene demonstrates strong spatial reasoning, surpasses dataset-bounded baselines, and generalizes to various layouts.
Source: §1 · p. 2Takeaways & Limitations
Abstract
from arXiv · showhide
Generalization remains the central challenge for interactive 3D scene generation. Existing learning-based approaches ground spatial understanding in limited scene dataset, restricting generalization to new layouts. We instead reprogram a pre-trained 3D instance generator to act as a scene level learner, replacing dataset-bounded supervision with model-centric spatial supervision. This reprogramming unlocks the generator transferable spatial knowledge, enabling generalization to unseen layouts and novel object compositions. Remarkably, spatial reasoning still emerges even when the training scenes are randomly composed objects. This demonstrates that the generator's transferable scene prior provides a rich learning signal for inferring proximity, support, and symmetry from purely geometric cues. Replacing widely used canonical space, we instantiate this insight with a view-centric formulation of the scene space, yielding a fully feed-forward, generalizable scene generator that learns spatial relations directly from the instance model. Quantitative and qualitative results show that a 3D instance generator is an implicit spatial learner and reasoner, pointing toward foundation models for interactive 3D scene understanding and generation. Project page: https://luling06.github.io/I-Scene-project/
1. Introduction
I-Scene addresses limited scene-dataset coverage by reprogramming a pre-trained 3D instance model as a spatial learner. A view-centric scene space and randomly composed training scenes support feed-forward generation that generalizes to unseen layouts.
1. Introduction
- Limited scene-dataset scale and diversity constrain generalization; 3D-FRONT contains about 20K bedroom and living-room scenes and under-represents small and supporting objects.Existing approaches learn spatial understanding from curated scene datasets.Source: §1 · pp. 1–2
- A pre-trained 3D instance model implicitly encodes transferable spatial knowledge, including depth, occlusion, scale, and support, despite producing single-mesh geometry.I-Scene reprograms this prior to provide scene-level spatial supervision without curated scene annotations.Source: §1 · p. 2
- View-centric scene space preserves layout relationships between the camera and scene, addressing canonical space’s loss of view-dependent layout cues.Each scene is represented in a view-dependent coordinate frame.Source: §1 · p. 2
- Trained solely on random layouts, I-Scene surpasses state-of-the-art methods trained on 3D-FRONT and transfers robustly to diverse unseen layouts.The formulation learns spatial relations directly from the instance prior without meaningful relations in its training scenes.Source: §1 · p. 2
2. Related Work
Prior work generates interactive 3D scenes through monolithic scene representations, compositional pipelines, or learning-based methods trained on curated datasets. I-Scene instead combines global scene reasoning and instance generation in a feed-forward, view-centric approach intended to generalize beyond dataset biases.
2. Related Work
- Monolithic scene-generation approaches offer view consistency and speed but lack instance accordance, limiting fine-grained editing, interaction, and physics reasoning.Examples include methods that generate Gaussian splats or layered surfels from a single view.Source: §2 · p. 2
- Compositional pipelines provide high fidelity and open-ended semantics but can suffer from perception or planning errors, retrieval or solver mismatch, and per-scene optimization costs.These costs limit throughput and scalability.Source: §2 · p. 2
- Learning-based scene methods generally learn layout distributions from curated datasets, and can struggle with rare layouts such as small objects supported by or occluded behind larger ones.This limitation applies to layout-first and end-to-end multi-instance approaches described in the passage.Source: §2 · pp. 2–3
- I-Scene jointly reasons over global scene context and instance generation in a feed-forward, view-centric design, aiming to generalize beyond dataset biases without retrieval or solver handoffs.The framework learns spatial knowledge from non-semantic synthetic layouts while preserving instance-level editability.Source: §2 · p. 3
3. Method
I-Scene predicts scene layout and instance geometry in a shared view-centric space, using scene-context attention to condition instance generation on global spatial guidance. Its training also uses randomly composed, collision-free scenes rather than requiring semantic layouts.
3.1. Problem Definition
- Given a scene image and instance masks, I-Scene generates independently manipulable 3D instances whose layout aligns with the image.Source: §3.1 · p. 3
3.2. Overview
- The spatial guidance branch predicts a sparse-voxel scene layout, which anchors instance generation; the instance branch uses that guidance to predict voxelized instance features and pose.The shared anchor keeps independently generated instances coherent, while allowing the instance branch to focus on geometry and follow the scene layout.Source: §3.2 · p. 3
3.3. Scene-Context Attention
- Scene-context attention conditions instance generation on spatial-guidance keys and values as well as the instance’s own, while making minimal changes to the base model’s prior.When scene and instance inputs are identical, the authors state that this attention is equivalent to the original self-attention.Source: §3.3 · p. 3
3.4. View-Centric(VC) Space
- View-centric space preserves the relationship between camera view and scene layout, unlike canonical space, where identical objects can be placed at the same position across views.The authors report better generalization in view-centric space in their ablation study.Source: §3.4 · pp. 3–4, Figure 3 · p. 4
3.5. Non-Semantic Synthetic 3D Scene
- Training scenes are randomly composed from diverse 3D instances with collision-free placement and basic spatial relations, without semantic layout labels.The authors report that this training further improves generalization and supports spatial reasoning while remaining agnostic to category semantics.Source: §3.5 · p. 4
3.6. Training
- Training uses a conditioned rectified-flow target, with x(t) interpolating between x_0 and noise ϵ over time t.The passage identifies vθ as the sparse-structure neural network.Source: §3.6 · p. 4
4. Experiment
I-Scene is evaluated on in-domain and out-of-domain synthetic scenes, as well as real-world and stylized inputs. The experiments report stronger geometry and layout results, show robustness across varied layouts, and test the roles of non-semantic training and the model’s design components.
4.1. Experiment Setting
- The evaluation compares MIDI, SceneGen, PartCrafter, and Gen3DSR using geometry, layout, and inference-time measures on 3D-FRONT, BlendSwap, Scenethesis, and in-the-wild inputs.The geometry measures include scene- and object-level Chamfer Distance and F-Score; layout uses volumetric IoU of scene bounding boxes.Source: §4.1 · p. 5
4.2. Quantitative Results
- 15.51s per instance; I-Scene leads the evaluated metrics and keeps OOD geometry close to ID, while running slower than PartCrafter (7.2 s) but faster than other baselines.The reported benchmark covers scene- and object-level geometry and volumetric IoU; PartCrafter is compared only on scene-level performance.Source: §4.2 · p. 5, Table 1 · p. 6
4.3. Qualitative results
- I-Scene produces clean, separated meshes and plausible support, proximity, and depth ordering across synthetic, in-the-wild, and stylized scenes.The qualitative comparisons report fewer floating or colliding placements and more consistent scale, uprightness, and instance separation across varied layouts.Source: §4.3 · p. 6
4.5. Ablation studies
- Removing SCA, VC, or NS degrades geometry and layout across datasets; view-centric encoding is especially important for coherent layouts and non-semantic scenes improve instance generalization.Without view-centric encoding, the model overfits to 3D-FRONT and exhibits duplicated or blended instances and contact violations; NS improves instance-level completeness and placement.Source: Table 3 · p. 7, §4.5 · pp. 8–9
4.4. Non-semantic random scene experiment
- Non-semantic random training surpasses 3D-FT on OOD scene metrics, while combining random scenes with curated training preserves near-optimal ID layout and achieves the best OOD results.Performance improves when random training scales from 15K to 25K scenes; the findings also indicate that geometric cues alone provide spatial-learning supervision.Source: Table 2 · p. 6, §4.4 · pp. 7–8
5. Conclusion
I-Scene reprograms a pretrained 3D instance generator as a scene-level spatial learner, using view-centric space and model priors to generate scenes without curated annotations.
5. Conclusion
- I-Scene reprograms a pretrained 3D instance generator as a scene-level spatial learner, replacing curated layout supervision with the model’s spatial prior.The formulation distills spatial cues into shared scene tokens that condition instance denoising.Source: §5 · p. 9
- View-centric scene space supports a fully feed-forward interactive scene generator that learns spatial relations from the instance prior.The paper reports that non-semantic synthetic scenes improve geometry, layout, and generalization.Source: §5 · p. 9
- I-Scene performs relatively poorly on tiny-resolution inputs and heavily occluded single-view cases.Proposed future work includes stronger occlusion augmentation, optional multi-view conditioning, and studying random-scene scaling.Source: §5 · p. 9
6. Overview
The supplementary material presents method details, experiment specifications, and qualitative comparisons, including failure cases.
6. Overview
- The supplement derives when scene-context attention is equivalent to standard self-attention and describes collision-free random layout generation.It also provides an HTML page for qualitative visualization and comparison.Source: §6 · p. 10
- Experiment-setting material covers model implementation, training datasets, metrics, and the customized robust ICP metric.Source: §6 · p. 10
- Qualitative results compare baselines on synthetic scenes, real-world and stylized images, and ablations.The supplement also reports failure cases when the instance mask is small, showing inputs, predicted scenes, and low-quality instance examples.Source: §6 · p. 10
7. Method
Scene-context attention conditions instance generation on spatial guidance while retaining the instance model’s prior; the method also specifies collision-free random layouts and supported object stacking.
7. Method
- Scene-context attention conditions instance generation on scene keys and values while making minimal changes to the instance prior.When scene and instance inputs coincide, it reduces to standard self-attention.Source: §7 · p. 10
- Collision-free random layouts sample instances and place them as ground-plane discs using variable-radius Poisson-disk sampling and a global clearance gap.Candidate centers must remain at least the two radii plus the gap apart; a table is placed first when present.Source: §7 · p. 11
- For a stacked object, the procedure extracts a polygon just below the table top, anchors placement at its centroid, and selects a scale that fits within the nearest edge.Source: §7 · p. 11
8. Experiment
The experiment section specifies training and inference settings, describes training data and view generation, and details a robust ICP alignment metric.
8. Experiment
- I-Scene is trained for 130K steps with AdamW at a 5e−5 learning rate and batch size 8 per GPU; inference uses 25 sampling steps and guidance ω = 3.0.Experiments use a cluster with 8 NVIDIA H100 GPUs.Source: §8 · p. 11
- Training uses roughly 24K processed 3D-FRONT scenes and 15K or 25K randomly generated scenes, each containing 2–12 objects.Each scene is rendered from 150 directions; views with fully occluded objects are discarded to avoid memorization or hallucination of invisible objects.Source: §8 · p. 11
- The robust ICP metric addresses unreliable alignments from prior ICP, which the authors attribute to getting stuck in local minima.The procedure uses yaw-sweep initialization, coarse-to-fine alignment, and fallback checks; results are selected across two normalization spaces.Source: §8 · p. 11
9. Qualitative results.
The qualitative results compare I-Scene with baselines across real-world, stylized, and synthetic inputs, and show ablation effects and a failure case.
9. Qualitative results.
- Figure 9 shows that adding SCA, VC, and NS significantly improves layout coherence and instance quality.The passage identifies these as the components added in the ablation study.Source: §9 · p. 12
- Figures 6 and 7 compare I-Scene with all baselines across multiple views using real-world and stylized images as input.Source: §9 · p. 12
- Figure 8 compares I-Scene with all baselines across multiple views using synthetic scene images as input.Source: §9 · p. 12
- Figure 10 shows a failure case in which instance quality is poor when the input instance mask is small in the image.Source: §9 · p. 12