PaperlayerView exact revision on arXiv

Source-linked AI summary

SceneGen: Single-Image 3D Scene Generation in One Feedforward Pass

Yanxu Meng, Haoning Wu, Ya Zhang, Weidi Xie

arXiv:2508.15769v2cs.CVcs.AI

TL;DR

Generating a scene requires plausible geometry for multiple assets and physically plausible relationships among them, while prior approaches face limitations in flexibility, efficiency, fidelity, or spatial accuracy. SceneGen jointly generates multiple assets’ geometry, texture, and positions in one feedforward pass, and evaluations report better generation quality and efficiency, including multi-image generalization.

  • Problem

    3D scene generation must create multiple assets with accurate geometry, texture, and spatial relationships, including physically plausible support and occlusion.

    Source: §1 · p. 2

  • Method

    SceneGen takes a scene image and object masks, fuses asset- and scene-level visual and geometric features, and decodes multiple assets’ geometry, texture, and relative positions in one feedforward pass.

    Source: §1 · p. 2

  • Results

    Evaluations report significantly better generation quality and efficiency than prior methods; a textured four-asset scene takes approximately 2 minutes on one A100 GPU.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SceneGen also generalizes from single-image training to multi-image input, where it yields better generation quality.

    Source: §1 · p. 2

    A narrow training-data distribution limits generalization beyond indoor scenes, and generated assets can occasionally overlap or have geometric inconsistencies.

    Source: §E.1 · p. 17

Abstract

from arXiv · show

3D content generation has recently attracted significant research interest, driven by its critical applications in VR/AR and embodied AI. In this work, we tackle the challenging task of synthesizing multiple 3D assets within a single scene image. Concretely, our contributions are fourfold: (i) we present SceneGen, a novel framework that takes a scene image and corresponding object masks as input, simultaneously producing multiple 3D assets with geometry and texture. Notably, SceneGen operates with no need for extra optimization or asset retrieval; (ii) we introduce a novel feature aggregation module that integrates local and global scene information from visual and geometric encoders within the feature extraction module. Coupled with a position head, this enables the generation of 3D assets and their relative spatial positions in a single feedforward pass; (iii) we demonstrate SceneGen's direct extensibility to multi-image input scenarios. Despite being trained solely on single-image inputs, our architecture yields improved generation performance when multiple images are provided; and (iv) extensive quantitative and qualitative evaluations confirm the efficiency and robustness of our approach. We believe this paradigm offers a novel solution for high-quality 3D content generation, potentially advancing its practical applications in downstream tasks. The code and model will be publicly available at: https://mengmouxu.github.io/SceneGen.

1. Introduction

SceneGen addresses the coupled challenge of generating plausible asset geometry and physically plausible inter-object arrangements from a scene image. It jointly generates multiple assets’ geometry, texture, and spatial positions in one feedforward pass, with reported quality and efficiency gains and multi-image generalization.

1. Introduction

  • Prior retrieval methods are constrained by asset-library coverage, while two-stage refinement relies on iterative optimization that causes inefficiency and error accumulation.PartCrafter and MIDI also have limited synthesis fidelity and inaccurate spatial relations.Source: §1 · p. 2
  • SceneGen jointly generates multiple assets’ geometry, texture, and positions in one feedforward pass, without per-scene optimization.The paper describes this as the first model capable of doing so.Source: §1 · p. 2
  • Visual and geometric encoders supply asset- and scene-level features, which local and global attention aggregate to support inter-asset interactions and plausible topologies.A position head and pretrained structure decoder produce relative positions, geometry, and texture.Source: §1 · p. 2
  • Despite single-image training, SceneGen generalizes to multi-image inputs with better generation quality, and evaluations report outperforming prior methods in quality and efficiency.It generates a textured four-asset scene in approximately 2 minutes on a single A100 GPU.Source: §1 · p. 2

2. Related Work

Related work progresses from single-asset 3D synthesis to multi-asset scene generation, but existing scene pipelines rely on asset libraries or refinement, and image-conditioned methods can sacrifice fidelity.

2. Related Work

  • Learning-based 3D asset synthesis spans explicit and implicit representations, but remains restricted to single-asset generation rather than complex multi-asset scenes.Later work uses VAEs and hybrid mesh-texture pipelines, including TRELLIS for scalable, high-fidelity generation.Source: §2 · p. 2
  • Text-based scene methods plan layouts and retrieve library assets, while image-based methods use segmentation, scene graphs, or depth and point-cloud alignment to assist generation and arrangement.Optimization-based methods refine structures with vision-language models but suffer from inefficiency.Source: §2 · p. 3
  • MIDI and PartCrafter generate scenes from a single image but sacrifice reconstruction fidelity, whereas SceneGen integrates asset- and scene-level features for robust, efficient generation.Figure 2 contrasts these approaches with SceneGen’s joint one-pass synthesis of geometry, texture, and spatial positions.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

SceneGen casts multi-object scene synthesis as a single-stage feedforward mapping from a scene image and masks to asset representations and query-relative poses.

3.1. Problem Formulation

  • Each asset’s position is represented relative to a selected query asset by translation, rotation, and scale.The formulation takes a scene image and corresponding object masks as input and generates asset structure and texture representations.Source: §3.1 · p. 3

3.2. SceneGen

  • Visual and geometric encoders supply local and scene features, which local attention refines and global attention links across assets while incorporating scene geometry.The global block uses scene-level self-attention for inter-object relationships and cross-attention to integrate geometric context.Source: §3.2 · pp. 3–4
  • A position head predicts relative asset poses, while off-the-shelf sparse-structure and structured-latent decoders produce geometry and texture.The decoded poses are 8D vectors relative to the query asset.Source: §3.2 · pp. 3, 5

3.3. Training

  • Training uses flow-matching, relative-position, and voxel-collision losses; only global attention blocks, learnable position tokens, and the position head are optimized.Translation error is normalized by scene scale to stabilize the position loss across query-asset configurations.Source: §3.3 · p. 5

3.4. Extension to Multi-view Inputs

  • Despite single-image training, multi-view inference extracts visual features per view, aggregates views for geometric features, and averages pose predictions across views.The authors report that this inference scheme improves generation quality through better geometric understanding without multi-view fine-tuning.Source: §3.4 · p. 5

4. Experiments

The experiments compare SceneGen with prior methods using geometric, visual, and efficiency evaluations, followed by qualitative comparisons and component ablations. SceneGen is reported to outperform prior methods and to extend to multi-view inputs.

4.1. Experimental Settings

  • SceneGen is evaluated on 4.8K 3D-FUTURE test scenes using geometry, texture-rendering, and inference-time measures, with baseline comparisons adjusted for available controls and texture rendering.Geometry uses point-cloud CD, F-Score, and bounding-box volumetric IoU; texture uses PSNR, SSIM, LPIPS, FID, CLIP, and DINOv2 similarity.Source: §4.1 · p. 6

4.2. Quantitative Results

  • SceneGen consistently outperforms existing methods across scene-level and asset-level metrics, and generates a four-asset textured scene within about 2 minutes.The paper also reports that SceneGen outperforms baselines across metrics despite possible overlap between some baselines’ 3D-FRONT training data and the test data.Source: §4.2 · pp. 6–7

4.3. Qualitative Results

  • On 3D FUTURE and ScanNet++, SceneGen is described as producing more accurate geometry and visual quality than prior methods, with precise spatial relationships and detailed textures.The comparisons note that PartCrafter can merge distinct assets and lacks mask-based controllability, while PartCrafter and DepR do not render textures.Source: Figure 4 · p. 7, §4.3 · p. 7
  • Multi-view inputs yield more complete geometry and finer texture details than single-image inputs, although the evaluation is qualitative because suitable quantitative datasets are unavailable.SceneGen was trained exclusively on single-image samples.Source: §4.3 · pp. 7–8, Figure 5 · p. 8

4.4. Ablation Studies

  • Removing any tested feature component degrades overall performance; geometric features chiefly affect structure, visual features affect visual quality, and scene-level attention supports inter-asset interactions.The ablations progressively remove global geometric features, global visual features, and mask visual features, or replace scene-level with asset-level self-attention.Source: §4.4 · p. 8, Table 2 · p. 8

5. Conclusion

The conclusion presents SceneGen as a single-pass framework for generating multiple assets and their relative positions. The preliminaries describe how TRELLIS and VGGT provide complementary asset and scene representations.

5. Conclusion

  • SceneGen takes a scene image and object masks to synthesize multiple assets’ structure, texture, and relative positions in one feedforward pass, with reported gains in generation quality and efficiency.The conclusion also reports that the framework generalizes to multi-image inputs with better generation fidelity.Source: §5 · p. 8

A. Preliminaries on 3D Foundation Models

  • TRELLIS represents an asset’s geometry and appearance with voxel positions paired with local latent features, and generates sparse structure before conditioned texture and appearance features.Its sparse structure generator and structured-latents generator are trained with a conditional flow-matching objective.Source: A. Preliminaries on 3D Foundation Models · p. 13
  • VGGT extracts scene geometric features from single- or multi-view RGB inputs, with its global- and local-attention features decoded into depth maps, point maps, and tracks.Source: A. Preliminaries on 3D Foundation Models · p. 13
  • SceneGen combines local asset-level and global scene-level features from complementary visual and geometric priors to address 3D scene generation.The framework is built on TRELLIS and uses DINOv2 and VGGT as visual and geometric encoders, respectively.Source: A. Preliminaries on 3D Foundation Models · p. 13

B. More Details about Training Data

Training uses 3D-FUTURE scenes, with augmentation and filtering choices intended to support learning spatial relationships and data quality.

B. More Details about Training Data

  • SceneGen is trained on 3D-FUTURE, whose rich textures and diverse lighting conditions are used to simulate real-world environments.Source: B. More Details about Training Data · p. 13
  • For scenes with more than seven assets, training randomly selects a seven-asset subset, and augmentation iteratively queries each asset while shuffling the remaining assets.The maximum scene size is set to seven assets on a single A100 GPU.Source: B. More Details about Training Data · p. 13
  • Assets with aesthetic scores below 4.5 are excluded using TRELLIS’s filtering criterion, and the paper illustrates the training scenes’ asset-count distribution in Figure 6.Source: B. More Details about Training Data · p. 13

C. More Implementation Details

The implementation-details appendix identifies two topics: extending SceneGen to multi-image inputs and describing evaluation protocols.

C. More Implementation Details

C.1. Extension to Multi-image Inputs

SceneGen extends from single-image training to multi-view inference without additional training or fine-tuning. Multi-view results are presented qualitatively because suitable training and quantitative evaluation data are lacking.

C.1. Extension to Multi-image Inputs

  • At inference, SceneGen can accept same-scene images from multiple viewpoints without additional training or fine-tuning.The geometric encoder integrates information across viewpoints, while asset and mask order must remain consistent; relative positions are averaged across views.Source: §C.1 · pp. 13–14
  • Multi-view results are qualitative only, as suitable training data and quantitative evaluation methods are currently lacking.The paper shows scenes sampled from ScanNet++ and leaves multi-view dataset and evaluation construction to future work.Source: §C.1 · p. 14

C.2. Evaluation Protocols

The evaluation combines geometric and visual protocols with an inference-time comparison, and additional visualizations contrast SceneGen with representative baselines.

C.2. Evaluation Protocols

  • SceneGen consistently outperforms MIDI under both ICP and FilterReg alignment strategies.The comparison concerns geometric metrics; FilterReg also yields better overall performance for both methods than ICP.Source: §C.2 · p. 15
  • Visual metrics compare aligned, identically rendered predictions against instance-masked images and ground-truth renders with complementary discrepancies.Instance-masked images can differ because of occlusion, while ground-truth renders lack scene-level illumination and complete textures.Source: Figure 7 · p. 15, §C.2 · p. 16
  • SceneGen generates a 4-asset 3D scene in under 2 minutes on one A100 GPU in a single feedforward pass.The comparison protocol reports average single-asset inference time over 500 trials on the same hardware.Source: §C.2 · p. 16

D. More Visualizations

  • Compared with baselines, SceneGen produces plausible geometry and detailed textures, while baseline methods share incorrect asset relationships.PartCrafter often misses or mixes assets and PartCrafter and DepR generate geometry without texture details.Source: D. More Visualizations · pp. 16–17

E.1. Limitations

SceneGen’s stated limitations concern its indoor training distribution, object interactions, and dependence on segmentation masks; the paper proposes data, modeling, and segmentation extensions.

E.1. Limitations

  • The narrow training distribution limits SceneGen’s generalization to non-indoor scenes.The proposed response is larger-scale training data covering more diverse indoor and outdoor scenarios.Source: §E.1 · p. 17, §E.2 · p. 17
  • Without explicit spatial or physical constraints, SceneGen can produce asset overlaps or geometric inconsistencies.The paper proposes incorporating physical priors or constraints to better learn complex object interactions.Source: §E.1 · p. 17, §E.2 · p. 17
  • SceneGen requires target-object masks, which limits direct use on in-the-wild data and may reduce robustness with low-quality masks.The current framework uses ground-truth masks or masks extracted by SAM 2; proposed directions include adding segmentation or integrating it natively.Source: §E.1 · p. 17, §E.2 · p. 17

E.2. Future Works

  • Future work includes multi-view scene-generation datasets to expand the application scope of existing models.The paper also proposes broader scene datasets, explicit physical priors, and integrated object segmentation.Source: §E.2 · p. 17
Loading 2508.15769v2…