PaperlayerView exact revision on arXiv

Source-linked AI summary

MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation

Zehuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, Lu Sheng

arXiv:2412.03558v3cs.CV

TL;DR

Single-image scene generation must infer 3D geometry and spatial relationships from limited visual clues, and existing approaches have accuracy or generalization constraints. MIDI extends pretrained object generators to jointly generate related instances using multi-instance attention. The paper reports state-of-the-art performance and strong generalization across synthetic, real-world, and stylized scenes.

  • Problem

    A single image provides limited spatial clues for inferring each object's 3D geometry and the relationships among scene instances.

    Source: §1 · pp. 1–2

  • Method

    MIDI extends pretrained image-to-3D object generators to jointly denoise multiple instances, using multi-instance attention and global-scene and local-object image conditioning.

    Source: §1 · p. 2, §4.1 · p. 4

  • Results

    MIDI achieves state-of-the-art performance with accurate geometries and spatial layouts, and the paper reports evaluations on synthetic, real-world, and stylized scenes.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Single-object data are incorporated during training to retain the pretrained model's generalization ability despite limited scene-level data.

    Source: §1 · p. 2

    Existing scene datasets contain relatively simple interaction relationships, so MIDI may struggle with intricate interactions such as dynamic interplays.

    Source: §10 · p. 14

Abstract

from arXiv · show

This paper introduces MIDI, a novel paradigm for compositional 3D scene generation from a single image. Unlike existing methods that rely on reconstruction or retrieval techniques or recent approaches that employ multi-stage object-by-object generation, MIDI extends pre-trained image-to-3D object generation models to multi-instance diffusion models, enabling the simultaneous generation of multiple 3D instances with accurate spatial relationships and high generalizability. At its core, MIDI incorporates a novel multi-instance attention mechanism, that effectively captures inter-object interactions and spatial coherence directly within the generation process, without the need for complex multi-step processes. The method utilizes partial object images and global scene context as inputs, directly modeling object completion during 3D generation. During training, we effectively supervise the interactions between 3D instances using a limited amount of scene-level data, while incorporating single-object data for regularization, thereby maintaining the pre-trained generalization ability. MIDI demonstrates state-of-the-art performance in image-to-scene generation, validated through evaluations on synthetic data, real-world scene data, and stylized scene images generated by text-to-image diffusion models.

1. Introduction

A single image provides limited clues for recovering object geometry and relationships, while existing reconstruction, retrieval, and staged-generation methods face accuracy, generalization, or pipeline challenges. MIDI extends pretrained object generators to generate multiple spatially related 3D instances together.

1. Introduction

  • Single-image scenes are difficult to reconstruct because partial views provide limited spatial clues for inferring 3D geometry and relationships.The task requires extensive prior knowledge of the 3D visual world.Source: §1 · p. 1
  • Reconstruction and retrieval methods lack accuracy and out-of-domain generalization for novel shapes and unseen layouts.Scene-level supervision is scarce, while single images provide limited clues for selecting and arranging database models.Source: §1 · pp. 1–2
  • Multi-step compositional pipelines are lengthy and prone to accumulated errors, while per-object generation without global context can misalign scene relationships.MIDI’s simultaneous-generation approach is presented as an alternative to that staged pipeline.Source: §1 · p. 2, Figure 2 · p. 2
  • MIDI extends pretrained 3D object generators with multi-instance attention to generate multiple instances simultaneously with spatial relationships.It supervises inter-instance relationships with limited scene-level data and uses single-object data to retain pretrained generalization.Source: §1 · p. 2
  • MIDI achieves state-of-the-art 3D scene-generation performance, with accurate inter-object relationships and better alignment to the input.Evaluations include synthetic datasets, real-world scenes, and stylized images generated by text-to-image diffusion models.Source: §1 · p. 2

2. Related Work

Prior scene-reconstruction methods use feed-forward prediction, database retrieval, or staged compositional generation, each with constraints on data, retrieval, or global context. MIDI builds on pretrained image-to-3D generation to produce related instances together.

2.1. Scene Reconstruction from a Single Image

  • Feed-forward reconstruction predicts scene properties from a single image but is limited by scarce 3D scene supervision and can generalize poorly to out-of-distribution images.These methods use end-to-end encoder-decoder systems trained with 3D supervision.Source: §2.1 · p. 3
  • Retrieval-based methods align database models to an image, but depend on database diversity and can misalign objects when a single view provides insufficient information.Their use of existing 3D assets can produce detailed objects.Source: §2.1 · p. 3
  • Recent compositional methods use pretrained image and 3D models in multi-stage pipelines, but per-object processing without global context can accumulate errors and misalign results.MIDI instead generates multiple instances together using a pretrained image-to-3D model.Source: §2.1 · p. 3

2.2. 3D Object Generation from a Single Image

  • Image-to-3D object generators produce high-quality geometry from single-object images, using either multi-view reconstruction or native 3D generation.MIDI fine-tunes an object geometry generator for compositional instances while retaining generalization ability.Source: §2.2 · p. 3

3. Preliminary: 3D Object Generation Models

Large-scale 3D object-generation models encode geometry into a low-dimensional latent space and use a denoising transformer to generate latent samples. Condition encoders supply image or text information to the denoiser.

3. Preliminary: 3D Object Generation Models

  • A transformer-based variational autoencoder encodes 3D geometry into a low-dimensional latent space and decodes it back into geometric representations.The components include an encoder E and decoder D.Source: §3 · p. 3
  • A denoising transformer learns to transform noise in the compressed latent space into the 3D data distribution.The model is trained on the compressed latent space.Source: §3 · p. 4
  • At inference, the decoder produces geometric representations that can be converted into a 3D mesh.The passage names marching cubes or an additional mapping network as conversion options.Source: §3 · p. 4

4. MIDI: Multi-Instance 3D Generation

MIDI extends object diffusion to generate multiple spatially correlated instances from a scene image. It shares a denoiser across instances, uses attention to model their interactions, and trains with scene-level and single-object data.

4. MIDI: Multi-Instance 3D Generation

  • MIDI generates spatially correlated latent tokens for the instances in a scene image, which can be decoded and combined into a 3D scene.The paper frames this as lifting object generation to compositional instance generation.Source: §4 · p. 4

4.1. Multi-Instance Diffusion Models

  • The framework denoises multiple instance latents in parallel with a shared network, adding multi-instance attention and image conditioning.The conditioning uses local object images and global scene context.Source: §4.1 · p. 4
  • Cross-attention integrates image features that encode local instance details and locations alongside global scene information.The conditioning representation combines each object's RGB image and mask with the global scene image.Source: §4.1 · p. 4

4.2. Multi-Instance Attention

  • Multi-instance attention connects instances in the denoising process so that each instance can attend to features from all instances, including itself.This changes generation from independent object processes into a synchronous interactive process intended to capture spatial dependencies.Source: §4.2 · pp. 4–5, Figure 4 · p. 4

4.3. Training

  • Training extends the base rectified-flow loss to multiple instances, sampling a shared noise level and perturbing their latents along a linear trajectory.The loss fine-tunes the denoising network and image encoder.Source: §4.3 · p. 5
  • Because scene-level training data are smaller than the object-model pretraining data, MIDI also trains on Objaverse as a single-object generator with multi-instance attention turned off.This occurs with a 30% chance to retain the original generalization capability.Source: §4.3 · p. 5

5. Experiments

MIDI is evaluated on synthetic and real-world scene benchmarks, stylized images, and component ablations. The experiments report strong comparative results and examine how attention depth, scene context, and mixed training affect generation.

5.1. Setup

  • MIDI was trained on approximately 15,000 cleaned 3D-Front scenes and evaluated across synthetic and real-world reconstruction benchmarks using scene and object geometry, layout accuracy, and per-scene runtime metrics.Its test set contains 1,000 randomly selected scene images; comparisons include feed-forward reconstruction, retrieval-based, and compositional-generation methods.Source: §5.1 · p. 5

5.2. Scene Generation on Synthetic Data

  • MIDI achieves the best performance among state-of-the-art methods across all evaluated metrics on synthetic datasets.The reported comparisons cover scene- and object-level geometry, object-layout alignment, and runtime.Source: §5.2 · p. 5, Table 1 · p. 6

5.3. Scene Generation from Real Images

  • On real images, MIDI significantly outperforms previous works in accuracy and completeness in a qualitative comparison of 10 sampled scenes.The scenes are from Matterport3D and ScanNet.Source: §5.3 · pp. 6–7, Figure 6 · p. 7

5.5. Ablation Study

  • The ablations report that K = 5 performs best, removing global scene context impairs coherent placement, and Objaverse mixed training preserves object geometry.With K = 0, layouts are incoherent; K = 21 overfits and distorts geometry. The authors choose K = 5 to balance interaction modeling and preservation of the pretrained prior; without single-object data, the model overfits the smaller scene dataset.Source: §5.5 · pp. 7–8, Table 2 · p. 7

5.4. Scene Generation from Stylized Images

  • Figure 8 presents qualitative comparisons of MIDI on stylized scene images generated by text-to-image diffusion models.The figure caption identifies the stylized inputs; this passage does not report a quantitative result.Source: Figure 8 · p. 8

6. Conclusion

MIDI extends pre-trained image-to-3D generation to multi-instance diffusion, modeling inter-object interactions and spatial coherence during generation. The paper reports state-of-the-art performance and strong generalization, while noting weaker performance on tiny-resolution inputs and complex interactions.

6. Conclusion

  • MIDI performs relatively poorly on tiny-resolution image inputs and complex inter-instance interactions.The paper identifies modeling more complex interactions as future work, noting that this requires specialized datasets.Source: §6 · p. 8
  • MIDI achieves state-of-the-art performance in single-image 3D scene generation while exhibiting strong generalization.The paper reports high-quality scenes with precise geometries and spatial layouts.Source: §6 · p. 8
  • Multi-instance attention captures inter-object interactions and spatial coherence, enabling simultaneous generation of instances with accurate spatial relationships.The approach extends pre-trained image-to-3D object generation models to multi-instance diffusion.Source: §6 · p. 8

7. Background

The base model compresses 3D geometry into a latent space, then trains a denoising network to transform noise toward the original 3D data distribution. Image conditioning enters the denoising transformer through cross-attention.

7. Background

  • A VAE maps 3D point positions and normals into a low-dimensional latent space and decodes the latents into signed distance function values.The VAE uses several transformer blocks.Source: §7 · p. 13
  • The denoising network transforms noise into the original 3D data distribution by learning the slope of a rectified-flow trajectory.The trajectory linearly perturbs original data z0 with noise; logit-normal sampling increases the weight of intermediate steps.Source: §7 · p. 13
  • The training objective minimizes squared error between z0 − ϵ and the denoiser prediction conditioned on the image encoding.The image encoder τθ encodes conditioning image y, which enters the denoising transformer through cross-attention.Source: §7 · p. 13

8. Implementation Details

MIDI is trained to generate up to seven instances, with scenes containing more than five objects represented by five selected objects. Inference segments the scene image before generating a 3D scene in approximately 40 seconds on an NVIDIA A100 GPU.

8. Implementation Details

  • MIDI is trained for up to seven instances; scenes with more than five objects contribute five representative objects selected by clustering.Five or fewer objects constitute the majority of scenes in 3D-FRONT; image conditioning is randomly dropped with probability 0.1.Source: §8 · p. 13
  • Table 3 reports training costs, and its caption specifies that batch size is set to 1.Source: Table 3 · p. 13
  • Training uses a learning rate of 5 × 10^-5 for five epochs on eight NVIDIA A100 GPUs.The model uses logit-normal sampling to increase the weight of intermediate diffusion steps.Source: §8 · p. 13
  • Inference segments scene images with Grounded-SAM, then uses multi-instance diffusion with classifier-free guidance for 50 steps at guidance scale 7.0.Generating a 3D scene takes approximately 40 seconds on an NVIDIA A100 GPU.Source: §8 · p. 13

9. Additional Discussions

MIDI jointly models object completion, 3D generation, and spatial relationships, in contrast to compositional methods that generate objects sequentially and optimize their layout. The discussion also reports manageable training costs at seven instances and describes a separate pipeline for adding textures.

9. Additional Discussions

  • Existing methods generate objects one by one and optimize their spatial relationships, whereas MIDI jointly models object completion, 3D generation, and layout.The discussion notes that sequential generation lacks global scene context and can reverse similar objects' positions; Figure 9 presents a detailed comparison.Source: §9 · p. 13, Figure 9 · p. 14
  • GPU memory and training time rise with instance count, but resource use remains manageable at N = 7.Table 3 presents training costs for different instance counts.Source: §9 · p. 13
  • For textured scenes, MIDI first generates geometry, then MV-Adapter generates each instance's texture using its partial image.Figure 10 shows visualizations of textured 3D scene generation with MV-Adapter.Source: §9 · pp. 13–14, Figure 10 · p. 14

10. Limitations

MIDI has two stated limitations: small objects may be lower resolution in the global scene coordinate system, and simple interaction relationships in existing datasets constrain its ability to generate intricate interactions.

10. Limitations

  • In the normalized global scene space, small objects may have lower resolution than in canonical spaces, where the model can focus its full capacity on one object.Generating objects in canonical spaces alongside their scene positions could address this issue.Source: §10 · p. 14
  • Because existing scene datasets contain simple interaction relationships, MIDI may struggle with intricate interactions such as objects with dynamic interplays.The authors anticipate that more complex and diverse training data would enhance generalization to object spatial interactions.Source: §10 · p. 14

Related papers

Loading 2412.03558v3…