PaperlayerView exact revision on arXiv

Source-linked AI summary

BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation

Zhennan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, Pan Ji

arXiv:2401.17053v4cs.CVcs.AIcs.GR

TL;DR

Existing 3D-generation methods mainly produce content with fixed spatial extent, while expandable scenes require high-fidelity shapes and coherent transitions. BlockFusion generates and extends block-based scenes using latent tri-plane diffusion and layout conditioning. The authors report diverse, geometrically consistent, unbounded scenes with high-quality shapes across indoor and outdoor scenarios.

  • Problem

    Existing methods mainly generate 3D content with fixed spatial extent, while expandable-scene generation must address high-fidelity scene shapes and harmonious transitions between existing and new regions.

    Source: §1 · p. 2

  • Method

    BlockFusion fits cropped scene blocks to tri-planes, compresses them into a latent space for diffusion, and extends scenes by conditioning new blocks on overlapping tri-plane features.

    Source: §1 · p. 2

  • Results

    BlockFusion generates diverse, geometrically consistent, unbounded large 3D scenes with high-quality shapes in indoor and outdoor scenarios.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that blend with existing scenes, while 2D layouts control element placement and arrangement.

    Source: §1 · p. 3

    The current implementation may fail to generate very fine geometric details, such as chair legs, because of limited tri-plane resolution.

    Source: §5 · p. 13

Abstract

from arXiv · show

We present BlockFusion, a diffusion-based model that generates 3D scenes as unit blocks and seamlessly incorporates new blocks to extend the scene. BlockFusion is trained using datasets of 3D blocks that are randomly cropped from complete 3D scene meshes. Through per-block fitting, all training blocks are converted into the hybrid neural fields: with a tri-plane containing the geometry features, followed by a Multi-layer Perceptron (MLP) for decoding the signed distance values. A variational auto-encoder is employed to compress the tri-planes into the latent tri-plane space, on which the denoising diffusion process is performed. Diffusion applied to the latent representations allows for high-quality and diverse 3D scene generation. To expand a scene during generation, one needs only to append empty blocks to overlap with the current scene and extrapolate existing latent tri-planes to populate new blocks. The extrapolation is done by conditioning the generation process with the feature samples from the overlapping tri-planes during the denoising iterations. Latent tri-plane extrapolation produces semantically and geometrically meaningful transitions that harmoniously blend with the existing scene. A 2D layout conditioning mechanism is used to control the placement and arrangement of scene elements. Experimental results indicate that BlockFusion is capable of generating diverse, geometrically consistent and unbounded large 3D scenes with unprecedented high-quality shapes in both indoor and outdoor scenarios.

1 INTRODUCTION

BlockFusion addresses the challenge of generating high-fidelity, expandable 3D scenes whose new regions remain semantically and geometrically coherent. It uses latent tri-plane diffusion to generate and extend block-based scenes, with 2D layouts controlling element placement.

1 INTRODUCTION

  • Existing methods mainly generate 3D content with fixed spatial extent, while expandable scenes require both high-fidelity scene-level shapes and harmonious transitions between old and new regions.Text2Room generates expandable scenes at room scale, but its camera-based expansion is difficult to extend beyond that scale because of occlusion.Source: §1 · p. 2
  • BlockFusion generates scenes as blocks and expands them by conditioning latent tri-plane extrapolation on overlapping features, producing transitions that blend with the existing scene.New empty blocks overlap the current scene; feature samples from overlapping tri-planes condition generation during reverse diffusion.Source: §1 · pp. 2–3
  • An auto-encoder compresses fitted raw tri-planes into a latent space for diffusion, addressing collapsed shapes observed when diffusion is trained directly on raw tri-planes.Training blocks are randomly cropped from complete 3D scenes and converted to tri-planes through per-block fitting.Source: §1 · p. 2
  • A 2D layout-conditioning mechanism lets users control element placement and arrangement by manipulating object bounding boxes.The paper also demonstrates creating scene color and texture with an off-the-shelf texture-generation tool.Source: §1 · p. 3
  • Experiments indicate that BlockFusion generates diverse, geometrically consistent, unbounded large 3D scenes with high-quality shapes in indoor and outdoor scenarios.The generated meshes can also be integrated with off-the-shelf texture-generation tools.Source: §1 · p. 3

2 RELATED WORK

The related work reviews diffusion-based generation and 3D representations, then situates BlockFusion among methods for generating scenes. It highlights latent tri-plane diffusion and progressive scene expansion as the paper’s approach to unbounded scenes.

2.1 Diffusion models

  • Latent diffusion compresses inputs with an auto-encoder before diffusion, reducing the spatial representation used for generation and supporting conditioned image synthesis.The section contrasts adding controls through training or fine-tuning with methods that modify generation at inference.Source: §2.1 · p. 3

2.2 3D shape generation

  • Direct 3D diffusion uses meshes, point clouds, grids, or neural fields; hybrid tri-plane fields are adopted for computational efficiency.Prior tri-plane diffusion work used restricted datasets, while BlockFusion compresses tri-planes into a latent space to improve stability, generalizability, and output quality.Source: §2.2 · pp. 3–4
  • BlockFusion’s training pipeline fits blocks to raw tri-planes, compresses them into latent tri-planes, then trains DDPM, with layout control integrated during diffusion.The paper presents latent tri-plane compression as a change to prior tri-plane diffusion approaches.Source: Figure 2 · p. 4, §2.2 · p. 4

2.3 3D scene generation

  • Scene-generation methods include object retrieval, image lifting, layout-conditioned room generation, and panorama or height-map approaches; some depend on input shapes that bound scene dimensions.BlockFusion addresses unbounded scene generation with an auto-regressive expansion algorithm based on tri-plane diffusion.Source: §2.3 · p. 4

3 METHOD

BlockFusion generates scenes as blocks and expands them progressively; its pipeline crops scene meshes, fits and compresses tri-planes, and uses latent diffusion for generation and extrapolation. It also refines seams and builds large scenes from blocks.

3.1 Crop training scenes into 3D blocks

  • Randomly cropped cubic blocks can split objects and contain widely varying arrangements; ground-plane category layouts are cropped alongside the meshes for conditioning.The tested scene types include rooms, cities, and villages.Source: §3.1 · pp. 4–5

3.2 Raw tri-plane fitting

  • A tri-plane stores geometry features on XY, YZ, and XZ planes, while an MLP decodes them into signed distance values for reconstructing each block.After fitting, the resulting raw tri-planes faithfully reconstruct the training blocks.Source: §3.2 · p. 5

3.3 Compressing to latent tri-plane space

  • Direct diffusion on raw tri-planes collapses, so an autoencoder compresses them into a lower-resolution latent tri-plane space that retains the three-plane structure.The authors prioritize geometry loss in training the autoencoder for faithful shape representation.Source: §3.3 · pp. 5–6

3.4 Latent Triplane Diffusion

  • Layout conditioning converts object bounding-box projections into a feature map and concatenates it to the noisy latent planes, controlling arrangement while preserving shape variation.The layout map has a channel for each object category.Source: §3.4 · p. 7
  • Diffusion denoises latent tri-planes; its U-Net uses transformer self-attention for cross-plane communication, then decodes the output into raw tri-planes and SDF shapes.The architecture unfolds planes for convolutions, concatenates their features as tokens for attention and residual blocks, then reassembles the planes.Source: §3.4 · pp. 6–7, Figure 5 · p. 7

3.5 Latent tri-plane Extrapolation

  • To extrapolate an overlapping block, BlockFusion synchronizes denoising with noisy features from the known block in overlapping planes; attention propagates information to the plane without explicit overlap.Resampling rolls denoising back at selected steps, and the reported ablation found that more resampling improved generation performance.Source: §3.5 · pp. 7–8

3.6 Surface refinement with non-rigid registration

  • After latent synchronization, non-rigid registration aligns the extracted meshes in the overlap while preserving the new block’s structure outside it.The registration cost uses Chamfer Distance and a dense non-rigid warping function.Source: §3.6 · p. 8

3.7 Building unbounded large scenes with BlockFusion.

  • For large scenes, independent non-overlapping seed blocks are generated in parallel, then the remaining overlapping blocks are extrapolated from them in parallel.This avoids the significant time required by serial block-by-block expansion.Source: §3.7 · p. 8

4 EXPERIMENTAL RESULTS

BlockFusion is evaluated on room, city, and village scenes, with comparisons and ablations testing its block-generation quality, latent representation, layout control, and scene expansion. Results include improvements over baselines, consistent transitions with resampling, and generation of large scenes conditioned on layouts.

4.1 Implementation details. / 4.2 Evaluation Metrics

  • Training uses randomly cropped room, city, and village blocks; evaluation includes reconstruction, unconditioned-generation, and user-study metrics.The room data yields 57K crops and 9,123 rooms; city and village datasets each yield 10K blocks. The user study involved 48 participants.Source: §4.1 · p. 9, §4.2 · p. 10

4.3 Comparison with SOTA.

  • Coverage increases over NFD by 29.17% under CD and 23.67% under EMD for unconditioned indoor block generation.NFD is retrained on indoor scene blocks; the authors also report that NFD cannot generate meaningful shapes.Source: §4.3 · p. 10, Table 3 · p. 11
  • Under a five-point system, BlockFusion leads by 2.52 points in geometric perceptual quality and 2.66 in geometric structure completeness; for textured generation, it leads by 1.81 points in textured perceptual quality and 1.88 in textured structure completeness.Source: §4.3 · p. 11

4.4 Ablation study

  • Latent tri-planes with 2 channels reduce data bits by 99.6% versus raw tri-planes while retaining decent shape representation; latent diffusion raises Coverage by 27.84% under CD and 25.83% under EMD.At a similar compression rate, the 4096-dimensional latent vector produces no reasonable shape, while raw tri-plane diffusion also produces no reasonable generation results.Source: §4.4 · p. 11, Figure 10 · p. 11
  • Layout conditioning generally guides object placement but not intricate shape details, while more resampling lowers overlap-region Chamfer Distance and stabilizes after three resamplings.Without synchronization, blocks generated independently under shared layouts have extremely high Chamfer Distance.Source: §4.4 · p. 12, Figure 12 · p. 13
  • Latent extrapolation produces meaningful transitions but can leave minor seams because the auto-encoder abstracts high-frequency details; non-rigid registration mitigates them.Figure 13 compares extrapolation before and after registration.Source: §4.4 · p. 13, Figure 13 · p. 13
  • BlockFusion can recombine existing elements under layout guidance into shapes absent from the training set, including a table shaped like “24” and a heart-shaped room.The authors describe the resulting scene elements as diverse and visually appealing.Source: §4.4 · p. 13, Figure 14 · p. 13

4.5 Large Scene Generation.

  • Layout-conditioned generation produces village, city, and room scenes at large scale, with scene scope described as infinitely expandable.Generating the large indoor scene shown in Fig. 7 takes around three hours.Source: Figure 7 · p. 9, §4.1 · p. 10, §4.5 · p. 13

5 CONCLUSION AND DISCUSSION

BlockFusion generates diverse, geometrically consistent, unbounded large scenes in indoor and outdoor settings, and is integrated into a Unity open-world game. The discussion identifies limits in fine-detail generation, object orientation control, and globally consistent texturing of large meshes.

5 CONCLUSION AND DISCUSSION

  • BlockFusion generates diverse, geometrically consistent, unbounded large 3D scenes with high-quality geometry in indoor and outdoor scenarios.Generated meshes can be integrated with off-the-shelf texture-generation tools for visually pleasing textured results.Source: §5 · p. 13
  • A Unity integration demonstrates BlockFusion as an open-world game map generator where players can roam without a predetermined world boundary.The paper points to a supplementary video demo.Source: §5 · p. 13
  • Unlike procedural content generation systems that require expertise and style-specific rules, BlockFusion learns scene distributions from data and automates generation.The paper describes its approach as rule-programming-free and easy to use.Source: §5 · p. 13
  • The current implementation may miss very fine geometric details, such as chair legs, primarily because of limited tri-plane resolution.Tri-plane super-resolution is proposed as a possible solution.Source: §5 · p. 13
  • Bounding-box conditioning controls approximate object placement but not orientation; globally consistent texturing for large scene meshes remains future work.The authors suggest conditioning on both bounding-box and object-orientation maps for precise orientation control.Source: §5 · p. 14

A QUALITATIVE ROOM GENERATION

The supplementary room-generation results include qualitative examples and quantitative indoor textured-shape results. The paper reports that Text2Room generates distorted shapes, while Meshy-generated textures may not correspond well to the input prompts.

A QUALITATIVE ROOM GENERATION

  • The study validated Meshy coloring of generated geometry for Text2Room and included nine people.Additional qualitative and quantitative results are referenced in Fig. 15 and Table 5.Source: A QUALITATIVE ROOM GENERATION · p. 16
  • Text2Room generates distorted shapes, and Meshy-generated textures may not correspond well to input prompts.These are the qualitative issues noted for the supplementary room-generation results.Source: Figure 15 · p. 16
  • Table 5 presents quantitative indoor scene-generation results for textured shapes.The supplied passage identifies the table's scope but gives no result values.Source: Table 5 · p. 16

B VAE STRUCTURE

The VAE compresses raw tri-planes into latent tri-plane space using multi-scale feature aggregation and cross-plane interactions. Its planes are processed separately by convolutions and jointly by transformer layers.

B VAE STRUCTURE

  • The architecture description gives ResNet block parameters as input and output channel counts, transformer parameters as channel counts, and tensor shapes as batch, channel, height, width.C denotes the number of channels compressed into latent tri-plane space.Source: B VAE STRUCTURE · p. 16
  • Feature Pyramid Networks aggregate multi-scale features, while transformer layers provide cross-plane dependence in the VAE.The tri-planes are unfolded into three independent planes for separate convolutional processing.Source: B VAE STRUCTURE · p. 16
  • The tri-planes are unfolded into three independent planes so convolutions can run separately on each plane.Transformer layers are then used to achieve cross-plane dependence.Source: B VAE STRUCTURE · p. 16
Loading 2401.17053v4…