WorldGrow: Generating Infinite 3D World

• Infinite 3D worlds are described as foundational for World Models and embodied AI systems, providing expandable environments for open-ended learning.The paper names navigation, planning, and interaction as activities that such environments support.Source: §1 · p. 2
• Prior 2D-lifting methods lack holistic 3D structure, scene-level 3D methods face limited data, and pretrained 3D models target objects rather than scenes.The paper associates these approaches with cross-view inconsistencies, limited scene-generation scale and generalization, and object-centric applicability, respectively.Source: §1 · p. 2
• Block-based generation must transfer object-level priors to contextually coherent blocks, maintain coherence between adjacent blocks, and preserve global structural plausibility and semantic diversity.Source: §1 · p. 2
• WorldGrow curates ground-truth scene blocks, adapts object-level 3D representations for scene blocks, inpaints missing blocks, and refines coarse layouts with fine geometry and textures.The coarse-trained model builds structure before the fine-trained model refines detail.Source: §1 · p. 2
• The paper presents a scene-block data construction pipeline and datasets intended to support scalable training and evaluation of block-based infinite scene generation.Source: §1 · p. 2

2.1. 3D Generation Pretrained Models

• Feed-forward models using representations such as triplanes and 3D Gaussian Splatting synthesize 3D content from a single image, while TRELLIS introduced SLAT for structured object latents.Extensions revise voxel sampling, and UniLat3D combines geometry and appearance in a compact latent space; scene-level applicability remains underexplored.Source: §2.1 · pp. 2–3

2.2. 3D Unbounded Scene Generation

• Scene-generation methods using 2D inpainting or camera-controlled video diffusion hallucinate multi-view images, while other methods target fuller 360-degree indoor geometry.These approaches typically generate only small-scale scenes, such as a single room.Source: §2.2 · p. 3
• Unbounded geometry methods reconstruct scene blocks with triplane extrapolation, TUDFs, or vector-set latents, but typically lack explicit texture modeling.They instead rely on external texture synthesis and mapping pipelines for realistic surface appearance.Source: §2.2 · p. 3
• SynCity scales scene construction through grid-wise captions, image synthesis, and 3D reconstruction, but rendering fidelity degrades away from diffusion-generation camera poses.Source: §2.2 · p. 3

• TRELLIS is a text/image-conditioned 3D generation model that denoises features in a sparse 3D latent space.Source: §3 · p. 4
• SLATs link latent features to active voxel locations, encoding coarse geometry structure and fine appearance through a Transformer-based variational autoencoder.Source: §3 · p. 4
• The encoder derives local visual features by projecting DINOv2 feature maps from multiview object renders onto voxels and averaging them; the decoder produces 3D representations.Decoded representations include 3D Gaussians, radiance fields, and meshes.Source: §3 · p. 4
• SLAT generation uses two flow-Transformer stages: Stage 1 predicts active voxel centers, and Stage 2 generates their latent features from text or image conditioning.The flow Transformer learns to reverse noise addition.Source: §3 · p. 4

4.1. Task Definition and Overall Framework

• WorldGrow defines a world as an unbounded composition of horizontally aligned blocks, each generated iteratively conditioned on previously synthesized blocks.Its pipeline moves from curated blocks and scene-adapted SLATs through block inpainting to coarse-to-fine generation.Source: §4.1 · p. 4

4.2. Data Curation

• Scene slicing extracts dense, reusable blocks from full scenes, repositioning cuboids until at least 95% of their surface contains visible content.The curated blocks are topologically consistent and intended to reduce unrealistic geometry compared with naive partitioning.Source: §4.2 · p. 4
• Coarse blocks cover four times the XY area of fine blocks at the same height, balancing broader context for layout learning against finer rendering detail.The two block datasets support global layout generation and local detail refinement.Source: §4.2 · pp. 4–5

4.3. Scene-friendly SLAT

• Scene-friendly SLAT better models scene blocks, particularly in occluded areas and near block boundaries.The figure highlights these regions as the focus of the comparison.Source: Figure 3 · p. 5

4.4. 3D Block Inpainting

• 3D block inpainting completes a missing target block from neighboring spatial context, first predicting structure and then reconstructing latent features.The model receives noisy latents, a binary inpainting mask, and the masked known region, conditioning completion on context and explicit missing-area cues.Source: §4.4 · p. 5
• Training divides each block into four quadrants, retaining one as context and masking the other three for inpainting.Structure uses a voxel-level binary mask, while latent inpainting uses a sparse mask to guide feature reconstruction.Source: §4.4 · p. 5

4.5. Infinite Scene Generation

• Generation starts from a seed block and extends across the XY plane, using a coarse model for global structure before fine models recover geometry and SLAT appearance.For each new block, left, top, and top-left neighbors provide context when available; a 3/8w margin is reused to inpaint the central 5/8w × 5/8w region.Source: §4.5 · p. 6
• Fine-level structure is refined from the upsampled coarse layout, while blockwise latent inpainting generates SLATs that are decoded into a renderable 3D world.Structure-guided denoising preserves space-distribution priors while enhancing detail.Source: §4.5 · p. 6

5.1. Experiment Settings

• Training uses 3,425 curated 3D-FRONT houses to generate 120k fine blocks and 38k coarse blocks, with UrbanScene3D used for additional verification.The 3,425 houses comprise 3,072 retained after filtering and 353 manually corrected houses.Source: §5.1 · p. 6

5.2. 3D Scene Generation

• WorldGrow achieves state-of-the-art performance across geometry metrics and significantly better perceptual quality than all baselines for generated scene blocks.The geometry comparison is reported in Table 1, and the visual-fidelity comparison in Table 2.Source: Table 1 · p. 7, Table 2 · p. 7, §5.2 · p. 8
• Human preference scores favor WorldGrow across indoor-scene criteria and unbounded-scene criteria, particularly for structure layout and continuity.The study evaluates structure plausibility, geometry detail, and appearance fidelity for indoor scenes, with continuity added for unbounded scenes.Source: §5.2 · pp. 8–9, Table 3 · p. 9
• WorldGrow maintains consistent quality in distant expansions, while SynCity degrades; SynCity fails in 70% of expansion attempts.The stability experiment evaluates outer-region blocks in generated 7 × 7 scenes; only SynCity's successful cases are reported.Source: §5.2 · p. 9, Table 4 · p. 9

5.3. Ablation Study

• Without data curation, generated objects interpenetrate and arrangements become implausible, whereas curated data produces spatially coherent scenes.Figure 9 illustrates a wardrobe intersecting multiple walls in the uncurated condition.Source: §5.3 · p. 9, Figure 9 · p. 10
• Occlusion-aware aggregation alone degrades reconstruction without decoder retraining, while combining both components improves coherent scene-level reconstruction.The ablation compares the original object-centric VAE, aggregation alone, decoder retraining alone, and both components together.Source: §5.3 · p. 9, Table 5 · p. 10
• Direct fine-scale generation produces implausible furniture arrangements, while coarse-to-fine generation balances global coherence with local realism.The strategy establishes structure at coarse scale before enriching details at fine scale.Source: §5.3 · p. 9, Figure 9 · p. 10

6. Discussion and Future Work

• WorldGrow extends scenes only in the XY plane, leaving vertical expansion needed for multi-story buildings as future work.The authors also identify dataset scale, variety, and semantic annotations as bounds on generation quality and diversity.Source: §6 · p. 10
• The block-wise design trades fine geometric detail for computational feasibility, prioritizing infinite generation over local detail resolution.The current implementation supports conditional control in principle but focuses on unconditional generation without semantic conditioning.Source: §6 · p. 10
• Future directions include multi-level generation, larger outdoor datasets, LLM-generated captions for semantic control, and integration with unified geometry-appearance models.The authors report promising preliminary city-scene experiments and describe integration with unified models as promising for more efficient pipelines.Source: §6 · p. 10

7. Conclusion

• WorldGrow combines block-based context-aware inpainting and coarse-to-fine refinement with pre-trained 3D priors for infinite 3D world generation.The conclusion describes its output as unbounded environments with coherent layouts and photorealistic appearance.Source: §7 · p. 10
• The authors report state-of-the-art performance in geometry reconstruction and visual fidelity, while generating large-scale scenes with local detail and global consistency.Source: §7 · p. 10
• The authors present WorldGrow as a practical path toward scalable, high-quality 3D content generation for future world models and embodied-AI training and simulation.Source: §7 · p. 10

A.1. Coarse-to-Fine Generation

• Each coarse block covers a 2 × 2 grid of fine blocks, allowing the coarse model to establish room-scale layout while the fine model generates detailed geometry.The stages operate at different semantic levels, separating layout reasoning from detail generation rather than representing the same content at different resolutions as LT3SD.Source: §A.1 · p. 14

A.2. Scene-Friendly SLAT

• Vanilla TRELLIS [65] aggregates features along ray-intersected voxels regardless of visibility, which can project visible-surface features onto occluded scene voxels.The passage describes this as producing artifacts and inconsistent appearance in scene contexts.Source: §A.2 · p. 14
• WorldGrow averages DINOv2 features only from views where each voxel is visible, using depth-test visibility masks to prevent feature contamination across occluded surfaces.Source: §A.2 · p. 14
• Occlusion-aware lifting preserves distinct textures on opposite wall sides, producing sharper boundaries and more accurate appearance modeling.Source: §A.2 · p. 14
• The SLAT decoder is retrained on scene-scale data with occlusion-aware features to improve photorealism, preserve geometric boundaries, and support coherent appearance across block transitions.Source: §A.2 · p. 14

B.1. Datasets

• Filtering 3D-FRONT [16] yielded 3,072 clean houses, with 353 additional houses with minor issues selected for manual refinement.Filtering removed houses with mesh penetration, incorrect furniture placement, small layouts, sparse furnishing, and other anomalies.Source: §B.1 · p. 14
• The pipeline extracts 100K fine and 30K coarse blocks from filtered houses, plus 20K fine and 8K coarse blocks from the manually refined subset.Models are trained on the larger filtered set, then fine-tuned on the curated subset.Source: §B.1 · p. 14
• Fine blocks are cubes of width h and coarse blocks span a 2 × 2 grid of fine blocks; both maintain height h, with standard house height h ≈ 3m.Source: §B.1 · p. 14
• For outdoor generalization, WorldGrow is trained on 10K fine and 3K coarse blocks extracted from Shanghai in UrbanScene3D [35].Each fine block covers 100m; Shanghai was selected for architectural diversity and texture quality.Source: §B.1 · p. 14

B.2. Text Prompts

• Indoor and urban-street generation use separate photorealistic scene prompts generated with GPT-4 and encoded with a frozen CLIP [45] text encoder.The prompts describe contemporary houses and daylight urban streets, respectively.Source: §B.2 · pp. 14–15

C.1. Metrics

• FID is computed on 3D mesh data using PointNet++ features, alongside MMD, COV, and 1-NNA implemented following NFD.This differs from LT3SD’s and TRELLIS’s image-based FID evaluation.Source: §C.1 · p. 15

C.2. Compared Methods

• The comparison includes image lifting, block-wise extrapolation, diffusion-based volumetric synthesis, and TRELLIS-family baselines.Baselines include Text2Room, BlockFusion, DiffInDScene, Vanilla TRELLIS, fine-tuned TRELLIS†, and SynCity.Source: §C.2 · p. 15
• LT3SD is excluded for missing reproducibility materials, while WonderWorld is omitted because it does not produce a globally consistent 3D scene.Source: §C.2 · p. 15

D.1. Outdoor 3D Scene Generation

• FID is 23.49 vs. 93.45 for SynCity, while WorldGrow has lower MMD and 1-NNA and higher COV.Table 6 reports the Shanghai evaluation, trained with 10K fine and 3K coarse blocks; each fine block covers 100 m.Source: Table 6 · p. 15, §D.1 · p. 15
• Coarse-to-fine refinement and masked conditioning remain effective in large-scale urban layouts.Source: §D.1 · p. 15
• Because the training subset is limited, the outdoor results are preliminary rather than a conclusive benchmark.Source: §D.1 · p. 15