Source-linked AI summary
Scaling 3D Generative Priors to Large-Scale Scene Meshes from Multi-View Images
SangEun Lee, Wonseok Chae, Hoyoung Yoo, Geunyong Kim, NackWoo Kim, Hyeonjin Kim
TL;DR
Object-centric 3D generative models have limited spatial extent, and tiled generation of large multi-view scenes must address geometric continuity and global appearance consistency. The paper presents a training-free framework that coordinates tiles locally and globally and adapts scene decomposition to input geometry. Experiments report improved geometric and appearance fidelity over existing approaches while enabling fine-grained large-scale scene meshes.
Problem
Pretrained 3D generators are designed primarily for object-scale spatial extents, while scaling tiled generation to multi-view scenes raises challenges in boundary continuity and distant-region appearance consistency.
Source: §1 · p. 1, §3.2 · p. 3Method
A training-free framework adapts tile decomposition to scene geometry, incorporates neighboring context during denoising and decoding, and aligns appearance across distant regions.
Source: §1 · p. 2Results
Takeaways & Limitations
The framework provides an approach to scaling pretrained 3D generative priors beyond their original object-centric domain for large-scale scenes.
Source: §6 · p. 8
Abstract
from arXiv · showhide
Pretrained 3D generative models produce detailed geometry and appearance but are primarily designed for object-centric generation within a limited spatial extent. Recent approaches address this limitation by partitioning large scenes into smaller spatial regions and applying pretrained 3D generative priors to each region. However, scaling tiled generation to large multi-view scenes makes it challenging to maintain local geometric continuity and global appearance consistency. We present a training-free framework for large-scale textured mesh generation from multi-view images. Our key idea is to scale tiled generation to large scenes with increased spatial detail while coordinating generation both locally and globally. We introduce local context tiled generation to improve geometric continuity between neighboring regions and global appearance alignment to reduce appearance discrepancies across distant regions. An adaptive scene decomposition further determines the number of tiles according to the input scene geometry. Experiments demonstrate improved geometric and appearance fidelity over existing approaches while enabling fine-grained generation of large-scale scenes.
1 Introduction
Object-focused 3D generators have limited spatial extent, and extending tiled generation to multi-view scenes raises challenges for geometric continuity and scene-wide appearance consistency. The framework addresses these challenges with adaptive decomposition and local and global coordination, improving fidelity in large-scale scene generation.
1 Introduction
- Object-focused 3D generators operate within fixed spatial extents, so applying their resolution to larger scenes reduces representable geometric detail.Multi-view observations provide spatially distributed scene input when a single image cannot capture the broader extent.Source: §1 · p. 1
- Extend3D generates scene regions at a pretrained model’s native resolution and blends predictions in overlapping areas during denoising.This tiled approach extends object-level generation to scenes while retaining local detail.Source: §1 · p. 1
- Neighbor-only interactions can leave distant regions with inconsistent appearance, while tile-wise decoding can introduce geometric discontinuities at tile boundaries.Jointly decoding the entire scene becomes increasingly memory-intensive as the number of tiles grows.Source: §1 · p. 2
- The framework combines adaptive scene decomposition with local context during denoising and tile-wise decoding, plus global appearance alignment for large-scale multi-view mesh generation.Tile count follows reconstructed scene geometry; local context targets boundary continuity, while global correspondences target appearance variation across distant regions.Source: §1 · p. 2
- Experiments report quantitative and qualitative improvements in geometric and appearance fidelity over existing approaches while enabling fine-grained large-scale scene generation.The introduction reports these overall improvements without specifying metric values.Source: §1 · p. 2
2 Related Work
Related work spans object-centric 3D generative priors, scene-scale generation, and tiled diffusion. These approaches motivate extending pretrained priors while coordinating neighboring and distant regions.
2.1 3D Generative Models
- Most existing 3D generative models target object-centric generation within bounded spatial domains, leaving extension to spatially extended scenes challenging.Recent models improve geometry and scalability, but the cited discussion still identifies object-centric spatial support as a limitation.Source: §2.1 · p. 2
2.2 Large-Scale 3D Scene Generation
- Scene-scale methods either train dedicated generative models, often for particular domains, or reuse object-level priors, with Extend3D relying mainly on local overlap coordination.GenRecon [30] uses posed multi-view images with additional scene-level training; large outdoor generation from calibrated views without scene-specific training is described as underexplored.Source: §2.2 · p. 2
2.3 Tiled Diffusion for Large-Scale Generation
- Tiled diffusion methods extend pretrained models to larger domains, but limited local receptive regions can leave long-range structure or appearance inconsistent.This motivates exchanging local information while also coordinating globally for 3D scenes.Source: §2.3 · p. 3
3 Preliminaries
The preliminaries describe TRELLIS2’s staged generation pipeline and the tiled formulation used to extend object-centric priors to larger scenes. They identify variable decomposition needs, decoding cost, and limited cross-tile coordination as scaling challenges.
3.1 Pretrained 3D Generative Priors
- TRELLIS2 generates sparse structure, geometry latents, and material latents in sequence, with geometry and material stages conditioned on the generated structure.The pretrained prior’s finite resolution limits the spatial extent it can represent while preserving fine geometric detail.Source: §3.1 · p. 3
3.2 Tiled 3D Scene Generation
- The tiled formulation divides a target scene into overlapping local regions generated at the pretrained model’s native resolution, with neighboring predictions blended to reduce boundary discrepancies.The regions retain local native resolution while allowing the generated scene to cover a larger extent.Source: §3.2 · p. 3
- Scaling tiled generation requires scene-dependent decomposition, while whole-scene joint decoding grows memory-intensive and tile-wise decoding can remove cross-tile interactions.Overlap blending connects neighboring tiles but does not enforce appearance consistency between distant scene parts.Source: §3.2 · p. 3
4 Method
The framework uses scene geometry to choose a tile decomposition, then coordinates generation locally across neighboring tiles and globally across distant regions. Its components target fine-grained scene coverage, boundary continuity, and appearance consistency.
4.1 Overview
- The framework starts from multi-view images and a reconstructed point cloud, then assembles its tile representations into a textured scene mesh.Camera poses associate observations with scene regions, while the point cloud supports scene decomposition.Source: §4.1 · pp. 3–4
4.2 Adaptive Tile Estimation
- Adaptive Tile Estimation sets the tile count from reconstructed scene extent and sampling density, avoiding overly coarse tiles while not over-decomposing sparsely sampled geometry.The method uses point spacing as a proxy for supported spatial detail.Source: §4.2 · p. 4
4.3 Local Context Tiled Generation
- Local Context Tiled Generation adds neighboring boundary features during denoising and a latent-voxel halo during tile-wise decoding, providing cross-tile context without jointly decoding the scene.Only the decoded core region is retained, preserving tile-wise processing's memory efficiency.Source: §4.3 · pp. 4–5
4.4 Global Appearance Alignment
- Global Appearance Alignment finds corresponding voxels in other tiles using view-aggregated image features, then aligns principal texture components while preserving local residual details.Correspondences use cosine similarity; the principal subspace is derived by PCA.Source: §4.4 · p. 5
5 Experiments
Experiments on diverse large-scale scenes show improved geometric and appearance fidelity, with multi-view comparisons, finer tile decompositions, and component ablations examining the method’s performance and scalability.
5.1 Main Results
- Across UrbanScene3D, the method outperforms existing approaches overall, with lowest LPIPS, highest SSIM, and strong Chamfer Distance and F-score; single-image methods lag, and GenRecon trails on outdoor synthetic scenes.Evaluation covers four synthetic and three urban scenes, using rendered-view LPIPS and SSIM for appearance and Chamfer Distance and F-score for geometry.Source: §5.1 · p. 6
5.2 Qualitative Comparison
- The generated scenes better preserve overall structure, building-height variation, geometry, facade detail, and texture than the smoother or less distinctive baseline results.Figure 4 compares New York and San Francisco scenes against TRELLIS2 and Extend3D, with enlarged regions for inspecting local details.Source: §5.2 · p. 6, Figure 4 · p. 7
5.3 Scalability Analysis
- 74,990→410,868 SLAT voxels, 1.13M→5.67M mesh faces, and LPIPS 0.683→0.605 as n rises from 3 to 7 at fixed Town-scene extent.Finer tiles allocate the pretrained generator’s native resolution to smaller regions, revealing finer geometry and appearance detail.Source: §5.3 · pp. 6, 8, Figure 5 · p. 7, Table 2 · p. 8
5.4 Ablation Study
- With global alignment, LPIPS reaches 0.206 and SSIM 0.749; cross-tile attention raises SSIM from 0.736 to 0.737, then context-aware decoding raises it to 0.747.Qualitatively, attention and context-aware decoding reduce seams and boundary artifacts, while global alignment makes distant ground patterns and colors more similar.Source: Figure 6 · p. 7, Table 3 · p. 8, §5.4 · p. 8
6 Conclusion
The paper presents a training-free framework that extends pretrained 3D generative priors to large-scale textured meshes from multi-view images. Adaptive decomposition and local and global coordination support improved fidelity and fine-grained scene generation without additional training.
6 Conclusion
- The framework extends pretrained 3D generative priors to large-scale textured mesh generation from multi-view images without additional training.Source: §6 · p. 8
- Adaptive scene decomposition and local and global coordination preserve geometric continuity and improve appearance consistency across spatial regions.Source: §6 · p. 8
- Experiments on diverse large-scale scenes demonstrate improved geometric and appearance fidelity while enabling fine-grained generation.Source: §6 · p. 8