Source-linked AI summary
GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction
Katharina Schmid, Nicolas von Lützow, Jozef Hladký, Angela Dai, Matthias Nießner
TL;DR
High-fidelity 3D scene reconstruction from multi-view RGB images is difficult because observations are insufficient in many regions, while existing methods can produce incomplete, noisy, or oversmoothed geometry. GenRecon extends an object-level generative prior to scene-scale reconstruction through overlapping chunk generation and projection-based multi-view conditioning. It produces faithful, editable PBR meshes and reports stronger reconstruction quality than evaluated baselines across real-world and synthetic scenes.
Problem
Multi-view scene reconstruction is underconstrained, and existing learned methods remain limited in fidelity and completeness in challenging, occluded, and unobserved regions.
Method
GenRecon jointly generates overlapping scene chunks using a pretrained 3D generative prior and a spatially grounded, permutation-invariant projection-based pathway for posed multi-view conditioning.
Results
GenRecon achieves the strongest overall reconstruction quality across both 2D and 3D metrics on unseen ScanNet++ and held-out 3D-FRONT scenes.
Takeaways & Limitations
The method produces complete, high-fidelity, faithful, editable PBR mesh reconstructions of indoor scenes suitable for downstream graphics, simulation, and embodied AI pipelines.
Takeaways & Limitations
Reconstructions are less reliable for non-Lambertian materials, chunking is designed for indoor scenes up to roughly 5m vertical extent, and weak evidence can cause occasional hallucinated content.
Abstract
from arXiv · showhide
We introduce a new approach to high-fidelity 3D scene reconstruction from multi-view RGB images that tightly couples reconstruction with a strong generative 3D prior. We cast scene reconstruction as conditional 3D generation over a set of spatially-localized, overlapping chunks that together tile the scene, scaling generation to large scene extents. Crucially, we inherit the fidelity and completeness of state-of-the-art generative shape models -- we use Trellis.2 as an example -- which we generalize to the scene level. To this end, we propose a projection-based conditioning mechanism that lifts posed multi-view image features into a coherent 3D representation aligned with the generative model, independent of view ordering and spatially anchored to the scene, yielding high-fidelity, multi-view consistent generated geometry. This enables lifting the strong object-level prior of Trellis.2 to multi-view, scene-scale generation, producing faithful, editable PBR mesh reconstructions of indoor environments. As a result, we obtain high-fidelity results that outperform cutting-edge reconstruction methods by 16%.
1 Introduction
High-fidelity scene reconstruction from multi-view RGB images is underconstrained, especially where views provide weak correspondences. GenRecon addresses this by combining overlapping chunk generation with a spatially grounded, permutation-invariant conditioning pathway and a strong generative prior.
- Motivation: High-fidelity reconstruction must recover complete surfaces and material properties suitable for relighting and editing, beyond metric geometry alone.These requirements are especially relevant to content creation and immersive applications.
- Motivation: Underconstrained regions caused by limited viewpoints, occlusions, repetitive patterns, and view-dependent effects often yield incomplete, noisy, or oversmoothed reconstructions.Reliable correspondences are difficult when captures lack diverse and accurate observations throughout the scene.
- Motivation: Learned feed-forward methods improve consistency but remain limited in fidelity, while generative shape models provide realistic, coherent, and complete object-level priors.The paper targets the gap between these capabilities for scene reconstruction.
- Approach: GenRecon formulates scene reconstruction as conditional 3D generation over spatially localized, overlapping chunks, enabling large-scale scene reconstruction with Trellis.2’s generative prior.Chunks are jointly synthesized under guidance from the input views.
- Approach: A projection-based 3D conditioning pathway lifts posed multi-view image features into a spatially grounded, permutation-invariant representation for controlled scene generation.The pathway enforces geometric consistency across views while preserving spatial alignment.
2 Related Work
Prior reconstruction methods rely on feature matching, per-scene optimization, or post hoc fusion, leaving ambiguous and occluded regions incomplete. GenRecon instead directly synthesizes complete, structured mesh geometry with a pretrained 3D generative prior over overlapping scene chunks.
- Reconstruction without learned priors: Classic multi-view stereo relies on feature matching, epipolar verification, and patch-based fusion, but struggles in weakly textured, occluded, or sparsely observed regions.These limitations arise because the approach relies primarily on photoconsistency.
- Reconstruction without learned priors: Neural implicit surface methods optimize continuous signed-distance or density fields, yet remain unreliable where triangulation is under-constrained.Monocular depth cues can augment, but do not eliminate, the ambiguity of per-scene optimization.
- Reconstruction with learned priors: Geometric foundation models and feed-forward volumetric fusion recover observed geometry, but their outputs require post hoc surface fusion and lack generative priors for completing occlusions.These methods regress depth, pointmaps, camera parameters, TSDFs, or occupancy fields from image information.
- GenRecon: GenRecon directly synthesizes complete, structured mesh geometry through one coherent conditional generation process over overlapping spatial chunks.This bypasses per-object decomposition, per-view fusion, and per-scene optimization.
3 Method
GenRecon reconstructs complete scene-level PBR meshes by jointly generating overlapping spatial chunks with a Trellis.2-based generative prior and spatially grounded multi-view conditioning. The method lifts posed image features into 3D, aggregates them globally, and merges jointly generated chunk predictions into a coherent scene.
- Scene representation: The method reconstructs a scene-level PBR mesh from sparse, unordered posed RGB images using overlapping spatial chunks.Each chunk covers a fixed-size canonical volume and is placed in world coordinates by a translation.
- Generative prior: Trellis.2 supplies the object-level generative prior, which is adapted for conditional generation of multi-view scene chunks.The generative model produces chunk latents representing geometry and appearance within each canonical volume.
- Multi-view conditioning: A projection-based pathway encodes each image with DINOv3, projects features into chunk-aligned 3D grids, and aggregates them into a spatially grounded condition.Projection ties each conditioning feature to an explicit 3D location, while aggregation is permutation-invariant across views.
- Conditioning injection: The aggregated 3D condition is injected residually into the generative denoiser at every block through zero-initialized layers.This preserves pretrained behavior at initialization while providing positional conditioning for view consistency and pose control.
- Scene reconstruction at test time: At test time, COLMAP calibrates the images and sparse point cloud, after which the scene extent is partitioned into fixed-size chunks and a global conditioning grid is cropped per chunk.The global grid is computed once over the scene volume rather than independently for every chunk.
- Joint chunk generation: All chunks are generated jointly in one flow-matching trajectory, with overlapping latent predictions averaged to enforce consistency across chunk boundaries.The merged global latent is decoded by Trellis.2 into the final PBR mesh; a boundary-sensitive variant improves seam coherence visually.
4 Experiments
Experiments evaluate the method on unseen real-world ScanNet++ and synthetic 3D-FRONT scenes, using 2D and 3D metrics alongside baseline and ablation comparisons. Results show strong reconstruction quality, effective pose-controlled conditioning, and PBR outputs, with limitations for certain materials, scene extents, and weakly observed regions.
- Evaluation Setup: The evaluation uses 25 scenes with 8 input views from both ScanNet++ and 3D-FRONT, with sparse points and camera poses provided.ScanNet++ represents out-of-domain real-world data, while 3D-FRONT provides held-out synthetic scenes.
- Evaluation Setup: 2D metrics cover geometric, perceptual, semantic, and completeness measures, while 3D metrics assess alignment, coverage, and normal consistency.The reported metrics include MAE, RMSE, AbsRel, SqRel, angular normal error, LPIPS, CLIP, Chamfer distance, F-score (10 cm), and normal consistency.
- Reconstruction Results: On unseen real-world ScanNet++ scenes, the method achieves the strongest overall reconstruction quality across both 2D and 3D metrics and the highest completeness.It also shows better perceptual and semantic alignment with ground-truth laser scans for depth and normals.
- Reconstruction Results: On held-out 3D-FRONT scenes, the method again achieves the strongest overall performance across both 2D and 3D metrics while avoiding oversmoothing and high-frequency artifacts.The evaluation uses scenes held out from training despite fine-tuning the occupancy stage on a small 3D-FRONT subset.
- Ablations: The projection-based 3D conditioning pathway enables pose-controlled chunk generation, whereas vanilla Trellis.2 and scene-data fine-tuning without 3D conditioning fail to reliably align geometry to the target pose.The ablation compares vanilla Trellis.2, scene-data fine-tuning, and the full method on 25 unseen SAGE-10k chunks.
- Ablations: Pose-correct chunk generation works from a single input image, and performance improves with additional views because more of each chunk is directly observed.This evaluates sensitivity to the number of conditioning views per chunk.
- PBR Texture and Relighting: The method produces scene-level meshes with albedo, metallic, and roughness properties that can be imported into rendering engines and relit without per-scene optimization.The recovered textures are visually plausible but do not match dedicated SVBRDF-estimation methods in absolute material accuracy.
- Limitations: Limitations include unreliable reconstruction of glass and mirrors, indoor chunking designed for vertical extents up to roughly 5m, and occasional hallucination where views provide weak evidence.The authors report that gains in completeness and surface fidelity outweigh the occasional hallucinations in their evaluations.
5 Conclusion
The method reconstructs scene-level PBR meshes by lifting an object-level generative prior to scene scale. It jointly generates overlapping spatial chunks from posed multi-view images through spatially grounded projection-based conditioning.
- The method reconstructs scene-level PBR meshes from posed multi-view images by lifting an object-level 3D generative prior to scene scale.
- Scene reconstruction is formulated as joint generation of overlapping spatial chunks conditioned through a spatially grounded projection-based 3D pathway.The pathway injects multi-view image features into the generative model while the chunks jointly cover the scene.
- The approach combines generative-prior fidelity and completeness with scene-scale reconstruction, producing faithful, editable PBR meshes for graphics, simulation, and embodied AI pipelines.
A Experimental Setup
The system adapts Trellis.2 with LoRA-based conditioning and trains on synthetic indoor scene chunks. Inference estimates overlapping chunks from SfM geometry and synchronizes their boundaries.
- Trellis.2 is used at resolution 512 with LoRA adapters on all attention layers, while conditioning enters each of 30 DiT blocks through residual projections.The LoRA adapters use rank 8 and scaling 16.
- Training samples 1 to 16 conditioning views per chunk uniformly and uses classifier-free guidance with condition dropping probability 0.1.
- Inference sets chunk size to 1.11 × scene height and places neighboring chunks with a minimum overlap margin of 0.25.Boundary-sensitive voxel synchronization excludes the outermost b = 1 rows from cross-chunk aggregation.
- The training data comprise 15,000 chunk-level examples from 5,000 SAGE-10k rooms, plus 1,200 3D-FRONT houses for occupancy training.
B Metrics Details
Evaluation measures reconstructed scenes in held-out 2D views and extracted 3D meshes, combining geometric, perceptual, semantic, completeness, alignment, coverage, and normal-consistency metrics.
- Reconstruction quality is evaluated both on unseen 2D test views and directly on extracted 3D meshes.
- 2D Metrics: 2D evaluation reports depth, normal, perceptual, semantic, and completeness metrics over valid pixels and held-out frames.Metrics include MAE, RMSE, AbsRel, SqRel, angular normal error, LPIPS, and CLIP score.
- 3D Metrics: 3D evaluation measures Chamfer distance, F-score@10cm, and normal consistency using sampled mesh points and nearest-neighbor correspondences.Normal consistency is zeroed for nearest-neighbor pairs farther than 0.2 m.
- Evaluation Masking: 3D metrics are computed within per-scene observation envelopes, and predicted geometry outside those regions is clipped before sampling.The ScanNet++ envelope is derived from scanner fusion and dilated by 15 cm; 3D-FRONT uses a ground-truth bounding box inflated by 20%.
C Further Results
Further results show high-fidelity reconstruction on large ScanNet++ scenes, pose-consistent generation from limited views, and additional relighting results. More input views improve reconstruction quality.
- Large Scene Generation: The method yields high-fidelity reconstructions on large 3D indoor scenes from ScanNet++.Figure 6 presents a top-down view alongside multiple close-ups.
- Ablation: The projection-based 3D conditioning enables pose-correct chunk generation from a single input image, with quality improving as additional views increase scene coverage.
- Ablation: Vanilla Trellis.2 fails on scene chunks, while scene fine-tuning alone improves local plausibility but cannot recover correct pose without 3D conditioning.
- Relighting: Additional results vary lighting configurations for scenes reconstructed from ScanNet++.
D Baseline Implementation Details
The evaluation adapts released baseline configurations to a common posed multi-view protocol, using method-specific settings for sparse-view reconstruction and mesh fusion.
- All baselines are evaluated in the same posed multi-view setting, with released configurations changed only as needed for the protocol.The stated adaptations fix applicability issues under sparse-view evaluation and are not intended to weaken the baselines.
- 2D Gaussian Splatting uses the official implementation with the Tanks-and-Temples large-scene preset and retains the top 50 TSDF-fused clusters.The cluster-selection change aims to avoid removing correct geometry from unconnected regions.
- Depth-Anything-3 predicts RGB-D jointly in one forward pass, then uses 0.01 m voxels, 0.04 m truncation, and 4.5 m maximum depth for TSDF fusion.The predicted depths are also reused as input to FineRecon.
- Murre uses its pretrained checkpoint and default setup, lowering the fusion consistency threshold from 3 to 2 for sparse view overlap.Sparse-depth conditioning uses the full available SfM point cloud regardless of the number of RGB reconstruction frames.
- MonoSDF uses the official implementation with Omnidata depth and normal supervision, following the ScanNet configuration for 200k iterations.
- FineRecon uses official ScanNet-trained weights and scene bounds from the ground-truth mesh axis-aligned bounding box enlarged by 20%.