Source-linked AI summary
SceneReGen: Generative Reconstruction of 3D Scenes from a Single Image
Zefan Tian, Yuteng Ye, Yiheng Zhang, Yuhang Yang, Xueqiang Lv, Shizhou Zhang, Le Liu, Di Xu
TL;DR
Single-image scene reconstruction must complete partially observed objects and place them coherently in a shared observation-aligned frame, but object generators typically use centered canonical frames. SceneReGen selectively encodes observed orientation in generated meshes and estimates translation and scale from scene evidence, achieving the strongest reported scene-level results on the 3D-FUTURE evaluation subset while supporting asset-centric outputs beyond indoor furniture.
Problem
Single-image reconstruction must recover complete partially observed objects in a shared scene frame, whereas object-level generative priors typically output centered, scale-normalized canonical meshes.
Method
SceneReGen uses selective pose factorization, with geometry-aware shape queries conditioning a pretrained 3D generator and scene-aware position queries estimating translation and scale.
Results
SceneReGen achieves the best scene-level CD, scene-level F-Score, and 3D bounding-box IoU, ties the best object-level CD, and ranks second in object-level F-Score on 3D-FUTURE.
Takeaways & Limitations
Qualitative autonomous-driving and embodied-AI outputs illustrate the potential of complete, orientation-aware object assets assembled in shared scene frames beyond indoor furniture.
Takeaways & Limitations
Texture synthesis is limited under heavy occlusion and extreme lighting, while performance degrades for low-resolution or blurry inputs and dense scenes can exhibit mesh penetration.
Abstract
from arXiv · showhide
Single-image 3D scene reconstruction must complete partially observed objects and place them coherently in a shared observation-aligned scene frame. Object-level generative priors offer strong completion ability, but their centered, scale-normalized outputs are typically expressed in an object frame, creating a fundamental representation gap between object generation and scene reconstruction. We introduce SceneReGen, a generative reconstruction framework that reinterprets scene reconstruction as the generation and assembly of complete object assets in a shared observation-aligned scene frame. SceneReGen addresses the generation-reconstruction gap through selective pose factorization: each object's observed orientation is encoded directly in the generated mesh, while translation and scale are estimated from instance-level and global scene evidence. Given a scene image and instance masks, a geometry encoder extracts dense cues; learnable shape queries condition a pretrained DiT-based 3D generator to produce complete meshes in their observed orientations, while position queries fuse object and scene features to assemble them in the shared frame. On the 3D-FUTURE evaluation subset, SceneReGen achieves the best scene-level CD, scene-level F-Score, and 3D bounding-box IoU among the evaluated methods, ties the best object-level CD, and ranks second in object-level F-Score. Qualitative outputs in autonomous-driving and embodied-AI scenes further illustrate the potential of asset-centric reconstruction beyond indoor furniture.
1 Introduction
SceneReGen frames single-image scene reconstruction as completing partially observed objects and assembling them coherently in a shared observation-aligned frame. It closes the representation gap by generating orientation-aware meshes while estimating translation and scale from scene evidence.
- Single-image reconstruction must complete partially observed objects and place their meshes coherently in a shared observation-aligned scene frame.
- Object-level generative priors complete geometry but typically produce centered, scale-normalized meshes in canonical object frames rather than scene coordinates.
- SceneReGen uses selective pose factorization: orientation is encoded in generated geometry, while translation and scale are estimated from scene evidence.
- Geometry-aware shape queries condition a pretrained 3D generator to produce complete meshes, while scene-aware position queries estimate translation and scale for shared-frame assembly.
- SceneReGen is evaluated on 3D-FUTURE, with additional qualitative outputs shown for autonomous-driving and embodied-AI scenes.
- The framework treats generated object assets as reconstruction primitives for recovering complete scenes.
3D Scene Reconstruction
Single-image scene reconstruction methods address incomplete geometry using feature-based, feed-forward, and generative approaches, but scene recovery remains challenged by missing content.
- COLMAP and PMVS rely on sparse feature matching and often produce incomplete geometry in textureless or occluded areas.
- DUSt3R and VGGT accelerate reconstruction through direct regression but still struggle with geometric sparsity and missing content.
3D Object Generation
3D object generation has advanced from image-derived diffusion priors toward large-scale text- or image-conditioned asset generation. These methods generally operate in canonical space, creating a mismatch with scene reconstruction in world coordinates.
- Generative models increasingly produce realistic 3D assets conditioned on text or images.
- Object-generation methods typically generate objects in canonical coordinate space without modeling global scene placement or orientation.
- Generative reconstruction combines generative priors with limited observations in object-level and scene-level paradigms.
- Object-level methods combine diffusion priors with reconstruction networks to produce high-fidelity single-view geometry and texture.
- Scene-level methods refine object details or jointly predict objects and spatial placements from reconstruction cues.
- SceneReGen decouples rotation from translation and scale, assigning rotation to generation and translation and scale to reconstruction.
3 Method
SceneReGen generates complete object meshes in their observed orientations, then estimates translation and scale to assemble them in a shared observation-aligned scene frame.
- Scene-Level Reconstruction: SceneReGen reconstructs scenes by generating complete object assets independently and assembling them through scene-level placement.The framework processes each masked instance separately before producing the final scene.
- Scene-Level Reconstruction: Rotation is encoded directly in each generated mesh, while translation and isotropic scale are estimated separately for scene placement.This selective pose factorization avoids treating rotation as an independent placement output.
- Rotation-Aware 3D Generation: Object patches are augmented with simulated occlusions before a geometry encoder extracts dense, geometry-aware features.Training places five square masks over 20% of foreground pixels to support completion under partial observations.
- Rotation-Aware 3D Generation: Learnable shape queries aggregate structural and orientation-related features into conditioning tokens for a DiT-based generator.The generator uses cross-attention to incorporate rotation information during denoising.
- Scene-Level Reconstruction: Position queries combine object-specific cues with global scene context to produce translation-and-scaling features for each object.These parameters scale and translate the rotation-aware meshes into the final global scene space.
- Training Loss: The training objective contains a flow-matching term for rotation-aware 3D generation and an L1 term for predicted translation and scale.The scene-level loss compares predicted position parameters with their ground-truth counterparts.
4 Experiments
Experiments on 3D-FUTURE compare SceneReGen with optimization and full-scene reconstruction baselines using quantitative and qualitative evaluations. Results show stronger scene-level reconstruction and alignment, competitive object-level geometry, improvements from geometry-aware encoding and occlusion augmentation, and limitations under challenging inputs.
- Experimental Setup: SceneReGen is evaluated on the 3D-FUTURE subset containing 14,761 training scenes and 5,479 test scenes.Each scene contains multiple objects with rendered images and ground-truth instance masks.
- Qualitative Comparison: SceneReGen retains more complete object surfaces than MIDI and SceneGen in the illustrated qualitative comparisons.Green boxes identify incomplete geometry; the comparison also examines orientation and relational placement across viewpoints.
- Quantitative Comparison: SceneReGen achieves the best CD-S, F-Score-S, and IoUB, ties SceneGen for best CD-O, and ranks second on F-Score-O.CD-S and F-Score-S measure scene-level geometry, CD-O and F-Score-O measure object-level geometry, and IoUB measures 3D bounding-box alignment.
- Ablation Study: Geometry-grounded encoders improve all four reconstruction metrics over DINOv2, with VGGT-Ω achieving the strongest encoder performance under the same augmentation setting.The evaluated geometry-grounded encoders are VGGT and VGGT-Ω.
- Ablation Study: Adding occlusion augmentation reduces CD-S from 0.010 to 0.009 and CD-O from 0.045 to 0.031, while increasing F-Score-S from 87.64 to 89.50 and F-Score-O from 60.52 to 68.95.The comparison retains VGGT-Ω and changes only the use of Mocc under the reported evaluation protocol.
- Limitations: Texture synthesis is less robust under heavy occlusions and extreme lighting, while low-resolution or blurry inputs degrade translation and scale estimation.These conditions restrict high-fidelity texture generation and scene placement accuracy.
5 Conclusion
SceneReGen reconstructs complete, rotation-aware object meshes and assembles them in a shared observation-aligned frame through selective pose factorization. On 3D-FUTURE, it achieves leading scene-level accuracy while retaining competitive object-level geometry and shows potential beyond indoor furniture.
- SceneReGen represents each object’s observed orientation in its centered, scale-normalized generated mesh, while estimating translation and scale from scene cues.
- Geometry-aware shape queries condition a pretrained 3D generator, while scene-aware position queries assemble completed assets in the shared observation-aligned frame.
- SceneReGen achieves the best scene-level CD, scene-level F-Score, and 3D bounding-box IoU among evaluated methods.
- SceneReGen ties the best object-level CD and ranks second in object-level F-Score.
- Cross-domain outputs illustrate the potential of asset-centric reconstruction for autonomous-driving scenario reconstruction and embodied-AI interaction.
6 Experimental Details
SceneReGen is trained with a filtered multi-source 3D asset dataset and evaluated under normalized geometric protocols. Inference uses masked object crops, fixed-resolution inputs, and shared sampling settings.
- Dataset Construction: The training dataset combines Objaverse, 3D-FUTURE, and MeshFleet to cover indoor furniture, vehicles, and everyday commodities.
- Dataset Construction: Multi-view rendering and watertightness inspection remove assets with degraded appearance, oversimplified geometry, clustered meshes, or flawed surfaces.
- Dataset Construction: The filtering pipeline yields around 25K object meshes, each rendered from 24 randomly sampled azimuth and elevation views with camera-aligned ground-truth meshes.
- Evaluation Protocol: Geometric quality is measured in normalized canonical 3D space bounded by x, y, z ∈[−1, 1], after rigid point-cloud registration to ground truth.
- Inference: Inference crops masked object regions, center-pads them to 512×512, and uses 50 denoising steps with CFG scale 3.0.
7 More Qualitative Results
Additional 3D-FUTURE visualizations render each reconstructed furniture instance from five viewing angles. These comparisons are intended to assess complete-mesh integrity and orientation consistency under arbitrary viewpoints.
- Each reconstructed furniture instance is rendered from five distinct viewing angles to display its complete rotation-aware generated mesh.
- The multi-view comparisons indicate stable geometric integrity and orientation consistency under arbitrary camera viewpoints.
8 Limitations and Future Work
SceneReGen’s limitations concern texture quality, dependence on clear inputs, and unconstrained object placement. Future work targets adaptive texturing, broader training data, and physical collision-aware scene assembly.
- Limitations: The off-the-shelf Hunyuan3D texture module lacks occlusion-aware optimization, causing blurriness and color distortion on heavily occluded objects.
- Limitations: Low-resolution or blurry input images significantly degrade reconstruction performance by removing fine-grained geometric and structural cues.
- Limitations: Independent translation-and-scale prediction lacks inter-object collision regularization, frequently causing mesh penetration and unnatural overlaps in dense indoor scenes.
- Future Work: Future work proposes occlusion-adaptive texture refinement and training on larger, more diverse multi-domain 3D scene datasets.
- Future Work: Future work will integrate explicit physical priors and collision-aware regularization to prevent mesh penetration and ensure natural spatial layouts.