Source-linked AI summary
WorldSculpt: Generating Compositional Worlds from Grounded Videos
Muyao Niu, Jixuan He, Ruihan Yu, Lian Fu, Yonghao Yu, Zheng-Hui Huang, Yifan Zhan, Fengbo Lan, Yongtao Ge, Yinqiang Zheng, Kaipeng Zhang, Zhixiang Wang
TL;DR
The paper addresses how to generate individually usable object meshes for heavily occluded scenes containing hundreds of objects. WorldSculpt adapts an object-level 3D generative prior with multi-view conditioning in canonical frames, then places generated meshes in a shared world frame. It consistently outperforms evaluated baselines across single-object, multi-object, and large-scale scene evaluations, with the largest gains under dense clutter and severe occlusion.
Problem
Generating compositional 3D scenes from multiple images is difficult because dense occlusion reveals only partial object geometry, while existing methods often fuse scenes or target simpler settings.
Method
WorldSculpt aligns each object's multi-view observations in a canonical frame, conditions an object-level generative prior, and transforms each generated mesh into the shared world frame.
Results
WorldSculpt consistently outperforms evaluated baselines across controlled single-object, multi-object, and UE-MeshyScene evaluations, with largest gains under dense clutter and severe occlusion.
Takeaways & Limitations
A generative prior trained only on individual canonical-space objects can generalize to large, densely occluded scenes without scene-level training.
Takeaways & Limitations
The method depends on camera poses, instance masks, and coarse 3D boxes, and currently generates geometry without object appearance.
Abstract
from arXiv · showhide
We study the problem of generating a compositional 3D representation of a cluttered scene containing hundreds of objects. The goal is to represent the scene as a collection of individual object meshes placed in a shared world frame, as required by downstream applications such as gaming, AR/VR, simulation, and robotics. This task is challenging in densely cluttered scenes, where objects heavily occlude one another and each view reveals only a fraction of their geometry. Geometry-based approaches typically reconstruct the scene as a single representation and leave incomplete geometry in occluded regions, while existing compositional methods with generative priors are largely limited to relatively simple scenes. We show that complex scenes with hundreds of objects can instead be generated compositionally by adapting a strong single-object 3D generative prior to multi-view observations. We instantiate this paradigm with Pixal3D, extending it with a multi-view conditioning pathway that grounds object generation in multiple posed observations. Although the model is finetuned entirely on single objects in canonical space, it generalizes to large scenes with severe occlusion without any scene-level training, demonstrating the feasibility and scalability of this paradigm. We further introduce UE-MeshyScene, a photorealistic benchmark of densely cluttered scenes with hundreds of objects, per-object annotations, and ground-truth meshes. Across single-object, controlled multi-object, and UE-MeshyScene evaluations, our method consistently outperforms prior approaches, with larger gains as scene complexity and occlusion increase. Finally, we demonstrate broader applicability by converting generated 3DGS worlds, such as Marble and HY-World 2.0, into compositional mesh scenes.
1 Introduction
WorldSculpt targets compositional 3D reconstruction of densely cluttered scenes by generating individually usable object meshes from grounded multi-view observations. It combines an object-level generative prior with multi-view conditioning and introduces a benchmark for evaluating this setting.
- Motivation: Compositional 3D reconstruction must align evidence across views, complete occluded geometry, and preserve each object as an independently usable asset.Dense packing and mutual occlusion leave only partial observations of many objects.
- Motivation: Existing geometry-centric methods often produce a single fused representation and incomplete geometry in occluded regions, while generative priors provide plausible completion beyond visible content.The two families address complementary aspects but do not fully satisfy all requirements together.
- Method: WorldSculpt maps multi-view observations into anchor-aligned canonical frames, conditions an object-level generative prior, and places each generated mesh into a shared world frame.The formulation avoids cross-object fusion and joint shape optimization.
- Method: A prior finetuned only on individual canonical-space objects generalizes to scenes with hundreds of densely occluded objects without scene-level training.Multi-view conditioning and a targeted augmentation curriculum support this generalization.
- Evaluation: UE-MeshyScene provides photorealistic cluttered scenes with hundreds of objects, per-object annotations, and ground-truth meshes for compositional 3D evaluation.The benchmark includes rendered indoor and outdoor environments with substantial mutual occlusion.
2 Related Work
Related work spans feed-forward and diffusion-based image-to-3D generation, amodal reconstruction, object-level datasets, scene-level compositional generation, and methods for recovering camera motion and structure from multiple images.
- Image and multi-view to 3D generation: Feed-forward image-to-3D methods infer meshes or Gaussian representations from one or a few images, while another family transfers 2D diffusion priors to 3D generation.The passage lists transformer-based systems including LRM, MeshLRM, InstantMesh, GRM, LGM, TripoSR, and SF3D.
- Amodal 3D reconstruction: Amodal 3D reconstruction seeks complete object geometry from partial observations affected by occlusion.Recent methods adapt native-3D generative priors or address sparse, unposed views directly.
- Object-level 3D datasets: Object-level 3D datasets provide ground-truth meshes for evaluating image-to-3D methods, but many evaluations use rendered images with clean backgrounds.The passage distinguishes synthetic, artist-created, scanned, catalog-based, and real-world datasets.
- Scene-level and compositional generation: Scene-level compositional generation jointly recovers object geometry and scene layout from images or real-world captures, commonly using synthetic scene datasets.The cited approaches include InstPIFu, PartCrafter, MIDI, SAM3D, SceneGen, SceneMaker, and RecGen.
- Structure and motion from multiple images: Multi-image object reconstruction relies on camera motion, coarse 3D structure, and object localization estimated through structure-from-motion, visual SLAM, or newer casual-video systems.These estimates provide the upstream geometric and pose information required by reconstruction pipelines.
3 Method
WorldSculpt canonicalizes each observed object into an anchor-aligned frame, conditions a Pixal3D geometry prior on aligned multi-view features, and places the generated meshes back into a shared world frame. Its design preserves individually addressable objects while accounting for crop geometry, occlusion, and variable view counts.
- Inference and scene composition: Each object is generated as an individual canonical mesh and transformed into the world frame, preserving a collection of independently addressable meshes rather than a fused representation.The canonical-to-world mapping is an isotropic similarity transformation, preserving the generated object’s proportions.
- Anchor-aligned object canonicalization: WorldSculpt constructs an anchor-aligned canonical frame for each object from posed images, instance masks, and coarse world-space localization boxes.The anchor fixes the canonical orientation, while the virtual cube is adjusted when needed to cover object masks across selected views.
- Anchor-aligned object canonicalization: Crop-aware projections map each canonical voxel through world and camera coordinates into resized object crops, accounting for pose, scale, and image location.The procedure masks backgrounds and resizes each projected crop to the generative model’s input resolution.
- Multi-view conditioned object generation: The aggregated voxel condition and anchor-view global tokens adapt Pixal3D’s first two geometry stages through zero-initialized injections, LoRA, and a frozen pretrained prior.The voxel grid contributes evidence from all selected views, while anchor tokens retain Pixal3D’s single-image conditioning interface.
- Multi-view conditioned object generation: WorldSculpt lifts DINOv3 features from each canonical observation into shared 3D voxel coordinates before aggregating views with an order-independent module.The aggregator accepts a variable number of views and combines a cross-view mean with a softmax-weighted residual.
4 Experiments
Experiments evaluate WorldSculpt from canonical single-object recovery through compositional scenes and large-scale clutter, showing stronger multi-view benefits as occlusion and scene complexity increase. The study also tests learned fusion, conversion of generated 3DGS worlds, and practical limitations.
- Evaluation settings: Evaluations progress from Toys4k object generation to Toys4k-Scene, HouseCat6D, and large-scale UE-MeshyScene scenes with hundreds of objects.The benchmarks increase in scene complexity and occlusion.
- Evaluation settings: UE-MeshyScene contains six Unreal Engine environments with 93–701 objects per scene and exact meshes, camera poses, and 3D bounding boxes.The dataset contains 2,299 objects overall and supports world-frame evaluation.
- Canonical-space generation: With one view, WorldSculpt is competitive with Pixal3D; performance improves consistently as additional views become available.This indicates that multi-view adaptation preserves single-view capability while exploiting additional observations.
- Canonical-space generation: With 16 views, performance changes only slightly up to 50% per-view occlusion, while single-view baselines degrade substantially.Additional views reduce occlusion effects across metrics, with distance errors remaining stable and F-Scores decreasing only mildly at high view counts.
- UE-MeshyScene generation: On UE-MeshyScene, WorldSculpt outperforms ShapeR across all reported metrics, with gains especially consistent for typical objects and heavily occluded small objects.The median improvement indicates that the gap is not driven only by a few difficult instances.
- Multi-view fusion ablation: Learned IBR fusion becomes more effective with more views and occlusion, reducing CD-ℓ2 from 12.91 to 10.03 and increasing F@.05 from 0.742 to 0.762 at 2 views and 75% occlusion.On UE-MeshyScene, IBR reduces CD-ℓ2 by 12% and raises F-Score from 0.944 to 0.951 over mean fusion.
- 3DGS-world application: WorldSculpt converts Marble-generated 3DGS worlds into individual meshes in a shared world frame using the standard inference pipeline.The conversion uses posed RGB frames, per-object mask tracks, and coarse 3D localization boxes.
- Limitations: The method depends on camera poses, instance masks, and coarse 3D localization boxes, while currently generating geometry only for static scenes.Large input errors can affect geometry, appearance is not modeled, and moving or deforming objects are unsupported.
5 Conclusion
WorldSculpt generates compositional 3D scenes from grounded multi-view observations by adapting an object-level generative prior, and generalizes from canonical single-object training to densely cluttered scenes without scene-level retraining. Across evaluations, it consistently outperforms baselines, with the largest gains under dense clutter and severe occlusion.
- WorldSculpt integrates anchor-aligned multi-view observations with an object-level generative prior to generate compositional 3D scenes.
- The model is trained entirely on individual objects in canonical space yet generalizes to large, densely cluttered scenes without scene-level retraining.
- WorldSculpt consistently outperforms evaluated baselines across single-object, multi-object, and UE-MeshyScene experiments.
- The largest gains occur under dense clutter and severe occlusion, while learned multi-view aggregation becomes more beneficial as observations grow incomplete and heterogeneous.
- Adapting object-level generative priors to multi-view compositional generation provides a practical path toward editable 3D representations of complex, cluttered environments.