Source-linked AI summary
MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generation
Baicheng Li, Dong Wu, Jun Li, Shunkai Zhou, Zecui Zeng, Lusong Li, Hongbin Zha
TL;DR
Layout-aware 3D generation remains limited by single-view input and independently estimated object poses that can yield implausible scenes. MV-SAM3D uses confidence-aware multi-view fusion and physics-aware pose optimization without retraining, and experiments report consistent gains in reconstruction fidelity and layout plausibility.
Problem
Current layout-aware methods use single-view input and independently estimate object poses, limiting complementary-view use and causing implausible multi-object layouts.
Method
MV-SAM3D formulates fusion as 3D-latent Multi-Diffusion with attention-entropy and visibility weighting, then applies collision- and contact-constrained pose optimization during and after generation.
Results
Experiments confirm consistent improvements in reconstruction fidelity and layout plausibility while faithfully leveraging complementary observations and resolving inter-object collisions.
Takeaways & Limitations
The training-free framework extends layout-aware 3D generation to multi-view input and physically plausible multi-object composition.
Takeaways & Limitations
Attention-entropy weighting can fail on symmetric or repetitive objects when visually similar but geometrically incorrect regions receive low entropy.
Abstract
from arXiv · showhide
Recent unified 3D generation models have made remarkable progress in producing high-quality 3D assets from a single image. Notably, layout-aware approaches such as SAM3D can reconstruct multiple objects while preserving their spatial arrangement, opening the door to practical scene-level 3D generation. However, current methods are limited to single-view input and cannot leverage complementary multi-view observations, while independently estimated object poses often lead to physically implausible layouts such as interpenetration and floating artifacts. We present MV-SAM3D, a training-free framework that extends layout-aware 3D generation with multi-view consistency and physical plausibility. We formulate multi-view fusion as a Multi-Diffusion process in 3D latent space and propose two adaptive weighting strategies -- attention-entropy weighting and visibility weighting -- that enable confidence-aware fusion, ensuring each viewpoint contributes according to its local observation reliability. For multi-object composition, we introduce physics-aware optimization that injects collision and contact constraints both during and after generation, yielding physically plausible object arrangements. Experiments on standard benchmarks and real-world multi-object scenes demonstrate significant improvements in reconstruction fidelity and layout plausibility, all without any additional training. Code is available at https://github.com/devinli123/MV-SAM3D.
1 Introduction
MV-SAM3D extends layout-aware 3D generation from single images to multi-view input while addressing unreliable fusion and physically implausible multi-object layouts. It combines confidence-aware latent fusion with physics-aware pose optimization, without architectural changes or retraining.
- SAM3D reconstructs segmented objects with high-quality geometry and texture at their correct scene positions, enabling multi-object scene reconstruction.
- Single-view layout-aware generation cannot use complementary views, while naive fusion can let unreliable hallucinations overwhelm reliable observations.
- Independent pose estimation can produce interpenetration, floating artifacts, and incorrect orientations, motivating physical constraints for multi-object composition.
- MV-SAM3D formulates multi-view generation as 3D-latent Multi-Diffusion and fuses velocity predictions conditioned on different viewpoints at each step.
- Attention-entropy and visibility weighting provide confidence-aware fusion by estimating local observation reliability implicitly and geometrically.
- Collision and contact constraints are injected during generation and followed by mesh-level refinement to produce non-penetrating, contacting object arrangements.
2 Related Work
3D reconstruction has progressed from single-image feed-forward generation to learning-based multi-view methods that predict geometry and camera information efficiently. Classical SfM and visual SLAM remain multi-view alternatives but require many views and rely on feature matching and bundle adjustment.
- Feed-forward methods such as LRM, InstantMesh, Unique3D, TRELLIS, and SPAR3D regress 3D representations from a single image in one forward pass.These approaches achieve near-real-time performance.
- Classical SfM and visual SLAM estimate camera poses and sparse or dense geometry through feature matching and bundle adjustment.They require many views and struggle with textureless regions.
- DUSt3R and MASt3R formulate stereo reconstruction as regression, predicting dense pointmaps from image pairs in a single forward pass.
- VGGT and Fast3R extend feed-forward reconstruction to arbitrary numbers of views using transformers, while DA3 recovers metrically consistent geometry and camera poses from uncalibrated multi-view inputs.
3 Method
MV-SAM3D extends layout-aware 3D generation to multi-view inputs by adaptively fusing viewpoint-conditioned predictions, then addresses physically implausible multi-object arrangements through pose-aware composition and refinement.
- Multi-Object Composition: The composition stage combines layout injection during generation with post-generation pose refinement to address collisions, floating artifacts, and pose errors.Independently estimated object poses can still produce inter-object collisions or physically implausible arrangements after per-object generation.
- Multi-View Fusion: The framework generates each object by fusing flow-matching predictions from multiple viewpoints in a shared 3D latent space.Each viewpoint supplies an independent conditioning signal for the same latent variable, and the predictions are combined during generation.
- Multi-View Fusion: Naive averaging treats every viewpoint equally, allowing hallucinated content from occluded regions to overwhelm reliable observations.Adaptive fusion instead estimates which viewpoint is most reliable at each 3D point.
- Attention-Entropy Weighting: Attention entropy provides an implicit confidence signal: low entropy indicates concentrated attention and high observation confidence, while high entropy indicates diffuse attention and lower confidence.Visible regions show low entropy and occluded regions show high entropy in the plush-toy visualization.
- Attention-Entropy Weighting: Entropy weighting in both generation stages recovers both the plush toy’s tail structure and its black label texture, whereas stage-1-only weighting leaves the label texture incorrect.Simple averaging produces an incorrect tail and misses the label; applying entropy weighting only in Stage 1 recovers structure but not the label texture.
- Visibility Weighting: Attention entropy can fail on symmetric or repetitive structures by confidently matching a 3D point to a geometrically incorrect but visually similar image region.A medicine box with distinct front and back textures exhibits this failure when entropy weighting is used alone.
- Visibility Weighting: Geometric visibility complements entropy by identifying which latent points are visible from each viewpoint, separating the medicine box’s front and back appearances.The visibility estimate uses recovered object pose and relative camera poses to transform the coarse structure into each camera frame.
4 Experiments
Experiments on GSO and MV-SAM3D-Scenes show that multi-view fusion improves reconstruction fidelity, while physics-aware optimization resolves inter-object layout violations.
- Experimental Setup: MV-SAM3D is evaluated on GSO and the real-world MV-SAM3D-Scenes dataset, alongside representative single-view and multi-view baselines.MV-SAM3D-Scenes contains scenes with 2 to 8 objects captured from 15 viewpoints.
- Multi-View Single-Object Generation: Multi-view methods outperform single-view baselines, and MV-SAM3D consistently surpasses EscherNet across most metrics while using fewer input views.Naive Multi-Diffusion fusion in TRELLIS lags behind, highlighting the role of confidence-aware weighting.
- Multi-View Single-Object Generation: MV-SAM3D produces reconstructions faithful to observed object appearance, whereas single-view methods hallucinate textures on unobserved sides.Compared with EscherNet using 2 and 5 input views, MV-SAM3D shows more faithful texture details in certain regions.
- Multi-Object Scene Composition: Multi-view fusion improves per-object geometry, but pose optimization is required to resolve collisions and floating artifacts in multi-object layouts.The full pipeline achieves faithful geometry with physically plausible arrangements, including touching contact surfaces and resolved collisions.
- Multi-Object Scene Composition: Physics-aware optimization produces the largest gains in spatial alignment, while multi-view fusion alone yields moderate improvements over single-view SAM3D.The reported evaluation is summarized in Table 3 for MV-SAM3D-Scenes.
- Ablation Studies: Entropy weighting improves structural accuracy, extending it to Stage 2 recovers texture fidelity, and visibility weighting helps with symmetric structures.The ablation uses 5 input views; more views consistently improve quality, with the strongest gain occurring from one to two views and diminishing returns thereafter.
- Ablation Studies: Layout injection nearly eliminates collisions during generation, post-generation refinement further improves alignment, and their combination performs best.The two stages provide complementary benefits: an initially plausible configuration followed by direct pose optimization on output meshes.
5 Conclusion
MV-SAM3D is a training-free framework for multi-view-consistent 3D generation and physically plausible multi-object composition. Its confidence-aware fusion and pose optimization improve reconstruction fidelity and layout plausibility.
- MV-SAM3D extends layout-aware 3D generation to multi-view input and physically plausible multi-object composition without retraining.
- Attention-entropy and geometric visibility weighting enable confidence-aware fusion, while physics-aware pose optimization resolves collisions and enforces surface contact.
- Experiments confirm consistent improvements in reconstruction fidelity and layout plausibility.