MV-SAM3D: Adaptive Multi-View Fusion for Layout-Aware 3D Generation

Introduction

• SAM3D [3] reconstructs segmented objects with high-quality geometry and texture at their scene positions, enabling scene-level reasoning beyond object-centric reconstruction.This capability motivates extending layout-aware generation to multiple viewpoints and physically plausible multi-object scenes.Source: §1 · pp. 1–2
• MV-SAM3D addresses single-view limitations and implausible independent poses with training-free interventions, without architectural changes or retraining.Single-view methods hallucinate unobserved regions, while independent pose estimation can cause interpenetration, floating artifacts, and incorrect orientations.Source: §1 · pp. 2–3
• Multi-view generation fuses view-conditioned velocity predictions at each step through Multi-Diffusion in 3D latent space, using attention-entropy and geometric visibility weighting.Attention entropy estimates local observation confidence implicitly; geometric visibility explicitly assesses whether a 3D point is visible from a camera.Source: §1 · pp. 2–3
• Physics-aware pose optimization injects collision and contact constraints during flow matching and refines output meshes to satisfy non-penetration and surface contact constraints.The constraints steer generation toward plausible configurations and resolve residual violations in a subsequent refinement stage.Source: §1 · p. 3

Multi-View 3D Reconstruction

• DA3 [17] supplies metric-scale pointmaps and camera poses for geometric generation inputs, visibility computation, and multi-object pose optimization.Unlike pure reconstruction, the generative backbone also provides plausible completion for regions unobserved across all views.Source: §2 · p. 4

Compositional Scene Generation and Reconstruction

• MV-SAM3D generates object assets with layout parameters and composes physically plausible scenes through physics-aware pose optimization.Prior work includes object-level reconstruction with physics, layout-guided generation, and physics-based resolution of inter-part penetrations in single-image compositional generation [35].Source: §2 · p. 4

3 Method

• The pipeline recovers metric-scale pointmaps and camera poses from masked multi-view images, generates each object with adaptive fusion, then composes the scene with layout injection and refinement.It uses DA3 for multi-view 3D reconstruction.Source: §3 · p. 4

3.1 Preliminaries: SAM3D

• SAM3D’s two-stage foundation first generates a sparse voxel structure encoding coarse geometry and pose, then produces detailed geometry and texture in structured latents.The sparse-structure stage uses an MM-DiT, and the structured-latent stage uses a DiT conditioned on the image.Source: §3.1 · p. 5

3.2 Adaptive Multi-View Fusion

• Multi-view fusion adapts Multi-Diffusion to a shared 3D latent, combining viewpoint-specific flow-matching velocities instead of uniformly averaging views that may be occluded or unreliable.The framework estimates per-view confidence using cross-attention entropy and geometric visibility.Source: §3.2 · p. 6

Attention-Entropy Weighting (Implicit Observation Confidence)

• Low cross-attention entropy indicates a directly observed region, while high entropy indicates diffuse attention and lower observation confidence.In a six-view plush-toy example, entropy weighting in both generation stages recovered the tail structure and black-label texture; Stage 1 weighting alone left the label white.Source: Attention-Entropy Weighting (Implicit… · pp. 6–7

Visibility Weighting (Explicit Observation Confidence)

• Geometric visibility complements attention entropy by identifying visible latent points from voxel structure and camera poses, avoiding texture confusion from visually similar surfaces.For a medicine box with distinct front and back textures, adding visibility weighting separates the appearances where entropy weighting alone mixes them.Source: Visibility Weighting (Explicit… · pp. 8–9, Figure 5 · p. 9

3.3 Physics-Aware Pose Optimization

• Independent object poses can cause collisions and implausible arrangements, so MV-SAM3D applies physics-aware optimization during generation and after mesh generation.The two mechanisms are layout injection and global pose refinement.Source: §3.3 · p. 9

Layout Injection

• During the later flow-matching trajectory, layout injection decodes approximate voxel occupancy and uses collision and contact loss gradients to steer generation away from physical conflicts.The collision term penalizes volumetric overlap, while the contact term encourages nearby surfaces to remain in contact.Source: Layout Injection · pp. 9–10

Post-Generation Pose Refinement

• After generation, global pose refinement optimizes each object’s similarity transform while keeping mesh geometry fixed, balancing point-cloud alignment, collision avoidance, contact, and pose regularization.Alignment uses Chamfer Distance to the corresponding point cloud from metric-scale depth maps.Source: Post-Generation Pose Refinement · p. 10

4.1 Experimental Setup

• Evaluation uses GSO rendered from known camera poses and real-world MV-SAM3D-Scenes (2–8 objects, 15 viewpoints), compares five named baselines, and follows EscherNet’s protocol for single-object GSO evaluation.The baselines are SAM3D [3], TRELLIS [33], DreamGaussian, SyncDreamer, and EscherNet.Source: §4.1 · p. 10

4.2 Multi-View Single-Object Generation

• On GSO, multi-view methods outperform single-view baselines, and MV-SAM3D usually surpasses EscherNet despite fewer views; naive Multi-Diffusion trails, while MV-SAM3D preserves object layout.Single-view approaches hallucinate textures on unobserved sides, whereas MV-SAM3D produces reconstructions consistent with observed views.Source: Table 1 · p. 11, Table 2 · p. 11, §4.2 · p. 11

4.3 Multi-Object Scene Composition

• Multi-view fusion alone moderately improves per-object quality and spatial alignment over single-view SAM3D, while physics-aware optimization yields the largest gains across spatial-alignment metrics.The full pipeline also achieves faithful geometry and plausible layouts, with contacting surfaces touching and collisions resolved.Source: Table 3 · p. 12, §4.3 · p. 12

4.4 Ablation Studies

• Stage 1 entropy weighting improves structure; Stage 2 entropy and visibility weighting improve texture and appearance, while added views help consistently, most from one to two, then with diminishing returns.Visibility weighting particularly helps symmetric objects, where the implicit entropy signal can be unreliable.Source: Table 4 · p. 13, Table 5 · p. 13, §4.4 · p. 13
• Layout injection nearly eliminates generation-time collisions and improves alignment; mesh-pose refinement further improves alignment precision and coverage, with both components together performing best.The two components play complementary roles: injection provides a physically plausible initial configuration, and post-generation refinement optimizes poses on the output meshes.Source: Table 6 · p. 14, §4.4 · p. 14

• MV-SAM3D extends layout-aware 3D generation to multi-view input and physically plausible multi-object composition without additional training.Source: §5 · p. 14
• Attention-entropy and geometric visibility weighting fuse complementary observations according to confidence.The framework uses confidence-aware fusion to leverage complementary observations.Source: §5 · p. 14
• Physics-aware pose optimization resolves inter-object collisions and enforces surface contact, while experiments confirm improvements in reconstruction fidelity and layout plausibility.Source: §5 · p. 14