Source-linked AI summary
Amodal3R: Amodal 3D Reconstruction from Occluded 2D Images
Tianhao Wu, Chuanxia Zheng, Frank Guan, Andrea Vedaldi, Tat-Jen Cham
TL;DR
Image-based 3D reconstructors often assume fully visible objects, leaving occluded-object reconstruction unresolved. Amodal3R extends a 3D generator with occlusion-aware conditioning and attention, and reports better performance than two-stage 2D-completion-then-3D methods, including on real-world images.
Problem
Image-based 3D reconstruction models fail to recover full 3D assets from partially occluded objects, while two-stage 2D completion can lack 3D reasoning and multi-view consistency.
Source: §1 · p. 1Method
Amodal3R extends a 3D generator for end-to-end reconstruction in 3D space, using mask-weighted cross-attention and an occlusion-aware layer to condition completion.
Source: §1 · p. 2, §3 · p. 3, §3.2 · p. 5Results
Takeaways & Limitations
Results on in-the-wild images indicate potential applications in 3D decomposition and scene understanding.
Source: §5 · p. 9Training uses only 20,627 synthetic 3D assets, predominantly furniture, and completions on complex or out-of-distribution objects may fail or produce unrealistic structures.
Source: §C.6 · p. 16
Abstract
from arXiv · showhide
Most image-based 3D object reconstructors assume that objects are fully visible, ignoring occlusions that commonly occur in real-world scenarios. In this paper, we introduce Amodal3R, a conditional 3D generative model designed to reconstruct 3D objects from partial observations. We start from a "foundation" 3D generative model and extend it to recover plausible 3D geometry and appearance from occluded objects. We introduce a mask-weighted multi-head cross-attention mechanism followed by an occlusion-aware attention layer that explicitly leverages occlusion priors to guide the reconstruction process. We demonstrate that, by training solely on synthetic data, Amodal3R learns to recover full 3D objects even in the presence of occlusions in real scenes. It substantially outperforms existing methods that independently perform 2D amodal completion followed by 3D reconstruction, thereby establishing a new benchmark for occlusion-aware 3D reconstruction.
1. Introduction
Existing image-based 3D reconstruction models struggle with partially occluded objects, while two-stage completion pipelines have limitations in 3D reasoning and multi-view consistency. Amodal3R instead performs occlusion-aware reconstruction in one stage and outperforms prior approaches across synthetic, scene, and real-image evaluations.
1. Introduction
- Image-based 3D reconstruction models fail to recover full 3D assets when objects are partially occluded.The paper frames reconstruction from one or more partially occluded views as the task of amodal 3D reconstruction.Source: §1 · pp. 1–2
- Earlier amodal pipelines combine 2D completion with 3D reconstruction, but 2D appearance priors lack explicit 3D reasoning and independent view completion can be inconsistent.These issues can confuse a multi-view 3D reconstructor.Source: §1 · p. 2
- Amodal3R extends the TRELLIS 3D generator with mask-weighted cross-attention and an occlusion-aware layer for single-stage reconstruction and completion in 3D latent space.The mechanisms guide attention to visible regions while supporting inference of occluded parts.Source: §1 · p. 2
- Amodal3R significantly outperforms existing two-stage pipelines on Replica scenes and real-world images, establishing a new benchmark for amodal 3D reconstruction.Evaluations also cover Google Scanned Objects and Toys4K augmented with occlusions.Source: §1 · p. 2
2. Background
Prior work addresses occlusion through 2D completion, 3D shape completion, or image-to-3D generation, but each leaves relevant limitations for full 3D reconstruction. Amodal3R’s overview combines object masks and image features to condition occlusion-aware 3D reconstruction.
2. Background
- 2D amodal completion can restore image textures, but limited 3D geometric understanding may yield implausible shapes, and completions can be inconsistent across views.The paper motivates a 3D-centric framework in response to these limitations.Source: §2 · p. 2
- 3D shape-completion methods recover geometry but not texture, and often require partial 3D geometry from an occluded image as input.The paper contrasts these methods with its end-to-end recovery of shape and appearance from an occluded image.Source: §2 · p. 2
- Latent-space image-to-3D generators can produce high-quality assets but assume fully visible inputs, limiting their use in scenes with frequent occlusions.Amodal3R’s overview shows object masks and DINOv2 features conditioning the 3D reconstructor, with mask-weighted and occlusion-aware attention in each transformer block.Source: §2 · p. 3, Figure 2 · p. 3
3. Method
Amodal3R extends TRELLIS to reconstruct complete 3D shape and appearance from partially occluded views, conditioning on visibility and occlusion masks. It combines mask-guided attention with synthetic occlusion data and supports visibility-prioritized multi-view conditioning.
3. Method
- Amodal3R conditions an end-to-end 3D generator on a partially visible object and visibility and occlusion masks to reconstruct complete shape and appearance directly in 3D.The method targets plausible completion and 3D consistency between visible and generated regions.Source: §3 · p. 3
3.1. Preliminaries: TRELLIS
- TRELLIS uses two diffusion stages to predict active voxel centers and then their latents, whose structured representation encodes coarse geometry and fine appearance details.The denoiser learns to remove Gaussian noise from latent codes.Source: §3.1 · p. 4
3.2. Mask-Conditional Generative Models
- Mask-weighted cross-attention favors visible image patches, while a following occlusion-aware layer uses occlusion masks to distinguish occluded areas from background.Patches with no visible pixels are skipped; the transformer structure applies to sparse-structure and SLAT diffusion models.Source: §3.2 · pp. 4–5, Figure 3 · p. 4
3.3. Simulating Occluded 3D Data
- Training uses rendered objects with randomly placed lines, ellipses, and rectangles as occlusions, while mesh-based masks with random-walk triangle selection provide multi-view-consistent contact-occlusion evaluation.The mesh masks are rendered with the RGB images’ camera parameters.Source: §3.3 · p. 5, Figure 4 · p. 5
3.4. Reconstruction with Multi-view Input
- For multi-view conditioning, Amodal3R accepts reference images without known camera poses and prioritizes views by visibility, using the most-visible images as primary inputs.Earlier denoising steps have stronger influence on final geometry because they establish coarse geometry before later detail refinement.Source: §3.4 · p. 5
4. Experiments
Amodal3R is evaluated on held-out object datasets, 3D scenes, and real images, with comparisons against pipelines that complete images before 3D reconstruction. It outperforms baselines across single- and multi-view metrics, and its ablations show complementary roles for its attention modules.
4.1. Experiment Settings
- Training uses 3D-FUTURE, ABO, and HSSD; evaluation includes held-out Toys4K and GSO, Replica, and in-the-wild inputs.The study measures rendered-image quality, 3D geometry, and consistency with ground-truth objects, and compares against 3D models using pix2gestalt completions.Source: §4.1 · pp. 5–6
4.2. Main Results
- Amodal3R significantly outperforms baselines across all metrics in single- and multi-view reconstruction, with results reported for GSO and Toys4K.Table 1 covers GSO, and Table 2 covers Toys4K.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 6
- Inconsistent multi-view 2D completions perform worse than a single completed view, whereas Amodal3R improves when given more occluded views.The authors attribute the baseline issue to inconsistent completions confusing reconstruction models.Source: §4.2 · pp. 6–7
- Across single- and multi-view tasks, Amodal3R produces high-quality assets; four occluded input views are used for the multi-view examples.The paper also reports that inconsistent 2D completions can mislead downstream reconstruction, as in its giraffe example.Source: §4.2 · p. 7, Figure 6 · p. 7
4.3. Ablation Study
- Naive token concatenation supports basic completion but yields inconsistent occluded-region textures and inaccurate geometry.The ablation reports a hole in the reconstructed shoe as one geometry example.Source: §4.3 · p. 8
- Mask-weighted attention improves appearance consistency but leaves geometry deficient, while the occlusion-aware layer improves geometry but alone yields unsatisfactory appearance.The ablation describes improved visible-area texture detail with mask-weighted attention and quantitative as well as qualitative geometry gains from the occlusion-aware layer.Source: §4.3 · p. 8
- The full model combining both modules achieves complete geometry and consistent textures.Source: §4.3 · p. 8
4.4. Real-World Generation / Completion
- On Replica scenes and in-the-wild images, Amodal3R generates reasonable, detailed assets, while pix2gestalt fails to infer complete shapes from the same inputs.Segment Anything supplies the visibility and occlusion masks for these evaluations.Source: §4.4 · p. 8
5. Conclusion
Amodal3R reconstructs complete 3D shape and appearance from partially visible images using mask-weighted cross-attention and an occlusion-aware layer. It performs better than sequential 2D completion and 3D generation methods, with in-the-wild results indicating potential downstream applications.
5. Conclusion
- Mask-weighted cross-attention and an occlusion-aware layer exploit visible and occluded information to reconstruct complete 3D shape and appearance.Source: §5 · p. 8
- Operating directly in 3D space yields better performance than sequential 2D completion followed by 3D generation.Source: §5 · p. 9
- In-the-wild results indicate potential applications in 3D decomposition and scene understanding.Source: §5 · p. 9
Supplementary Material
The network extends TRELLIS with mask-weighted multi-head cross-attention and a following occlusion-aware layer to condition 3D reconstruction on visible and occluded regions.
A.1. Network Design
- A mask-weighted multi-head cross-attention layer is integrated into each of TRELLIS’s 24 DiT blocks, followed by an occlusion-aware cross-attention layer.Both attention layers use 1024-dimensional features; the flattened occlusion mask is replicated to match that dimension.Source: §A.1 · p. 14
- DINOv2 image features and visibility and occlusion masks are patchified and flattened to aligned sequences of length 1,374.The image is resized to 518×518 for 14×14 patches, with CLS and register-token mask weights set to 1.Source: §A.1 · p. 14
- Visibility weights are applied simultaneously to each attention head to encourage focus on visible object regions.The H head outputs are concatenated to form the multi-head attention output.Source: §A.1 · p. 14
A.2. Training Details
Training fine-tunes two TRELLIS flow transformers and uses generated masks to simulate occlusions; inference uses 3D-consistent masks and takes under 10 seconds per object.
A.2. Training Details
- Training fine-tunes TRELLIS’s sparse structure flow transformer and structured latent flow transformer while retaining the other modules’ pretrained weights.The training masks combine 1–3 drawn lines, circles, and ellipses with 3–7 expanded rectangular regions.Source: §A.2 · p. 14
- Random mask generation connects drawn regions with expanded rectangles to simulate less-fragmented real-world occlusions.Figure A.1 marks visible areas in white, occluded areas in gray, and background in black.Source: §A.2 · p. 14, Figure A.1 · p. 14
A.3. Inference Details
- At inference, each object receives a 3D-consistent mask ratio randomly selected between 0.4 and 0.6.This setting produces a variety of reasonable mask areas.Source: §A.3 · p. 14
- Amodal3R generates and renders each object in under 10 seconds, with inference time comparable to baselines despite added cross-attention layers.Source: §A.3 · p. 14
B.1. Evaluation Protocol
Supplementary results report higher-quality Amodal3R reconstructions than the 2D-completion-plus-3D-generation baselines across multiple categories and in multi-view examples.
B.1. Evaluation Protocol
- Evaluation uses 1,030 GSO objects and 1,500 Toys4K objects, measuring rendering and semantic quality with FID, KID, and CLIP, and geometry with P-FID, COV, and MMD.Four views are rendered per object; geometry metrics use point clouds sampled to 4,096 points.Source: §B.1 · p. 15
C.1. Baselines using occluded input
- With occluded images used directly as input, baseline models often fail to recover complete and intact 3D assets.The examples show incomplete or broken structures.Source: §C.1 · p. 15
C.2. More single-view to 3D examples
- Amodal3R yields higher-quality 3D assets across multiple categories than 2D amodal completion plus 3D generation baselines.The comparison is illustrated with additional single-view-to-3D examples.Source: §C.2 · p. 15
C.3. More multi-view to 3D examples
- Independent pix2gestalt completion across views produces obvious inconsistency, while using Zero123++ to create four consistent views improves consistency and 3D generation quality.Figure C.3 visualizes the pix2gestalt and pix2gestalt + MV settings.Source: Figure C.3 · p. 15, §C.3 · p. 16
- In additional multi-view examples, Amodal3R again generates higher-quality 3D assets than the baselines.Source: §C.3 · p. 16
C.4. More diverse 3D reconstruction results
Multiple stochastic samplings from an occluded input produce diverse, reasonable 3D reconstruction results.
C.4. More diverse 3D reconstruction results
- Multiple stochastic samplings produce diverse, reasonable 3D results from the same occluded input.Source: §C.4 · p. 16
C.5. More in-the-wild results
Additional in-the-wild examples compare Amodal3R with a 2D-completion pipeline and visualize geometry details using colored normal maps.
C.5. More in-the-wild results
- Amodal3R produces more plausible geometry and richer texture than pix2gestalt + TRELLIS in the additional in-the-wild comparisons.The 2D-completion pipeline can make improper completions, such as turning a chair armrest into a backrest.Source: §C.5 · p. 16
- In the chair example, pix2gestalt + TRELLIS completes the armrest as a backrest, while Amodal3R yields a more plausible result.Source: §C.5 · p. 16
- Figures C.8 and C.9 show in-the-wild results with colored normal maps for geometry detail; Segment Anything supplies the segmentation masks.Source: §C.5 · p. 16
C.6. Discussion, Limitation and Future Work
The discussion identifies limits in training data, real-world adaptation, and controllability, and points to larger datasets and added user conditions as future directions.
C.6. Discussion, Limitation and Future Work
- Training uses only 20,627 synthetic 3D assets, predominantly furniture, so complex or out-of-distribution objects may receive unrealistic completions or fail.Source: §C.6 · p. 16
- Because training is synthetic-only, Amodal3R cannot use environmental cues and must reconstruct from visible object portions; real-world 3D modal data is proposed to improve real-scene application.Source: §C.6 · p. 16
- Completion is currently determined by the model, without user control over its style.The authors identify additional conditions such as text and user editing as future directions.Source: §C.6 · p. 16