Source-linked AI summary
MarrNet: 3D Shape Reconstruction via 2.5D Sketches
Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, William T Freeman, Joshua B Tenenbaum
TL;DR
Single-image 3D reconstruction is under-determined, and scarce real-image 3D annotations make synthetic-to-real transfer difficult. MarrNet addresses this with an end-to-end two-step pipeline that predicts 2.5D sketches before 3D shape, achieving strong results across synthetic and real data while exposing limitations on thin structures and inaccurate masks.
Problem
Single-image 3D reconstruction requires strong shape priors, while accurate 3D annotations for real images are scarce and synthetic training can suffer domain adaptation issues.
Method
MarrNet sequentially estimates depth, surface-normal, and silhouette sketches, regresses 3D shape from them, and uses differentiable consistency constraints for end-to-end training.
Results
MarrNet performs well on 3D shape reconstruction across synthetic ShapeNet and real PASCAL 3D+ data, with MarrNet achieving IoU 0.39 versus DRC 0.34 on PASCAL 3D+.
Takeaways & Limitations
2.5D sketches support transfer across domains and categories, while differentiable sketch-shape consistency enables annotation-free fine-tuning on real images.
Takeaways & Limitations
MarrNet performs poorly on complex thin structures and can fail when the estimated object mask is very inaccurate.
Abstract
from arXiv · showhide
3D object reconstruction from a single image is a highly under-determined problem, requiring strong prior knowledge of plausible 3D shapes. This introduces challenges for learning-based approaches, as 3D object annotations are scarce in real images. Previous work chose to train on synthetic data with ground truth 3D information, but suffered from domain adaptation when tested on real data. In this work, we propose MarrNet, an end-to-end trainable model that sequentially estimates 2.5D sketches and 3D object shape. Our disentangled, two-step formulation has three advantages. First, compared to full 3D shape, 2.5D sketches are much easier to be recovered from a 2D image; models that recover 2.5D sketches are also more likely to transfer from synthetic to real data. Second, for 3D reconstruction from 2.5D sketches, systems can learn purely from synthetic data. This is because we can easily render realistic 2.5D sketches without modeling object appearance variations in real images, including lighting, texture, etc. This further relieves the domain adaptation problem. Third, we derive differentiable projective functions from 3D shape to 2.5D sketches; the framework is therefore end-to-end trainable on real images, requiring no human annotations. Our model achieves state-of-the-art performance on 3D shape reconstruction.
1 Introduction
MarrNet addresses single-image 3D reconstruction by separating recovery of appearance-invariant 2.5D sketches from estimation of full 3D shape. Its differentiable, end-to-end framework is designed to transfer across domains and train on real images without annotations.
- MarrNet sequentially recovers depth and normal maps as 2.5D sketches before estimating the object’s full 3D shape.
- 2.5D sketches abstract away texture, lighting, material, and background variations while retaining information about the object’s 3D shape.
- The 3D-shape stage can be trained purely on synthetic data because realistic depth and surface-normal sketches are straightforward to render.
- Differentiable constraints enforce consistency between estimated sketches and 3D shape, enabling end-to-end training and annotation-free refinement on real images.
- The framework is evaluated on ShapeNet and PASCAL 3D+ and performs well qualitatively and quantitatively on 3D shape reconstruction.
2 Related Work
Related work covers 2.5D sketch estimation, single-image 3D reconstruction with learned shape priors, and consistency constraints linking 2D observations to 3D shape. MarrNet combines these directions while targeting full object-shape reconstruction.
- 2.5D Sketch Recovery: Prior work estimates depth, surface normals, and intrinsic images from RGB or RGB-D data using classical methods and deep networks.
- 2.5D Sketch Recovery: MarrNet uses 2.5D estimation as an intermediate component but targets reconstruction of the full 3D shape of an object.
- Single Image 3D Reconstruction: Single-image 3D reconstruction methods use shape repositories such as ShapeNet to learn priors and commonly regress voxelized 3D shapes directly.
- 2D-3D Consistency: Researchers have used 2D-3D consistency constraints for reconstruction and shape completion, including differentiable constraints between silhouettes and 3D shape.
3 Approach
MarrNet reconstructs 3D shape by estimating intrinsic 2.5D sketches from an RGB image, inferring a voxel shape prior, and enforcing differentiable reprojection consistency between the two representations.
- 2.5D Sketch Estimation: MarrNet predicts depth, surface normal, and silhouette images from an RGB image to distill intrinsic object properties for reconstruction.The estimator uses an encoder-decoder architecture and outputs the three sketches at 256×256 resolution.
- 3D Shape Estimation: The 3D shape estimator uses masked normal and depth images to infer a 128×128×128 voxel-based reconstruction while learning a shape prior from synthetic data.Because 2.5D sketches are straightforward to render, this component can be trained without realistic image rendering.
- Reprojection Consistency: The model introduces reprojection consistency losses that align predicted 3D shape with estimated depth, silhouette, and surface normal sketches.The depth criterion places occupied voxels at the estimated surface depth and empty voxels in front; lines without shape intersections satisfy the silhouette criterion.
- Reprojection Consistency: The reprojection formulation assumes orthographic projection and constrains voxel values to [0, 1].Surface-normal constraints apply only to target voxels inside the estimated silhouette.
- Training Paradigm: Training first fits the sketch and shape components separately on synthetic ShapeNet images, then fine-tunes the network on real images using reprojection consistency.A straightforward real-image fine-tuning implementation can produce unrealistic shapes by overfitting sketches without preserving the learned 3D prior, so the 3D decoder is fixed and only its encoder is fine-tuned.
4 Evaluation
MarrNet is evaluated on synthetic ShapeNet images and real PASCAL 3D+, IKEA, and multi-category images, with qualitative and quantitative comparisons against direct or prior reconstruction methods. Results show improved shape detail and human preference, while failures remain for thin structures and inaccurate masks.
- ShapeNet: 0.57 IoU vs. 0.52 for direct RGB prediction shows MarrNet outperforming the baseline on synthetic ShapeNet reconstruction.The model also produces smoother surfaces and finer details by estimating normal and depth maps before 3D shape.
- Ablation: Fixing the decoder during fine-tuning preserves the learned shape prior and yields more detailed shapes, whereas unfixed fine-tuning can produce impossible views.The unfixed model explains the 2.5D sketches but loses the learned prior.
- PASCAL 3D+: MarrNet achieves 0.39 IoU vs. DRC’s 0.34 on PASCAL 3D+, although the authors consider IoU suboptimal for rough shape annotations.The authors note that IoU favors mean shapes, requires scale search, and relies on only rough CAD-based annotations.
- PASCAL 3D+: Human subjects prefer MarrNet reconstructions 74% of the time over DRC and 42% of the time over PASCAL 3D+ ground truth.Each comparison was shown to 10 human subjects, using visual similarity to the input image as the criterion.
- Limitations: MarrNet struggles with complex thin structures and sometimes fails when the estimated mask is very inaccurate.These are reported failure cases on PASCAL 3D+.
- IKEA and other categories: On IKEA images, 61% of subjects prefer MarrNet reconstructions to 3D-VAE-GAN, and the model handles mild real-world occlusions.Additional experiments recover details on cars and airplanes and successfully reconstruct multiple object categories.
5 Conclusion
MarrNet explicitly models 2.5D sketches for single-image 3D shape reconstruction, improving performance and adaptation across domains or categories. Differentiable consistency losses enable end-to-end fine-tuning on real images without annotations, and experiments show strong performance and human preference over competitors.
- MarrNet explicitly models 2.5D sketches for single-image 3D shape reconstruction.
- Using 2.5D sketches enhanced performance and facilitated adaptation across domains or categories.
- Differentiable consistency losses between 3D shape and 2.5D sketches allow end-to-end fine-tuning on real images without annotations.
- Experiments found that MarrNet performs well and is preferred by human annotators over competitors.