Source-linked AI summary
DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction
Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, Ulrich Neumann
TL;DR
Single-view 3D reconstruction methods often struggle with limited resolution and fine-grained structures. DISN predicts signed distance fields using global image features plus projected local features, achieving high-quality detail-rich models and superior benchmark performance, while relying on aligned training models for viewpoint estimation.
Problem
Single-view reconstruction methods based on voxels, point clouds, or global information have limited resolution or often miss fine details such as holes and thin structures.
Method
DISN predicts the signed distance value of each 3D point from an input image, combining global image features with local features extracted around the point’s projection using an estimated camera pose.
Results
DISN achieves superior qualitative and quantitative single-view reconstruction performance, capturing fine-grained details and outperforming other models in EMD and IoU.
Takeaways & Limitations
Implicit SDF prediction provides a flexible way to generate topology-variant, high-quality 3D meshes with detail-rich structures from single-view images.
Takeaways & Limitations
Viewpoint estimation is trained using aligned ShapeNet models, making alignment an assumption of the camera-pose setup.
Abstract
from arXiv · showhide
Reconstructing 3D shapes from single-view images has been a long-standing research problem. In this paper, we present DISN, a Deep Implicit Surface Network which can generate a high-quality detail-rich 3D mesh from an 2D image by predicting the underlying signed distance fields. In addition to utilizing global image features, DISN predicts the projected location for each 3D point on the 2D image, and extracts local features from the image feature maps. Combining global and local features significantly improves the accuracy of the signed distance field prediction, especially for the detail-rich areas. To the best of our knowledge, DISN is the first method that constantly captures details such as holes and thin structures present in 3D shapes from single-view images. DISN achieves the state-of-the-art single-view reconstruction performance on a variety of shape categories reconstructed from both synthetic and real images. Code is available at https://github.com/xharlie/DISN The supplementary can be found at https://xharlie.github.io/images/neurips_2019_supp.pdf
1 Introduction
DISN addresses the limited resolution, fixed-topology, and detail-recovery problems of prior single-view 3D reconstruction methods by predicting signed distance fields with global and projected local image features. Experiments on synthetic and real images report improved reconstruction quality and fine-grained detail recovery.
- Motivation: Prior voxel- and point-cloud-based methods are often resolution-limited, while explicit surface methods commonly assume fixed topology.Chamfer Distance and Earth-mover Distance also provide only approximated shape-similarity metrics for point- and mesh-based training.
- Approach: DISN predicts signed distance fields from single-view images, providing an implicit surface representation without a fixed topology assumption.Sampling different 3D points produces an implicit field that can be converted into a surface using methods such as Marching Cubes.
- Motivation: Global image features alone often miss holes and thin structures because these details occupy a small portion of 3D space and contribute little to the loss.Such omissions can nevertheless produce visually unsatisfactory reconstructions.
- Approach: DISN projects each query point onto the input image and combines corresponding local features with global features to predict its SDF value.The local feature module is designed to learn relations between projected pixels and 3D points, improving fine-grained detail reconstruction.
- Results: DISN captures details such as bench-back patterns and rifle-handle holes that previous state-of-the-art methods fail to produce.The authors report this as the first deep learning model able to capture such high-quality details from single-view images.
- Results: Evaluations across shape categories using synthetic and online product images report superior qualitative and quantitative performance over state-of-the-art methods.The model is also extended to multi-view reconstruction and applications including shape interpolation.
2 Related Work
Related work spans explicit representations such as voxels, points, primitives, geometry images, skeletons, and parametric surfaces, alongside newer implicit representations. These approaches address different reconstruction settings but remain associated with resolution, topology, or fine-detail limitations.
- Explicit representations: Prior learning-based methods represent shapes with voxels, octrees, points, primitives, geometry images, skeletons, or parametric surface elements.Some approaches require additional shape-primitives data, while others generate surfaces through specialized representations.
- Implicit representations: Explicit representations often suffer from limited resolution and fixed mesh topology, motivating implicit alternatives.DISN adopts signed distance functions as an implicit surface representation.
- Implicit representations: Recent implicit methods have used occupancy prediction or related fields, but prior approaches can fail to recover fine-grained details in single-view reconstruction.Occupancy-based methods predict whether volumetric cells are inside or outside a 3D model and may refine active cells iteratively.
3 Method
DISN represents a 3D shape as a continuous signed distance function and predicts it from a single image using camera-aware global and local image features. The resulting field is sampled and converted into a mesh, while local features and residual prediction target fine-grained structures.
- SDF Representation: DISN maps each 3D point to a signed distance value, whose zero iso-surface implicitly represents the reconstructed shape.The sign indicates whether a point is inside or outside, and the absolute value gives its distance to the surface.
- SDF Prediction: The network predicts a continuous SDF from an input image, allowing the field to be sampled at arbitrary resolution.Unlike fixed-resolution volumetric grids, DISN predicts an SDF value for any queried 3D point.
- Camera-Aware Local Features: DISN estimates camera parameters, projects each 3D query point onto the image plane, and collects corresponding multi-scale local features.Features are retrieved at the projected location on each encoder feature map and concatenated for the query point.
- SDF Prediction: Global and local image features are combined with point features to predict the SDF, with separate decoders summing global and local predictions.The local branch predicts a residual SDF that complements the overall shape recovered from global features.
- Camera Pose Estimation: Camera pose is learned through point-cloud alignment, using mean squared error between transformed world-space points and ground-truth camera-space points.The predicted pose transforms the point cloud before the loss is computed.
- Loss and Mesh Extraction: DISN regresses continuous SDF values with a weighted loss that assigns higher weight to points whose signed distance is below threshold δ.This weighting concentrates learning near and inside the zero iso-surface.
- Loss and Mesh Extraction: A dense 3D grid of predicted SDF values is converted into a mesh by extracting the zero iso-surface with Marching Cubes.The same implicit representation supports mesh generation after field prediction.
4 Experiments
DISN is evaluated on ShapeNet and rendered image data through quantitative, qualitative, camera-pose, and ablation studies. It outperforms competing models on key reconstruction metrics, while local features improve detail recovery and estimated camera poses remain effective.
- Experimental setup: DISN is evaluated on ShapeNet Core across 13 categories using quantitative and qualitative comparisons with state-of-the-art reconstruction methods.The experiments also include rendered product-image data, camera-pose evaluation, and ablation studies.
- Single-view reconstruction: Implicit surface reconstruction produces topology-variant meshes, and pointwise SDF prediction yields smoother surfaces than fixed-volume 3D CNN generation.The paper attributes the smoother surfaces to SDF continuity with respect to point locations.
- Camera pose estimation: A 2D reprojection error of 4.38 pixels is achieved on 224 by 224 images for the new dataset, or 2.67 pixels normalized to 137 by 137 resolution.Using a 6D rotation representation, the method outperforms [28] by 2 pixels in 2D reprojection error.
- Ablation studies: Local feature extraction improves reconstruction quality by a large margin and helps recover detailed patterns that global features alone miss.The ablation compares the Global setting, which predicts SDF values solely from global image features, with DISN.
- Ablation studies: The Two-stream architecture is slightly superior to One-stream, indicating that DISN remains robust across the tested network structures.One-stream concatenates global, local, and point features before a single SDF decoder.
5 Conclusion
DISN reconstructs detailed single-view 3D shapes by predicting SDF values from image features and projected local features. The method achieves strong reconstruction performance but is limited to objects with clear backgrounds because it is trained on rendered images.
- Conclusion: DISN predicts SDF values for 3D points using an input image, estimated camera pose, and local features extracted from projected image locations.These local features help the model capture fine-grained details and generate high-quality 3D models.
- Conclusion: Qualitative and quantitative experiments validate DISN’s superior performance over state-of-the-art methods and the flexibility of its model.The paper also extends DISN to multi-view reconstruction and shape interpolation.
- Conclusion: DISN handles objects with clear backgrounds because it is trained with rendered images.The authors identify extending SDF generation with texture prediction using a differentiable renderer as future work.