Source-linked AI summary
3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks
Zhaoliang Lun, Matheus Gadelha, Evangelos Kalogerakis, Subhransu Maji, Rui Wang
TL;DR
The paper tackles the time-consuming problem of reconstructing 3D shapes from line-drawing sketches, including sketches from one or multiple views. It uses an encoder-decoder network to predict multi-view depth and normals, fuses them into a point cloud, and converts that point cloud into a mesh. Across character and man-made object datasets, the authors report significantly more accurate reconstructions than competing voxel-based and alternative methods, including on human drawings.
Problem
Converting sketches into 3D models is time consuming, while voxel-based shape generation scales poorly with grid resolution.
Method
A ConvNet predicts multi-view depth and normal maps from one or more sketches, which are fused by energy minimization into a dense point cloud and polygon mesh.
Results
The method produces much more accurate reconstructions than competing methods across synthetic and human line drawings, and human participants most often judged its shapes the best match.
Takeaways & Limitations
View-based reconstruction is reported to be significantly more accurate than voxel-based reconstruction and can generalize to human-drawn sketches.
Abstract
from arXiv · showhide
We propose a method for reconstructing 3D shapes from 2D sketches in the form of line drawings. Our method takes as input a single sketch, or multiple sketches, and outputs a dense point cloud representing a 3D reconstruction of the input sketch(es). The point cloud is then converted into a polygon mesh. At the heart of our method lies a deep, encoder-decoder network. The encoder converts the sketch into a compact representation encoding shape information. The decoder converts this representation into depth and normal maps capturing the underlying surface from several output viewpoints. The multi-view maps are then consolidated into a 3D point cloud by solving an optimization problem that fuses depth and normals across all viewpoints. Based on our experiments, compared to other methods, such as volumetric networks, our architecture offers several advantages, including more faithful reconstruction, higher output surface resolution, better preservation of topology and shape structure.
I. INTRODUCTION
The paper addresses time-consuming 3D reconstruction from sketches and proposes a learned multi-view architecture that maps one or more line drawings to 3D shapes. It combines feed-forward depth and normal prediction with point-cloud optimization to avoid voxel-resolution limitations.
- Sketches convey figure-ground boundaries, surface curvature, and occlusions, but converting them into 3D models is time consuming and cumbersome.
- Prior ConvNet approaches commonly use voxel-based 3D representations, whose resolution scales poorly for high-resolution shape generation.
- The method targets sketches from one or more object views and uses a view-based representation to process depth and normals at higher resolution and speed than voxel-based representations on existing hardware.
- The proposed architecture predicts multi-view depth and surface normals from line drawings, then optimizes a dense point cloud consistent with those predictions.
- Experiments on character and man-made object datasets report significantly better reconstruction results than alternative approaches across several metrics.
II. RELATED WORK
Earlier sketch-based reconstruction methods rely on hand-designed geometric constraints or category-specific models, limiting their applicability. Learned retrieval methods recognize existing shapes but do not synthesize novel geometry.
- Geometric inference methods reason from properties such as convexity, parallelism, orthogonality, and discontinuity, but often target specific families of polyhedral objects.
- Methods for smooth or freeform surfaces exploit specialized line types and require accurate drawings, reconstructing only what is explicitly drawn.
- Sketch-retrieval ConvNets retrieve existing 3D shapes or parts but provide no means to synthesize novel shapes or parts from scratch.
- Category-specific parametric approaches recover only the parameters or rules of predefined models, failing when a drawing depicts a shape outside those model families.
III. METHOD
The method translates one or more line drawings into multi-view depth, normals, and foreground maps, then fuses them into a coherent 3D point cloud and polygon mesh. Joint optimization links the predicted views to a consistent reconstruction.
- A deep network translates line drawings into surface depth and normal images across several output viewpoints before fusion into a point cloud and polygon mesh.
- The network supports either a single sketch or multiple sketches from different object views, addressing missing shape information that one view may not convey.
- The maps are fused into a coherent 3D point cloud, which is converted into a surface mesh and can optionally be fine-tuned through geometric deformations.
A. Network Architecture
The network encodes input sketches into a compact shape representation and decodes it through separate viewpoint branches. Each branch predicts depth, normals, and foreground probabilities for its output view.
- Network Architecture: The network uses 12 uniformly sampled output viewpoints positioned at the vertices of a regular icosahedron.
- Network Architecture: Available input sketches are concatenated as channels in a 256 × 256 × C input, with a separately trained network for each input-view configuration.
- Network Architecture: The encoder applies convolutional layers with kernel size 4 and stride 2, producing a 2×2×512 representation that encodes shape information.
- Network Architecture: The decoder has 12 parameter-independent branches, each outputting a 256 × 256 × 5 image for its corresponding viewpoint.
- Network Architecture: Each branch predicts one depth channel, three unit-norm normal channels, and one foreground probability channel used to mask the predicted surface maps.
B. Training
Training combines synthetic line drawings with multi-view depth and normal supervision. The network uses pixelwise, mask, and adversarial losses to learn these predictions.
- Sketch generation: Synthetic sketches are generated from contours, suggestive contours, curvature-based feature curves, or filtered shaded renderings.Four variants are created per shape and input view, also serving as data augmentation.
- Sketch generation: Training sketches and ground-truth depth and normal maps are rendered under orthographic projection across the output viewpoints.Perspective projection is possible, but produces small differences because rendered objects have a relatively short depth range.
- Loss function: The depth loss uses per-pixel ℓ1 differences, while the normal loss uses cosine differences between predicted and ground-truth normal directions.Both losses are computed only on foreground pixels, with depths normalized to [−1, 1].
- Loss function: The mask loss penalizes disagreement between predicted and ground-truth foreground labeling.Foreground masking restricts depth and normal comparisons to valid shape pixels.
- Loss function: An adversarial network discriminates ground-truth five-channel depth-normal-foreground maps from generated maps.The generator and discriminator are trained alternately.
C. Point Cloud and Mesh Generation
The method converts multi-view predictions into a coherent point cloud by aligning views and optimizing pixel depths. It then generates a mesh and deforms it toward the input contours while preserving surface details.
- Point-cloud consolidation: Predicted multi-view depths and normals can be inconsistent across viewpoints or internally inconsistent with one another.These inconsistencies motivate optimization before mesh generation.
- Point-cloud consolidation: Each predicted depth map produces a point set with per-point normals, and ICP rigidly aligns point sets from all output views.ICP helps address inconsistencies between predicted depth maps.
- Point-cloud consolidation: Without optimization, concatenated aligned point sets remain noisy and contain misaligned regions that a smooth mesh cannot fit.Mesh generation amplifies the effect of residual depth-map inconsistencies.
- Depth optimization: The energy E(D) combines network-depth agreement, depth-derivative and normal orthogonality, and cross-view consistency.The optimized depths remain close to network predictions while enforcing geometric relationships across viewpoints.
- Depth optimization: The view-consistency term penalizes cross-view depth disagreement and deviations between surface tangents and normals, while excluding background projections and removing likely outliers.Points projected onto background in most views are removed from the point cloud.
- Mesh generation: The mesh is smoothly deformed so projected mesh points match extracted input contours while preserving surface Laplacians.This refines contour agreement without discarding underlying surface details.
IV. EVALUATION
The evaluation compares the method with baselines on synthetic and human sketches across characters and man-made objects. The method produces more accurate reconstructions in all reported cases, and human participants usually prefer its outputs.
- Datasets: The evaluation uses separate character, airplane, and chair shape collections, with synthetic and human line drawings in the test set.The test set contains 120 shapes: 90 with synthetic sketches and 30 with human drawings.
- Evaluation measures: Reconstructions are compared using Chamfer, Hausdorff, surface normal, depth map, and volumetric Jaccard distances.Lower values indicate better performance for the comparison table.
- Quantitative evaluation: Our method produces more accurate reconstructions than competing methods for both synthetic and human line drawings.The comparison excludes mesh fine-tuning to isolate reconstruction performance from optional post-processing.
- Baseline comparison: The U-net enhancement improves Tatarchenko et al.’s baseline but still performs worse than our method, especially for man-made objects.This indicates that U-net connections substantially strengthen that baseline without closing the performance gap.
- User study: Human participants chose our method as producing sketches’ best-matching shapes most of the time, including against nearest-neighbor retrieval.The supplementary material reports additional user-study details.
V. CONCLUSION
The paper presents a view-based ConvNet approach that predicts depth and normals from sketches and fuses them into a 3D point cloud. It reports higher accuracy than voxel-based reconstruction and generalization to human-drawn sketches, while noting that fine details remain limited.
- Method: The method predicts depth and surface normals from multiple viewpoints, then consolidates them into a 3D point cloud through energy minimization.The resulting reconstruction is evaluated on characters and man-made objects.
- Results: View-based reconstruction is reported as significantly more accurate than voxel-based reconstruction.This conclusion is stated across the evaluated shape categories.
- Results: The method generalizes to human-drawn sketches in addition to synthetic sketches.The conclusion presents this as an observed capability of the evaluated approach.
- Limitations and future use: The reconstructed shapes often lack fine details desired in production-quality 3D models, so they can serve as starting proxies for artists.The paper suggests interactive modeling techniques as a direction for improving this workflow.