3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks

• Sketch contours convey figure-ground boundaries, surface curvature, and occlusions, and artists commonly use them because they are relatively easy to draw; converting sketches to 3D remains cumbersome.Source: §I · p. 1
• Voxel representations scale poorly with grid resolution [9],,, while meshes resist consistent ConvNet parameterization and regular-grid convolutions, limiting their use largely to categories with correspondence-based surfaces,,,.Source: §I · p. 1
• A feed-forward network predicts multi-view depth maps and surface normals; energy minimization fuses them into a dense point cloud, which is converted to a polygon mesh and optionally optimized against the sketches.Source: §I · p. 1
• On character-model and man-made-object datasets, the approach achieves significantly better reconstruction than alternatives across Hausdorff distance, Chamfer distance, voxel intersection over union, depth and normal-map errors, and a user study.Source: §I · p. 1
• The view-based representation processes depth and normals at higher resolution and speed than voxel-based representations on existing hardware, while combining feed-forward and mesh-based representations for higher-quality shape prediction.Source: §I · p. 1
• Training uses automatically generated synthetic sketches without human-line-drawing supervision, yet the method generalizes to approximate, noisy, cross-view-inconsistent human drawings; learned descriptors also enable sketch-based 3D shape retrieval.Source: §I · p. 1

3D geometric inference from line drawings

• Geometric methods use line-derived constraints to infer shape, but struggle with organic surfaces, require accurate drawings, and reconstruct only what is drawn,,,,,,,,.Humans can interpret sparse, approximate drawings, suggesting purely geometric methods may not capture human shape understanding,,.Source: §II · pp. 1–2

Learning-based methods for shape synthesis

• Tatarchenko et al. predict color and depth from natural images and viewpoints, whereas this method predicts depth and normals from line drawings and jointly optimizes their integration into a mesh.It adapts a U-net with multi-view decoder branches and structured loss to resolve drawing ambiguities, and compares view-based and voxel-based methods using shape metrics and a perceptual study.Source: §II · p. 2

Sketch-based 3D shape retrieval

• Sketch-retrieval methods use hand-engineered descriptors or learned ConvNet representations to find existing 3D shapes, but do not synthesize novel shapes or parts,,,,,,,,,,.Category-specific parametric reconstruction methods,, recover only model parameters or rules; this method instead predicts shapes without predefined parametric models.Source: §II · p. 2

III. METHOD

• The encoder-multi-view-decoder predicts surface depth and normals from line drawings, fuses the maps into a coherent point cloud, and converts it into a polygon mesh.The mesh can then be fine-tuned through geometric deformations to match the input drawings more precisely.Source: §III · p. 2, Figure 1 · p. 3
• Fusing both depth and normal predictions produces better reconstructions than inferring surface normals from depth alone.Source: §III · p. 2
• Users can provide sketches from multiple views at once or progressively add views, updating the shape after each sketch until satisfied, then edit the mesh directly.For progressive input, a new sketch can be drawn over a semi-transparent rendering of the current shape, similar to ShadowDraw.Source: §III · p. 2

A. Network Architecture

• Outputs use 12 uniformly sampled viewpoints at the vertices of a regular icosahedron, with cameras facing the center and training shapes normalized to fit and consistently oriented.Each camera is oriented toward the upright axis.Source: A. Network Architecture · p. 3
• The network accepts 256 × 256 intensity sketches, concatenates available views as channels, and uses a separately trained network for each input-view configuration.For front-and-top inputs, sketches are concatenated in that order; the paper notes that specific-view training can improve reconstruction over arbitrary-view training and that canonical views are common in CAD.Source: A. Network Architecture · p. 3
• The encoder uses stride-2, kernel-4 convolutional layers with batch normalization and leaky ReLUs to produce a 2 × 2 × 512 shape representation that can also support sketch-based retrieval.The leaky ReLU slope is 0.2.Source: A. Network Architecture · p. 3
• Twelve independent decoder branches map the encoder representation to 256 × 256 × 5 outputs for corresponding viewpoints, containing depth, normal, and foreground-probability maps.Foreground probabilities above 50% define a binary mask that is applied to the depth and normal maps. Following U-net, decoder layers concatenate corresponding encoder features; normal maps are ℓ2-normalized, and the output layer uses tanh.Source: A. Network Architecture · p. 3

Generating training sketches

• Four synthetic sketch variants per shape and input view provide training augmentation, with sketches and target maps rendered under orthographic projection.The variants use silhouettes; silhouettes with suggestive contours; silhouettes with suggestive contours, ridges, valleys, and apparent ridges; or edge-preserving filtering on shaded renders.Source: B. Training · p. 4

Loss function

• The training objective combines depth, normal, foreground-mask, and large-scale structural losses, with weights λ1–λ4 set to 1.0, 1.0, 1.0, and 0.01 on a hold-out set.The structural term uses an adversarial network on concatenated depth, normal, and foreground maps; the two networks are trained alternately.Source: B. Training · p. 4

Per-pixel depth and normal loss

• Depth uses ℓ1 distance and normals use cosine-angle differences on ground-truth foreground pixels; ℓ2 depth loss produced less sharp maps.Training depths and clamped predictions both range from −1 to 1, making per-pixel depth and normal terms comparable in scale.Source: B. Training · p. 4

Mask and adversarial losses

• Foreground-mask disagreement is penalized with cross-entropy, while the adversarial loss discriminates ground-truth maps from generated maps.Its input is a five-channel image containing depth, three normal channels, and a foreground map; the discriminator shares the encoder architecture except for its probability-output layer.Source: B. Training · p. 4

C. Point Cloud and Mesh Generation

• Foreground pixels are mapped to 3D using predicted depth, orthographic camera parameters, and predicted normals, then ICP rigidly aligns point sets from all view pairs.Pixels with foreground probability above 50% are included; ICP helps address depth-map inconsistencies between views.Source: C. Point Cloud and Mesh Generation · pp. 4–5
• Optimizing depths across views addresses residual map inconsistencies and produces a smoother point cloud after 3–5 iterations.The quadratic energy keeps depths close to network predictions, encourages depth-derived tangents to be orthogonal to predicted normals, and enforces consistency with corresponding points in other views; it is solved as a sparse linear system.Source: C. Point Cloud and Mesh Generation · p. 5
• Screened Poisson reconstruction converts the fused points and normals to a mesh, which can be deformed to match external and internal sketch contours while preserving surface Laplacians.The contour-driven deformation combines constraints from all input views in a sparse linear system.Source: C. Point Cloud and Mesh Generation · pp. 5–6
• End-to-end shape generation takes about 10 seconds, including 1.5 seconds for network inference, 3 seconds for fusion, and 4 seconds for mesh reconstruction and fine-tuning.The network timing is reported on a TitanX GPU; fusion and mesh processing timings are reported on a dual Xeon E5-2699v3 CPU.Source: C. Point Cloud and Mesh Generation · p. 6

IV. EVALUATION

• Training data comprise 3D characters from The Models Resource and airplanes and chairs from ShapeNet, selected for their geometric and structural variation.The three shape collections were paired with synthetic sketches; Table I reports training-shape counts and the views used to generate those sketches.Source: §IV · p. 6, Table I · p. 6
• The test set contains 120 sufficiently distinct reference shapes, with 90 synthetic sketches and 30 human drawings, balanced across characters, airplanes, and chairs.Shape distinctness was checked using Chamfer distance to training shapes, nearest-neighbor distance between sketch-encoder representations, and visual inspection; two artists drew the human sketches.Source: §IV · p. 6
• Evaluation uses Chamfer, Hausdorff, surface-normal, depth-map, and volumetric Jaccard distances to compare reconstructions with reference shapes.Methods include ShapeMVD, volumetric and view-based decoder alternatives, R2N2, and nearest sketch-based retrieval; all use the same category-specific training data and test sets.Source: §IV · p. 7
• ShapeMVD produces more accurate reconstructions than competing methods across the reported cases, and human participants most often choose it as best matching the input sketches.Comparisons exclude mesh fine-tuning; the authors attribute improved sketch matching to better preservation of shape structure, topology, and coarse geometry. Nearest-neighbor retrieval can retain fine details that the reconstruction methods do not capture.Source: Table II · p. 8, §IV · p. 8
• The supplementary evaluation includes degraded ShapeMVD variants and comparisons using one versus two input sketches.Figure 4 presents results for synthetic and human sketches; its meshes use fine-tuning, unlike the main evaluation comparisons.Source: §IV · p. 8

V. CONCLUSION

• View-based reconstruction was significantly more accurate than voxel-based reconstruction across evaluations on characters and man-made objects.The method was also evaluated in variants and shown to generalize to human-drawn sketches.Source: §V · p. 8
• A ConvNet predicts depth and normals from multiple viewpoints, which are consolidated into a 3D point cloud through energy minimization.Source: §V · p. 8
• The authors propose exploring network-integrated fusion and an architecture that reconstructs shapes from arbitrary viewpoints.Source: §V · p. 8
• Reconstructed shapes often lack fine details, so the authors suggest using them as proxies for artists and integrating interactive modeling techniques.Source: §V · p. 8