PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches

Benoit Guillard, Edoardo Remelli, Pierre Yvernay, Pascal Fua

arXiv:2104.00482v2cs.CV

TL;DR

Sketch-based 3D reconstruction is difficult because sketches are sparse and ambiguous, and existing image-based methods can lack compact representations suited to editing. Sketch2Mesh encodes sketches as latent mesh representations and refines projected contours; the authors report style robustness, competitive performance, and refinement from partial strokes.

  • Problem

    Sketches provide sparse, ambiguous information, while image-plane SVR methods can lack compact shape representations for downstream editing.

    Source: §1 · pp. 1–2

  • Method

    Sketch2Mesh encodes a sketch into a latent mesh representation and refines the latent vector to align projected external contours with a camera-calibrated sketch.

    Source: §1 · p. 2, §3.1 · p. 3, §3.4 · p. 5

  • Results

    Sketch2Mesh/Chamfer works as well as or better than Sketch2Mesh/Render, and the approach handles untrained sketch styles, outperforms state-of-the-art methods, and supports partial-contour refinements.

    Source: §1 · p. 2, §5 · p. 7

  • Takeaways & Limitations

    With perspective camera parameters, the method allows interactive refinement by specifying partial 2D contours, including with stylus-drawn strokes.

    Source: §5 · p. 7

    The learned priors can be too constraining or insufficient for partial refinements, and the authors identify adding stronger shape constraints as future work.

    Source: §5 · pp. 7–8

Abstract

from arXiv · show

Reconstructing 3D shape from 2D sketches has long been an open problem because the sketches only provide very sparse and ambiguous information. In this paper, we use an encoder/decoder architecture for the sketch to mesh translation. When integrated into a user interface that provides camera parameters for the sketches, this enables us to leverage its latent parametrization to represent and refine a 3D mesh so that its projections match the external contours outlined in the sketch. We will show that this approach is easy to deploy, robust to style changes, and effective. Furthermore, it can be used for shape refinement given only single pen strokes. We compare our approach to state-of-the-art methods on sketches -- both hand-drawn and synthesized -- and demonstrate that we outperform them.

1. Introduction

Sketch2Mesh/Chamfer uses an encoder/decoder latent representation and external-contour matching to reconstruct and refine shapes from sketches. The paper reports that it works as well as or better than Sketch2Mesh/Render while avoiding an auxiliary image-translation network.

Motivation

  • The paper frames sketch-based 3D reconstruction as a way for designers, engineers, and artists to digitize legacy models and interact with CAD shapes by drawing in 2D.The authors contrast 2D drawing with sculpting shapes produced by 3D scanners.Source: §1 · p. 1
  • Existing sketch-regression approaches produce coarse, cumbersome-to-edit shapes or require multiple sketches or fixed viewpoints, while image-plane SVR methods lack compact shape representations for editing.Sparse sketch images and variation in drawing style also make generalization difficult for SVR networks relying on local image-plane feature pooling.Source: §1 · pp. 1–2

Approach

  • Sketch2Mesh trains an encoder/decoder to produce a mesh and compact latent representation, then refines the latent vector so the projected shape better matches an unseen camera-calibrated sketch.The refinement compensates for differences between the input sketch style and training sketches.Source: §1 · p. 2

Approach and results

  • Sketch2Mesh/Chamfer works as well as or better than Sketch2Mesh/Render, uses external contours for refinement, and avoids an auxiliary image-translation network.The authors state that external contours help generalization because graphics designers draw them similarly.Source: §1 · p. 2

2. Related Work

Related work spans mesh and implicit surface representations, sketch reconstruction, and single-view reconstruction. Sketch2Mesh combines a compact latent surface representation with contour-based alignment, addressing limitations in prior approaches for editing and sketch-style generalization.

Surface Representation

  • Mesh representations are versatile but challenging for deep learning, while template deformation fixes topology and surface patches can fail to stitch into watertight surfaces.Implicit fields allow topology changes, and differentiable mesh extraction from deep signed-distance functions provides an explicit mesh alternative used by the paper.Source: §2 · p. 2

3D Reconstruction from Sketches

  • Prior sketch methods include view-constrained or fixed-topology models, multi-view or fixed-viewpoint deep systems, and volumetric outputs with limited resolution or generalization.The cited methods vary in their input views, representation, and inference constraints.Source: §2 · pp. 2–3

Single View Reconstruction

  • Many image-plane feature-pooling SVR methods lack compact latent representations for downstream refinement or editing.Sketch2Mesh instead combines silhouette alignment with a more powerful latent representation.Source: §2 · p. 3

Refinement using differentiable rendering

  • Differentiable rendering can refine encoder/decoder reconstructions, but sketch-based use requires estimating foreground/background masks, which can generalize poorly across drawing styles.The text attributes this difficulty to the lack of diversified large-scale line-drawing datasets.Source: §2 · p. 3

3. Method

Sketch2Mesh encodes a sketch as a latent mesh representation and refines it by matching projected external contours. Its partial-sketch formulation preserves distant surface regions while allowing edits from incomplete contours.

3.1. Formalization

  • The encoder and decoder map a binary sketch to a mesh parameterized by a latent vector, which is subsequently refined because its projection only roughly matches the sketch initially.The paper uses the MeshSDF architecture for this encoding and decoding.Source: §3.1 · pp. 3–4

3.2. Using Differential Rendering

  • Sketch2Mesh/Render synthesizes a foreground/background image from the sketch and refines the mesh by differentiably rasterizing it against that target image.The approach depends on both a rasterizer and an image-translation network.Source: §3.2 · p. 4

3.3. Minimizing the 2D Chamfer Distance

  • Sketch2Mesh/Chamfer directly optimizes projected mesh contours against sketch contours, avoiding the auxiliary image-translation network used by the rendering alternative.This simpler technique is introduced as an approach that can be implemented from scratch.Source: §3.3 · p. 4, §3.1 · p. 4, §3.2 · p. 4

3.3.1 Finding External Contours in 2D and 3D

  • The Chamfer approach identifies visible mesh points projecting to exterior-contour pixels and uses their 2D projections as the mesh contour set.The corresponding 3D points are differentiable with respect to the latent vector through their barycentric coordinates on mesh facets.Source: §3.3.1 · p. 4

3.3.2 Objective function

  • The objective filters the sketch to its external contours and minimizes a bidirectional 2D Chamfer distance between those pixels and the projected mesh contours.The projected contour points and loss are differentiable with respect to the latent vector.Source: §3.3.2 · p. 4

3.4. Using a Partial Sketch

  • Sketch2Mesh reconstructions are reported as similar across Suggestive, SketchFD, and hand-drawn input styles and close to ground truth, unlike Pix2Vox, MeshRCNN, and DISN.Source: Figure 3 · p. 5
  • The partial-sketch formulation minimizes contour distance while regularizing distant projected regions to retain their original normals and boundaries.A distance mask is zero near the sketch and one farther away; the approach requires a differentiable renderer but not a trained image-translation network.Source: §3.4 · pp. 4–5

4. Results

The evaluation compares reconstructions across rendered and hand-drawn sketches, then selects Chamfer refinement and compares it with other methods. Results also demonstrate interactive editing from partial sketches.

4.1. Datasets

  • Because paired public line-drawing datasets are rare, evaluation uses ShapeNet-rendered cars and chairs, five novices’ 113 hand-drawn cars, and 177 ProSketch chairs with views unlike training.Rendered sketches use two styles and 16 sampled camera angles per object.Source: §4.1 · p. 5

4.2. Metrics

  • They report CD-l2 (lower is better) from 10,000-point mesh samples and normal consistency (higher is better) from pixel-wise normal-map dot products.CD-l2 compares reconstructed and ground-truth point clouds.Source: §4.2 · p. 6

4.3. Choosing the Best Method

  • Both refinement variants improve initial metrics, but Chamfer is more robust to sketch-style changes and avoids Render’s auxiliary image-translation network.Render overfits its training style, so the paper uses Chamfer for the remainder.Source: §4.3 · p. 6

4.4. Comparison against State-of-the-Art Methods

  • On cars, Sketch2Mesh clearly outperforms other methods; on chairs, MeshRCNN is competitive in CD-l2, but its meshes are hardly usable while ours need no preprocessing.The comparison covers models trained on Suggestive or SketchFD contours.Source: §4.4 · p. 6

4.5. Interactive 3D editing

  • The web interface accepts mouse or touch sketches and lets users submit successive partial sketches for optimizer-based matching.The approach can refine shapes from a single pen stroke.Source: §4.5 · p. 6

5. Conclusion

Sketch2Mesh uses an encoder/decoder to derive a latent surface representation from a sketch, then refines the mesh to match the sketch’s external contours. The approach handles style changes, supports interactive refinement from partial 2D contours, and aims to improve the learned priors and incorporate interior lines.

5. Conclusion

  • Sketch2Mesh derives a latent surface representation from a sketch and refines the mesh to match its external contours.The approach supports interactive refinement from partial 2D contours when camera parameters are available.Source: §5 · pp. 6–7
  • Sketch2Mesh/Chamfer handles thin chair components better than Sketch2Mesh/Render by leveraging sparse information.Figure 7 reports CD-l2=1.98 for Sketch2Mesh and CD-l2=1.91 for MeshRCNN, while describing the Sketch2Mesh reconstruction as more usable for further processing.Source: Figure 6 · p. 7, Figure 7 · p. 7
  • The method handles sketches drawn in styles it was not specifically trained for and outperforms state-of-the-art methods.Table 1 compares reconstruction metrics across cars and chairs and three datasets; its caption notes that Chamfer performs better than Render on styles it was not trained for.Source: §5 · p. 7, Table 1 · p. 7
  • The authors identify learned priors that can be too constraining or insufficient for partial refinements, and propose adding interior lines as future work.They also suggest turning some priors into constraints, such as requiring round wheels or equal-length chair feet.Source: §5 · pp. 7–8

7. Supplementary Material

The supplement describes how Sketch2Mesh extracts external contours and examines refinement behavior across training styles, thin components, and rasterization settings. It reports that Chamfer is more robust to style changes and sensitive to thin components, while higher faces-per-pixel settings improve Render refinement when predicted masks are inaccurate or noisy.

7.1. External Contours

  • External contours are estimated for both the reconstructed mesh and the input sketch because Chamfer refinement matches the two contour sets.The input-sketch method differs because line drawings may not be watertight.Source: §7.1 · p. 11, §7.1.2 · p. 11

7.1.1 External Contours of Reconstructed Shapes

  • For reconstructed shapes, a foreground/background mask is flood-filled from an image corner, dilated, and multiplied by the original mask to extract an external contour that ignores inner holes.The procedure supplies the reconstructed-shape contour used for refinement.Source: §7.1.1 · p. 11, Figure 9 · p. 11

7.1.2 External Contours of Input Sketches

  • Because input line drawings may not be watertight, their external contours are extracted with an image-space algorithm instead of the reconstructed-shape method.Rays from image borders preserve only the first encountered stroke; rotating the sketch at several angles and aggregating those pixels yields the contour.Source: §7.1.2 · p. 11, Figure 10 · p. 11
  • To reduce sensitivity to pen size, the input-contour algorithm keeps entry and exit pixels for thick strokes but only entry pixels for thin strokes.The thick-line decision uses the average image-wide distance between entry and exit pixels relative to a threshold.Source: §7.1.2 · p. 11

7.2.1 Training on SketchFD

  • On datasets whose styles were not used for training, Sketch2Mesh/Chamfer performs better than Sketch2Mesh/Render, indicating greater robustness to style change.This comparison is reported for networks trained on SketchFD and tested on all three datasets.Source: §7.2.1 · p. 11, Table 5 · p. 14

7.2.2 Gradients and Sensitivity to Thin Components

  • Chamfer refinement is more sensitive to thin components such as chair legs because its distance objective accounts for deformation distance and spatial extent.Removing a thin component affects only a few pixels in LF/B, whereas LCD accounts for distance and spatial extent.Source: §7.2.2 · p. 11, Figure 11 · p. 12

7.2.3 Differentiable Rasterization Hyperparameters

  • For Render refinement, more faces per pixel produce smoother gradients and more accurate reconstructions when predicted masks are inaccurate or noisy.The experiments set faces per pixel=25.Source: §7.2.3 · p. 11, Figure 12 · p. 13

Related papers

Loading 2104.00482v2…