Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches

1. Introduction

• Single-sketch reconstruction is challenging because free-hand drawings can be poorly drawn, geometrically distorted, and lack textures or shading.The introduction describes existing options as requiring precise multi-view line drawings or retrieving existing models, limiting ease of use and customizability together.Source: §1 · p. 1
• Specifying the viewpoint can reduce the reconstruction search space and may resolve ambiguities when a sketch's pose is unclear or multiple shapes fit.The authors connect viewpoint specification to applying visual rules about what an object should look like from a particular viewpoint.Source: §1 · p. 1
• The view-aware method separates image features into latent shape and view codes, predicts the sketch viewpoint, and conditions mesh generation on view information.A view auto-encoder constrains the view space, random-view training encourages decoder dependence on the given view, and domain adaptation bridges synthetic and real sketches.Source: §1 · p. 2
• The method generates higher-quality shapes and better conveys user intentions than alternative baselines across various datasets.The authors also report a ShapeNet-Sketch dataset containing 1,300 sketch-shape pairs.Source: §1 · p. 2
• The paper presents single free-hand sketch mesh generation as a fast, easy-to-use 3D content-creation option for novice users.The authors identify this as the first investigation of that specific problem.Source: §1 · p. 2

2.1. Single-View 3D Reconstruction

• Single-view reconstruction methods infer 3D forms from images using data-driven models and differentiable rendering, and this method extends [22] with separate latent shape and view spaces.The paper states that its training process does not require 3D information.Source: §2.1 · p. 2

2.2. Sketch-Based 3D Modeling

• Prior sketch-modeling approaches include evocative retrieval or primitives and constructive stroke-to-model methods, while few deep-learning works target free-hand sketches.Wang et al. [35] generate point clouds and rotate them to a canonical view; this paper instead uses viewpoint as an input to interpret sketches and bridges the domain gap in feature space.Source: §2.2 · pp. 2–3
• The proposed architecture decomposes image features into shape and view spaces, trains a view encoder-decoder, and uses random views and a shape discriminator.These components aim to isolate view information, make the decoder use it, and preserve shape quality.Source: Figure 2 · p. 3

3.1. View-Aware Generation

• The baseline encodes the sketch into a feature vector and deforms a template mesh, trained end-to-end using rendered silhouette comparisons without ground-truth 3D shapes.The method extends this single-view encoder-decoder framework to condition generation on viewpoint.Source: §3.1 · p. 3, §3 · p. 3
• The view-aware encoder maps image features into latent view and shape spaces, and the decoder uses both codes to generate a deformed mesh.A view auto-encoder maps viewpoints to and from the latent view space, which is intended to hold view-related information.Source: §3.1 · p. 3
• Random-view training reinforces viewpoint conditioning, while a shape discriminator helps keep generated meshes from becoming distorted or unlike real objects.The method enforces silhouette consistency under a random viewpoint and trains the discriminator adversarially; the two techniques balance view awareness and shape quality.Source: §3.1 · pp. 3–4

3.2. Training and Inference

• Feature-space domain adaptation trains an encoder adversarially against a domain discriminator to make synthetic-sketch and hand-drawn-sketch features indistinguishable.The motivation is a performance drop when training on synthetic sketches and testing on hand-drawn sketches.Source: §3.2 · p. 4
• Training combines silhouette, geometry-regularization, viewpoint-prediction, and discriminator losses, using multi-scale progressive silhouettes and mesh-structure regularization.Silhouette comparisons use IoU; higher-resolution silhouettes enter as training progresses, while flatten and Laplacian losses regularize mesh structure.Source: §3.2 · p. 4
• At inference, users can generate a mesh using the predicted viewpoint or provide a specified viewpoint to obtain a different view-conditioned result.Figure 3 compares results for the same sketch under different viewpoints, with projections onto each specified view shown against the input sketch.Source: §3.2 · p. 5, Figure 3 · p. 5

4.1. Datasets

• ShapeNet-Synthetic contains 13 ShapeNet categories rendered from 20 views; ShapeNet-Sketch pairs 1,300 free-hand sketches with 3D models for evaluation, alongside qualitative tests on Sketchy and Tu-Berlin.ShapeNet-Sketch was collected by asking people with different drawing skills to sketch rendered objects on a touch pad.Source: §4.1 · pp. 5–6

4.4.1 Comparisons on ShapeNet-Synthetic Dataset

• Our method surpasses baselines on voxel IoU for most ShapeNet-Synthetic classes with predicted or ground-truth views, and achieves the best overall Chamfer Distance.Ground-truth views serve as the upper bound and predicted views as the lower bound for the view-aware design.Source: Table 1 · p. 6, §4.4.1 · p. 7

4.2. Implementation Details

• The implementation uses ResNet-18 features, SoftRas rendering, fixed camera distance, class-specific models, and elevation and azimuth to represent viewpoints.Comparisons include a SoftRas encoder-decoder and retrieval from nearest training edge maps, evaluated quantitatively with voxel IoU.Source: §4.2 · p. 6, §4.4 · p. 7

4.3. Effectiveness of View-Aware Design

• Changing elevation and azimuth changes generated geometry to match the sketch: azimuth selects van versus truck, while elevation affects table height and top thickness.The paper also reports that random-view training and a shape discriminator together yield view-aware, promising-quality shapes.Source: §4.3 · p. 6

4.4.2 Comparisons on ShapeNet-Sketch Dataset

• Domain adaptation achieves the highest ShapeNet-Sketch IoU, with significant improvements on airplane, car, and sofa; projected silhouettes also match sketches better than baselines.CycleGAN-generated fake sketches did not significantly improve results.Source: Table 3 · p. 7, §4.4.2 · p. 7, Table 4 · p. 8

4.4.1 Comparisons on ShapeNet-Synthetic Dataset

• Without random-view reinforced training, the model fails on ground-truth views; without a shape discriminator, inaccurate predicted views can produce distorted meshes.Multi-scale progressive training further improves performance, and all three strategies achieve the highest IoU in most cases with or without ground-truth views.Source: Table 2 · p. 6, §4.4.1 · p. 7

4.4.3 Evaluations on Free-Hand Sketches

• On Sketchy and Tu-Berlin sketches, generated mesh boundaries align better with inputs than baseline methods, while balancing sketch alignment and shape realism on out-of-distribution shapes.These results use predicted viewpoints; users can specify new viewpoints to control output shapes.Source: Figure 6 · p. 8, §4.4.3 · p. 8

• The view-aware architecture conditions generation on viewpoints, improving generation quality and giving users control over output shapes.Source: §5 · p. 8
• Across several free-hand sketch datasets, the method balances user intentions with shape quality, especially for poorly drawn sketches.Source: §5 · p. 8
• The authors hope the view-aware paradigm will inspire further research in single-view 3D reconstruction and sketch-related areas.Source: §5 · p. 8