Learning to generate line drawings that convey geometry and semantics

1. Introduction

• Photograph-to-drawing methods face scarce, difficult-to-scale human-created datasets, while geometry and meaning are challenging to define computationally.Photographs also often lack ground-truth geometry and contain complex scenes with multiple subjects and interactions.Source: §1 · p. 1
• The method treats line drawings as encodings and guides unpaired translation with objectives that decode depth, semantics, and appearance.Appearance preservation follows from cycle consistency [47].Source: §1 · pp. 1–2
• The authors apply their unsupervised line-generation method across styles and analyze conveyed geometry and semantics, baseline comparisons, and ablations.They present the method as explicitly instilling geometry and semantic information in drawings.Source: §1 · p. 2

2. Related Work

• Geometry-based methods draw from 3D models but cannot handle arbitrary photographs without 3D geometry, and most produce only one style.Neural Strokes addresses the style constraint; this method instead targets stylized drawings from 2D photographs that convey 3D geometry.Source: §2 · p. 2
• Supervised 2D approaches use stroke or vector data, differentiable drawing and painting, or paired images often collected for specific tasks [18] [25] [29] [3] [23] [37] [77] [37] [64] [52].The proposed method instead translates between sketch domains using unpaired data.Source: §2 · p. 2
• UPDG also translates unpaired data, but specializes in portraits; this method targets arbitrary photographs with geometry- and semantics-focused losses.Those objectives reduce reliance on cycle consistency or appearance reconstruction.Source: §2 · p. 2
• CLIPDraw [24] matches drawings to text prompts, whereas this method conditions on photographs and generates multiple styles using CLIP features.Unlike CyCADA, which uses a pretrained classifier for domain adaptation, this method uses CLIP embeddings to describe complex scenes.Source: §2 · p. 2
• Unpaired translation methods handle shape-changing domains but can produce sparse line drawings missing key strokes [40] [42] [11] [22] [40] [1] [12] [45] [47] [36] [41].The paper distinguishes its goal of drawings that communicate geometry and semantics from general image translation.Source: §2 · p. 2

3. Method

• Unpaired translation maps photographs in domain A to line drawings in a chosen style domain B, guided by objectives assessing communicated geometry and semantics.The setup adds these information-focused objectives to appearance preservation through cycle consistency.Source: §3 · pp. 2–3
• Adversarial training matches generated drawings to the target style, while CLIP, appearance, and geometry losses target semantic, appearance, and geometric information.The method uses generator and discriminator networks for both domains.Source: Figure 2 · p. 3, §3 · p. 3

3.1. Losses

• To avoid encouraging imperceptible depth signals, the method predicts depth from image features shared by photographs and drawings using a pretrained depth network.It pretrains GGeom on ImageNet features and then fine-tunes it during line-drawing generation; using the pretrained photograph-depth network directly on drawings is unsuitable because of the domain gap.Source: §3.1 · p. 3
• The geometry loss compares drawing-derived depth predictions with pseudo-ground-truth depth maps predicted for the input photograph.The pseudo-ground-truth maps are needed during training, not at test time; the method uses a pretrained depth network because ground-truth depth is usually unavailable.Source: §3.1 · p. 3
• The semantic loss minimizes the distance between CLIP embeddings of the photograph and its generated drawing, while cycle consistency preserves appearance.The shared visual-text CLIP embedding is used to represent semantic information from entire scenes.Source: §3.1 · p. 4

3.2. Full Objective

• The full objective combines CLIP, geometry, adversarial, and cycle-consistency losses with respective weights 10, 10, 1, and 0.1.The implementation uses an encoder-decoder generator with Res-Net blocks and a patch-based discriminator.Source: §3.2 · p. 4

4. Experiments

• Training uses 10,000 COCO photographs and separate drawing datasets; quantitative evaluation covers Contour Drawings and Anime Colorization, with additional qualitative styles.The experiments compare against unpaired image-translation methods and include portrait comparisons with specialized methods.Source: §4.1 · p. 4, §4 · p. 4

4.1. Line Drawings from Photographs

• Users overwhelmingly preferred the method over unpaired translation baselines for photographs, and comparisons showed more detailed drawings with recognizable features and boundaries.Figure 4 compares methods trained on the same data in two styles; the user study reports preferences across all cases.Source: §4.1 · pp. 4–5, Table 1 · p. 5, Figure 4 · p. 6
• The CLIP loss was essential across styles, depth loss mattered more for Contour Drawings than Anime, and appearance loss improved results slightly.The ablation comparison reports the percentage of user choices favoring the full method over versions without each loss.Source: §4.1 · p. 5, Table 2 · p. 6
• Relative-depth judgments favored the method over CycleGAN in both evaluated styles, with a larger improvement for Contour Drawings than Anime.The depth loss greatly improves relative-depth predictions and drawing quality for Contour Drawings, while its effect is smaller for Anime.Source: Table 3 · p. 6, §4.1 · p. 6
• Users produced more accurate descriptions and found drawings more recognizable with the method than with comparison methods.The evaluation compares caption embeddings by cosine similarity and reports the percentage of images users could not identify.Source: Table 4 · p. 6, §4.1 · p. 7

4.2. Line Drawings from Portraits

• For portraits, users preferred supervised APDrawingGAN over the method trained on APDrawings, but found the Helen-trained method preferable or comparable; users slightly preferred it for style 1.Qualitative portrait quality also improved substantially when training on Helen rather than APDrawings.Source: §4.2 · p. 7

Results

• On APDrawings test data, the approach produces accurate, well-formed drawings when trained on APDrawings or Helen in the APDrawings style.The perceptual-study table describes comparisons with baselines, but the supplied passage gives no user-preference values.Source: Figure 6 · p. 8, Table 5 · p. 8

Loss formulations

• Depth prediction was chosen over normal maps because normal estimates are often noisy for photographs and depth predictions are robust.The authors identify novel-view prediction and other 3D approaches as future directions.Source: §5 · pp. 7–8

Limitations

• The method relies on pseudo-ground-truth depth from a pretrained network, so its drawings inherit that network’s failure cases and biases.The authors also note failures under some lighting and texture conditions, and that the CLIP loss can sometimes produce grayscale-photo-like results.Source: §5 · p. 8

Negative impacts

• The method can learn training-data bias and may reflect sensitive content present in artistic datasets.The Anime sketch dataset is described as containing mostly feminine subjects; the full Anime dataset may contain nudity or weapons.Source: §5 · p. 8

Conclusion

• The paper frames line drawings as encodings of scene geometry, semantics, and appearance, evaluated through depth prediction, CLIP features, and image reconstruction.Its conclusion is that this approach creates compelling line drawings from unpaired data.Source: §5 · p. 8

• Figure 7 shows the input photographs and three model variants above their predicted depth maps: without semantic loss, without depth loss, and the full model.The figure caption does not state a comparative outcome.Source: Figure 7 · p. 13

6.1. More Results

• The OpenSketch and Cole et al. methods tend to focus on image centers and sometimes omit lines across four additional drawing styles.The passage identifies these tendencies but does not provide a direct outcome comparison for the additional styles.Source: §6.1 · p. 13

6.1.1 Depth Reconstruction

• Depth predictions from line drawings are coarse but often capture key elements and relative scene depth, though boundary lines can be mistaken for close objects.Depth is less useful when objects are all very close or very far from the camera; adding semantic loss increases predicted depth-map quality. Table 6 reports MSE for ablations, and the text says depth loss improves contour-style maps while anime already portrays depth well without it.Source: §6.1.1 · p. 13, Table 6 · p. 13

6.1.2 Painterly styles

• For Cezanne-style paintings, the method, CycleGAN, and U-GAT-IT handle the style well and produce comparable results.The authors note that geometry and semantics loss works best with sparser styles.Source: §6.1.2 · p. 13, Figure 8 · p. 13

6.1.3 Portraits in Style 1

• In the style-1 portrait comparison, both the proposed method and UPDG create what the authors describe as nice portraits.The passage supplies no quantitative comparison.Source: §6.1.3 · p. 13

• DeepLabv3 features produced confusing drawings, while Inception v3 and CLIP produced reasonable ones; CLIP was chosen for better handling of lighting, textures, backgrounds, and faces.Source: §6.2 · p. 13

Generators and Discriminators

• GA and GB use encoder-decoders with three Res-Net blocks, while their discriminators are 70 × 70 PatchGANs.The networks are based on architectures from [42] and [40], respectively.Source: §6.3 · p. 13

Comparison with UPDG

• Unlike portrait-specific UPDG [83], this method applies to arbitrary photographs; both approaches often produce nice drawings of the subject.The comparison used portrait photographs and line art style 1, though the training data were not exactly matched.Source: Figure 9 · p. 14

Depth Networks

• The pseudo-ground-truth depth maps come from pretrained system F, which is built around MiDaS [69].F is presented by Miangoleh et al..Source: §6.3 · p. 14
• GGeom translates ImageNet features into depth maps and is pretrained on features from real photographs.Its architecture is based on the Global Generator from pix2pixHD.Source: §6.3 · p. 14, Table 7 · p. 14

Image Features

• 33.5% nearest-neighbor accuracy for line drawings comes from Mixed 6b features, versus 24.5% with penultimate-layer features.The reported comparison uses Inception v3 features.Source: Table 8 · p. 14, §6.3 · p. 14

Semantic Networks

• The semantic constraint uses pretrained CLIP with a ViT-B/32 vision transformer [21].Source: §6.3 · p. 14

Photograph Datasets

• The photograph training set is a randomly selected 10,000-image COCO subset, and MIT-Adoke 5k is used for evaluation.Source: §6.4 · p. 14

Line Drawing Datasets

• The Contour Drawings dataset contains 5,000 scene-boundary annotations, and the study uses drawings with width 5.Source: §6.4 · p. 14
• The Anime Sketch Colorization database has 14,224 color-drawing/sketch pairs; the study uses sketches and selected 2,256 images without sensitive content for release.Source: §6.4 · pp. 14, 16
• OpenSketch contains 420 product-design sketches of 12 objects, while Cole et al. provide 207 artist sketches used to study where people draw lines.Source: §6.4 · pp. 16–17

Portrait Datasets

• APDrawings provides 140 portrait-line drawing training pairs, or 420 after rotational augmentation, and 70 test pairs.Its portraits are centered with similar poses and lighting, despite the dataset's high image quality.Source: §6.4 · p. 17
• Helen contains 2,000 training and 330 test portrait photographs, supporting evaluation across varied poses, lighting, and image conditions.Source: §6.4 · p. 17
• For style 1, the authors trained with Helen and 88 Charles Burns line-art portraits because they lacked both full training sets.They describe this comparison with UPDG as not exact.Source: §6.4 · p. 17

Human subjects and User studies

• All user studies used Amazon Mechanical Turk and had IRB approval; participants were told responses could appear in papers and meetings, no personal information would be stored, and they could withdraw.Source: §6.5 · p. 17
• Most comparisons collected 1,000 judgments, while APDrawings comparisons collected 1,200; participants viewed up to 100 or 70 images, respectively.The APDrawings test set contains 70 images.Source: §6.5 · p. 17