PaperlayerView exact revision on arXiv

Source-linked AI summary

Free-hand Sketch Synthesis with Deformable Stroke Models

Yi Li, Yi-Zhe Song, Timothy Hospedales, Shaogang Gong

arXiv:1510.02644v1cs.CV

TL;DR

Free-hand sketch synthesis beyond faces has been limited by category-specific models and aligned training data. This paper learns deformable stroke models from sketches and fits them to images; its evaluations report plausible free-hand outputs across categories, with some limits on difficult cases.

  • Problem

    Existing free-hand synthesis largely targeted faces and relied on aligned, purpose-built image-sketch data, leaving synthesis across diverse categories less explored.

    Source: §1 · p. 1

  • Method

    The model learns category-level deformable stroke models from sketch collections, then fits free-hand strokes to images through detection.

    Source: §1 · p. 1

  • Results

    The system generates plausible free-hand sketches resembling input images across several categories with diverse appearance and structural variation.

    Source: §5 · p. 12

  • Takeaways & Limitations

    The learned model supports image-to-sketch synthesis beyond a single object category using public sketch datasets without required image-sketch pairs or alignment.

    Source: §7 · p. 16

    Extreme topology diversity in training sketches can yield a poor model, and unsupervised learning struggles with objects with complex inner structure.

    Source: §6 · p. 16

Abstract

from arXiv · show

We present a generative model which can automatically summarize the stroke composition of free-hand sketches of a given category. When our model is fit to a collection of sketches with similar poses, it discovers and learns the structure and appearance of a set of coherent parts, with each part represented by a group of strokes. It represents both consistent (topology) as well as diverse aspects (structure and appearance variations) of each sketch category. Key to the success of our model are important insights learned from a comprehensive study performed on human stroke data. By fitting this model to images, we are able to synthesize visually similar and pleasant free-hand sketches.

1 Introduction

Existing free-hand sketch synthesis was largely tailored to faces and relied on aligned, purpose-built training data. This work learns deformable stroke models from sketch collections and uses them to synthesize sketches across diverse categories.

1 Introduction

  • Earlier sketch-synthesis systems largely targeted faces and relied on aligned image-sketch data, limiting their extension to broader categories and free-hand styles.The proposed approach instead uses public sketch datasets without alignment or part labels and fits strokes to input images.Source: §1 · pp. 1–2
  • The learned DSM guides image-to-sketch synthesis by fitting free-hand strokes to an image, capturing its structural and appearance variation.The model balances matching an image edge map against a prior supplied by the learned sketch model.Source: §1 · pp. 1–2
  • With a few similar-pose sketches, the iterative process discovers category parts and learns a DSM encoding shared topology plus structural and appearance variation.Perceptual grouping, category-level DSM learning, and DSM-guided grouping are iterated until convergence.Source: §1 · p. 2, Figure 1 · p. 2
  • A stroke-data study examines how stroke length relates to semantics and how reliable temporal order is, informing a grouper that combines stroke-specific cues.The grouper accounts for stroke length and temporal drawing order alongside other cues.Source: §1 · p. 2
  • Evaluation uses six non-expert TU-Berlin categories and professional Disney portrait sketches at two abstraction levels, with user studies and experiments.The TU-Berlin categories are horse, shark, duck, bicycle, teapot, and face.Source: §1 · p. 3

2 Related work

Prior sketch-synthesis work spans style-oriented rendering, edge-map conversion, and data-driven face synthesis, while related models offer structural tools with trade-offs in supervision and fit to free-hand sketches.

2 Related work

  • Earlier synthesis approaches broadened style or converted images to edge maps, while data-driven methods focused on faces and assumed aligned professional image-sketch pairs.Face systems used holistic exemplars or patch-level mappings (Chen et al (2002); Liang et al (2002); Wang and Tang (2009); Wang et al (2012)).Source: §2 · p. 3
  • Berger et al (2013) used professional portrait strokes and artist-specific libraries, but relied on a manually fitted face mesh that complicates extension beyond faces.Their synthesis converted image edges to vector curves and replaced them with strokes matched by shape, curvature, and length.Source: §2 · p. 3
  • Contour models learn parts from edge fragments but can duplicate or omit parts, impose rigid structure, and miss local appearance variation; pictorial structures require intensive labeling.Pictorial structures model mandatory parts and spatial relations using a minimum spanning tree and a statistical MAP framework.Source: §2 · p. 3
  • The proposed DSM combines pictorial-structure learning and detection with perceptual grouping, iterative learning, and multiple stroke exemplars per part.This combination targets sophisticated sketch topology, effective stroke-to-edge registration, and local appearance variation.Source: §2 · p. 3
  • Prior stroke studies examined ordering, importance, abstraction, or style, while this work emphasizes stroke semantics and incorporates temporal order into grouping.Earlier grouping work used temporal order as a soft constraint or hypothesis; the authors say they provide evidence supporting its use (Huang et al (2014)).Source: §2 · p. 4

3 Stroke analysis

The stroke analysis finds that stroke length is informative about semantic parts, but stroke order lacks a consistent global hierarchy. Local runs of strokes can still correspond to individual parts, while length-based grouping helps form meaningful semantic strokes.

3 Stroke analysis

  • Medium strokes tend to represent semantic parts, while short strokes are often too small and long strokes can span multiple parts.Because medium strokes are relatively scarce, the analysis motivates grouping short strokes into chains and cutting very long strokes into segments.Source: §3 · pp. 4–5
  • Across six non-expert categories, stroke-length distributions are long-tailed, with most strokes under 1000 pixels and few exceeding 2000 pixels.Source: §3 · p. 4, Figure 2 · p. 4
  • In professional sketches, stroke lengths decrease substantially at the more abstract 90s level, making grouping more necessary for extracting semantic parts.At the 30s level, most semantic strokes fall in the middle length range, while short strokes remain the largest group.Source: §3 · p. 5
  • Stroke lengths appear in no consistent global drawing hierarchy, but consecutive local strokes often depict one semantic part.The ordering of those parts appears random.Source: §3 · p. 5

4 A deformable stroke model

The paper learns a deformable stroke model from sketches of similar poses, using semantic parts as its building blocks. The model represents part clusters and their spatial relations, while the grouping scale affects whether parts are fragmented or improperly combined.

4 A deformable stroke model

  • A deformable stroke model is learned from similar-pose sketches, with perceptual grouping, model learning, and matching or stroke labeling performed interchangeably.The framework also uses the learned model to detect on sketches and images.Source: §4 · p. 6

4.1 Model definition

  • The DSM is an undirected graph whose vertices are category-level semantic-part clusters, with edges linking parts whose locations are closely related.Part exemplars represent each cluster, and connected clusters encode spatial relations.Source: §4.1 · p. 6
  • At λ = 1500, horse tails and back legs group better than at λ = 500, while λ = 3000 can improperly merge distinct parts.For clearly drawn horses, the stroke-length term has less influence on grouping.Source: Figure 6 · p. 7

4.2 Perceptual grouping

Perceptual grouping builds semantic parts from raw strokes using geometric, shape, length, temporal-order, and—after the first iteration—model-label cues. The resulting groupings usually correspond across sketches, while variations such as shark tails and lips remain for iterative learning to address.

4.2 Perceptual grouping

  • The pairwise grouping error combines proximity, continuity, similarity, stroke length, local temporal order, and model labels from the second iteration onward.Proximity, continuity, and length raise error; similarity lowers it, while temporal order and model labels further adjust it.Source: §4.2 · pp. 7–9
  • Similarity helps group repetitive short strokes that form a semantic part even when continuity would separate them.The method compares stroke shape-context descriptors to calculate the similarity bonus.Source: §4.2 · p. 8, Figure 7 · p. 8
  • Local stroke temporal order adjusts grouping error according to how close two strokes are in drawing order.The paper illustrates corrections to beak and feet groupings when temporal order is used.Source: §4.2 · p. 8, Figure 8 · p. 8
  • First-iteration groupings show reasonable part correspondence, but differ on parts such as shark tails and lips across sketches.The authors attribute these disagreements to different ways of drawing the same semantic stroke; iterative learning addresses such variations.Source: §4.2 · p. 9

4.3 Model learning

Model learning clusters grouped semantic strokes into category-level parts, then learns stroke exemplars and a tree of spatial relations to form the deformable stroke model.

4.3 Model learning

  • Semantic strokes from training sketches are pooled and spectrally clustered into category-level groups with shared appearance and geometry.Mirrored poses are flipped to the same direction before pooling.Source: §4.3 · p. 9

4.3.1 Spectral clustering on semantic strokes

  • Spectral clustering uses an affinity combining shape-context matching and the distance between semantic-stroke centers, scaled by their local scales.The number of clusters is set by the mean number of semantic strokes produced per sketch.Source: §4.3.1 · p. 10

4.3.2 Model parameter learning

  • The model parameters include representative stroke exemplars and spatial relations arranged as a minimum spanning tree learned from semantic-stroke clusters.Exemplars are chosen by average shape-context matching cost; spatial parameters are learned from clusters rather than training images.Source: §4.3.2 · p. 10

4.4 Model matching

Model matching registers a deformable stroke model to an image by sampling candidate configurations and refining them through energy minimization. The final detection is the configuration with the lowest refined cost, followed by a small local aesthetic adjustment.

4.4 Model matching

  • Fast directional chamfer matching supports model configuration sampling and energy minimization for stroke registration to sketches or images.The framework uses automatic matching during both iterative training and image-to-sketch synthesis.Source: §4.4 · p. 11

4.4.1 Configuration sampling

  • Each sampled configuration selects one stroke exemplar per cluster and assigns it a location, balancing image-edge appearance against model-prior geometry.Candidate exemplar locations are found with chamfer matching within a region based on the cluster's average bounding box.Source: §4.4.1 · p. 11

4.4.2 Energy minimization

  • Dynamic programming refines each configuration using appearance and deformation costs, and the lowest-cost refined configuration becomes the final detection.A further local energy-minimization pass turns off appearance costs and shifts strokes within a small region to correct placements attributed to edge noise.Source: §4.4.2 · pp. 11–12

4.5 Iterative learning

Iterative learning alternates perceptual grouping, model learning, and matching on training sketches. The authors select the iteration with the smallest variance in semantic-stroke counts to train the final model.

4.5 Iterative learning

  • The process alternates perceptual grouping, model learning, and matching on training data, using model-assigned labels to guide the next grouping iteration.Raw strokes are labeled by overlap or distance to stroke exemplars, and an overly long stroke can be cut to fit multiple exemplars.Source: §4.5 · p. 12
  • The variance of semantic-stroke counts decreases over iterations, and the model is trained from the iteration with the smallest variance.The authors also report that strokes form medium-length groups with more appropriate semantics as variance converges.Source: §4.5 · p. 12

4.6 Image-sketch synthesis

After learning the final deformable stroke model (DSM), the system matches it to an image edge map to synthesize a sketch, assuming an approximate object bounding box and a category selected in advance.

4.6 Image-sketch synthesis

  • Image-edge-map matching applies the learned DSM to synthesize a sketch.The process avoids the localization challenge by assuming an approximate object bounding box is given.Source: §4.6 · p. 12
  • The correct DSM category must be selected before synthesis.The paper describes both this selection and the bounding-box assumption as easy to engineer in practice.Source: §4.6 · p. 12
  • The synthesis stage uses the final DSM obtained through iterative learning.Source: §4.6 · p. 12

5 Experiments

Experiments report plausible free-hand sketches resembling input images across categories and structural variations, with user studies assessing recognizability and image-sketch matching. They also describe the datasets and parameter choices used to evaluate the framework.

5 Experiments

  • Synthesized outputs resemble input images in plausible free-hand style across object categories with diverse appearance and structural variations.The evaluation combines qualitative examples with two user studies.Source: §5 · p. 12
  • Training used six TU-Berlin categories and portrait sketches from two Disney artists at 90s and 30s abstraction levels.Testing used ten images per category, with the face category drawn from the Center for Vital Longevity Face Database; 270s sketches were excluded for computational cost and 15s sketches because many were incomplete.Source: §5 · pp. 12–13
  • By iteration 3, perceptual grouping showed better consensus on horse legs and neck than at iteration 1.The figure attributes the improvement to increased model quality, especially for those parts.Source: Figure 13 · p. 13

5.1 Free-hand sketch synthesis evaluation

  • TU-Berlin results preserve major semantic strokes without missing or duplicated parts while representing configuration and part differences within categories.Examples include varying horse leg configurations, duck feet styles, and teapot body curves.Source: §5.1 · pp. 13–14
  • Professional portrait training sketches yield more precise synthesized faces resembling input photos; hair is too diverse to model well.The paper reports similar results to Berger et al (2013), especially at more abstract levels, while noting that category-specific alignment supervision could improve aesthetic quality, particularly at less abstract levels.Source: §5.1 · pp. 14–15

5.2 Perceptual study

  • Human recognition tests found synthesized sketches semantically recognizable, with performance trends across categories similar to those reported for real sketches.The study used 60 synthesized sketches across six categories; the authors attribute their higher overall performance than Eitz et al (2012) most likely to the smaller study scale.Source: §5.2 · p. 15, Table 1 · p. 15
  • Image-sketch matching accuracy was significantly above random guessing for all categories except duck and Artist E 90s.The weaker TU-Berlin duck and teapot results were attributed mainly to insufficient training-sketch variation; Artist E 90s sketches used many short, detailed strokes and appeared messier.Source: §5.2 · p. 15, Table 2 · p. 15

5.3 Parameter tuning

  • Parameter settings vary with abstraction level: 90s portrait sketches use higher semantic stroke counts and omit the continuity term.For TU-Berlin and 30s portraits, proximity, continuity, and stroke-length weights are each 0.33; for 90s portraits, proximity and stroke-length weights are each 0.5.Source: §5.3 · p. 16

6 Further discussions

The discussion identifies limits tied to training-data topology, training-example coverage, and the model's treatment of noise and aesthetic detail.

6 Further discussions

  • Extreme diversity in training sketches' part existence, number, or layout can yield a poor model.Selecting fine-grained sketch subcategories could alleviate this, but would require more constrained training-sketch collection.Source: §6 · p. 16
  • Synthesis quality depends strongly on training data; without a roughly similar training sketch, producing a good-looking result for an input can be difficult.The paper gives specially shaped teapot images as an example.Source: §6 · p. 16
  • Severe noise can affect detection accuracy, and the system uses only basic aesthetic-refinement post-processing.The discussion also notes that unique parts such as a horse's saddle are not shared across sketches and that advanced NPR rendering could improve aesthetic quality.Source: §6 · p. 16

7 Conclusion

The system extends free-hand sketch synthesis beyond a single object category using publicly available sketch datasets. From a few similarly posed sketches, it learns common semantic parts and their structural and appearance variations, then fits the model to an image to generate a resembling sketch.

7 Conclusion

  • The system works beyond a single object category and uses publicly available sketch datasets drawn by non-experts or professionals.Source: §7 · p. 16
  • With a few similarly posed sketches and minimum supervision, the model discovers category-common semantic parts and encodes their structural and appearance variations.Training does not require paired photo-sketch images or aligned sketches.Source: §7 · p. 16
  • Fitting the learned model to an input image automatically generates a free-hand sketch that resembles it.The authors state that results from the preceding section confirm the model's efficacy.Source: §7 · p. 16
Loading 1510.02644v1…