Source-linked AI summary
Learning Deep Sketch Abstraction
Umar Riaz Muhammad, Yongxin Yang, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales
TL;DR
Free-hand sketches vary in abstraction, complicating recognition and photo matching. The paper learns a stroke-removal model with reinforcement learning to balance compactness and recognizability. It applies the model to sketch analysis and synthesis, including photo-only FG-SBIR training.
Problem
Free-hand sketches vary substantially in abstraction, making recognition and matching sketches to photos challenging.
Source: §1 · p. 1Method
A reinforcement-learning model predicts which strokes can be removed while preserving recognizability, using classification-rank rewards to train its stroke-removal policy.
Source: §1 · p. 1, §3.1.4 · p. 4Results
Takeaways & Limitations
Photo-only FG-SBIR training could scale to arbitrary object categories when sufficient photos are available.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Human free-hand sketches have been studied in various contexts including sketch recognition, synthesis and fine-grained sketch-based image retrieval (FG-SBIR). A fundamental challenge for sketch analysis is to deal with drastically different human drawing styles, particularly in terms of abstraction level. In this work, we propose the first stroke-level sketch abstraction model based on the insight of sketch abstraction as a process of trading off between the recognizability of a sketch and the number of strokes used to draw it. Concretely, we train a model for abstract sketch generation through reinforcement learning of a stroke removal policy that learns to predict which strokes can be safely removed without affecting recognizability. We show that our abstraction model can be used for various sketch analysis tasks including: (1) modeling stroke saliency and understanding the decision of sketch recognition models, (2) synthesizing sketches of variable abstraction for a given category, or reference object instance in a photo, and (3) training a FG-SBIR model with photos only, bypassing the expensive photo-sketch pair collection step.
1. Introduction
Free-hand sketches vary widely in abstraction, complicating recognition and photo matching. The paper models abstraction as balancing recognizability against stroke count and applies the model to sketch analysis tasks.
1. Introduction
- Sketches of the same category or instance differ in abstraction, making recognition and photo matching difficult; abstraction also varies with task and individual preference.The section frames this variation as expected in human sketching.Source: §1 · p. 1
- The proposed abstraction model treats sketch abstraction as a trade-off between recognizability and compactness, seeking the smallest recognizable subset of input strokes.The authors present it as the first model of deep sketch abstraction.Source: §1 · p. 1
- An RNN with B-GRU and moving-window MLP context models stroke-removal decisions sequentially, trained with reinforcement learning and classification-rank rewards.Removal effects depend on which other strokes are kept or removed.Source: §1 · p. 2
- The model supports stroke-saliency estimation, abstraction control for category-level sketches, photo-to-sketch synthesis, and photo-only FG-SBIR training.For photo-to-sketch synthesis, an edge-map is abstracted into a free-hand-style sketch.Source: §1 · p. 2
- The authors report controllable sketch synthesis and photo-to-sketch synthesis, and a competitive FG-SBIR model trained without photo-sketch pairs.They note this could scale to arbitrary object categories when sufficient photos are available.Source: §1 · p. 2
2. Related Work
Prior work spans free-hand sketch recognition, sketch synthesis, photo-to-sketch translation, and sketch-based image retrieval. This paper positions its abstraction model as a distinct approach to stroke-level abstraction and as a way to address FG-SBIR data and matching challenges.
Sketch recognition
- Earlier sketch recognition work progressed from hand-crafted features to CNNs, while this paper uses a sketch recognizer to reward abstraction and a stroke-order-aware RNN classifier.The RNN approach contrasts with treating sketches like static photos.Source: §2 · p. 2
Category-level and instance-level sketch synthesis
- Prior category-level synthesis used a sequence-to-sequence VAE; this work combines an unconditional synthesizer with abstraction control.For instance-level synthesis, it frames photo-to-sketch generation as abstraction, unlike prior work focused on sketch-to-photo synthesis.Source: §2 · p. 2
Sketch based image retrieval
- FG-SBIR requires matching a sketch to the same photo instance; collecting paired training data is costly, and abstraction variation complicates matching.The authors address both problems with sketch abstraction and photo-to-sketch synthesis.Source: §2 · p. 3
Visual abstraction
- Earlier face-sketch abstraction replaced edge-map edges with similar artist strokes, whereas this paper models abstraction as progressively removing less important parts to trade compactness for recognizability.The authors distinguish this model-based approach from prior visual-abstraction methods in the photo domain.Source: §2 · p. 3
3.1. Sketch abstraction
The section represents sketches as coordinate sequences and formulates abstraction as sequential keep-or-skip decisions trained to retain recognizability while reducing stroke-segments. A contextual RNN agent uses classifier-derived rewards and shifted action probabilities to control abstraction.
3.1.1 Sketch representation
- Sketches use coordinate sequences (Δx, Δy, p), with each stroke-segment grouping five consecutive coordinates.Each stroke contains a variable number of stroke-segments.Source: §3.1.1 · p. 3
3.1.2 Problem formulation
- An agent processes stroke-segments sequentially, choosing whether to keep or remove them to compactly encode a recognizable sketch.Reinforcement learning seeks a policy maximizing expected discounted cumulative reward.Source: §3.1.2 · p. 3
3.1.3 Model
- A B-GRU encodes sketch segments, while a moving-window MLP predicts keep-or-skip actions and the environment updates the sketch after each action.The model schematic is shown in Figure 2.Source: §3.1.3 · p. 4, Figure 2 · p. 4
3.1.4 Reward scheme
- Classifier-based rewards favor skipping segments while rewarding recognizable outputs; ranked rewards also value improving the ground-truth class rank during processing.The basic scheme gives positive reward for skipping, negative reward for keeping, and a large final reward or penalty according to recognition.Source: §3.1.4 · p. 4
3.1.5 Training procedure
- Policy-gradient training updates the agent toward greater expected return, accumulating trajectories in a buffer and updating parameters every N trajectories.The policy is sampled and parameters are adjusted by gradient ascent.Source: §3.1.5 · p. 5
3.1.6 Controlling abstraction level
- Abstraction control shifts the agent’s action probabilities by δ, biasing outputs toward skipping or keeping segments to obtain different abstraction levels.Varying softmax temperature was empirically unsatisfactory, so the authors use the probability shift instead.Source: §3.1.6 · p. 5
3.2. Sketch stroke saliency
The agent’s output distribution provides stroke saliency values that indicate which strokes it learns to retain for recognition or ranking accuracy.
3.2. Sketch stroke saliency
- The trained agent’s output distribution is used to compute a saliency value for each stroke.Saliency is defined on a scale from 0 to 1.Source: §3.2 · p. 5
- A stroke’s score aggregates the agent’s keep probabilities over the time steps covering its stroke-segments.The first and last corresponding time steps define that span.Source: §3.2 · p. 5
- Strokes the agent learns are important for recognition or ranking accuracy receive greater saliency.Source: §3.2 · p. 5
3.3. Category-level sketch synthesis
The category-level synthesis approach applies the abstraction model to sketches generated by a VAE RNN synthesizer, producing versions at desired abstraction levels.
3.3. Category-level sketch synthesis
- The approach combines the abstraction model with the VAE RNN category-level sketch synthesizer [15].Source: §3.3 · p. 5
- The synthesizer first generates a sketch for the requested category.Source: §3.3 · p. 5
- The abstraction model then produces different versions of that input sketch at the desired abstraction level.Source: §3.3 · p. 5
3.4. Photo to sketch synthesis
The photo-to-sketch pipeline vectorizes distorted photo edges, simplifies them, and feeds them to a fine-tuned abstraction model to generate abstract sketches.
3.4. Photo to sketch synthesis
- The pipeline extracts a photo edge-map and vectorizes it without thresholding away noisy edges.A line-tracing algorithm converts the raster edge-map to vector form.Source: §3.4 · p. 5
- Global and stroke-level distortions roughen the vectorized contours to resemble less-smooth human sketches.Global changes include rotation, translation, rescaling, and skew; stroke-level changes include translation and curvature jitter.Source: §3.4 · p. 5
- Fixed-length resampling simplifies the distorted edges before a model fine-tuned on preprocessed edge-maps generates the abstract sketch.Source: §3.4 · p. 5
3.5. Fine-grained SBIR
The photo-only FG-SBIR pipeline turns each training photo into simplified edge maps and sketches at multiple abstraction levels, then trains a triplet-ranking model. At test time, it abstracts real sketches at multiple levels and fuses the model’s scores.
3.5. Fine-grained SBIR
- Photos alone are converted into simplified edge maps and sketches at three abstraction levels, creating synthetic photo-sketch training pairs.The three abstraction levels use δ values of −0.1, 0.0, and +0.1.Source: §3.5 · pp. 5–6
- A three-branch Siamese CNN learns joint photo-sketch embeddings with triplet ranking, placing each query sketch closer to its positive than negative photo.The triplet loss uses Euclidean distance and a required margin between positive and negative distances.Source: §3.5 · p. 6
- Training uses the photo that generated a sketch as its positive and a training photo of a different object as its negative.Source: §3.5 · p. 6
- At test time, the gallery contains unseen object instances, and each real free-hand query is paired with three abstracted variants for score-level fusion.The four query sketches are fed to the trained model.Source: §3.5 · p. 6
4. Experiments
Experiments evaluate stroke-level abstraction on QuickDraw sketches, use it to control category- and photo-to-sketch synthesis, and test photo-only training for fine-grained retrieval. Results include preserved recognizability after stroke removal, qualitative advantages over a synthesis baseline, and competitive retrieval performance without paired training data.
4.1. Sketch abstraction
- At abstraction level 3, 70.40% recognition after about 40% of segments were removed exceeded random removal by 6.20 percentage points (64.20%).The original test sketches averaged 64.79 data segments; the model reduced this to 39.48 at level 3.Source: §4.1 · p. 6, Table 1 · p. 7
- Stroke-saliency maps highlight distinctive category features, such as shoe contours over laces and face contours and eyes over less consistent features.The maps are presented on input sketches alongside three model-generated abstraction levels.Source: §4.1 · pp. 6–7, Figure 5 · p. 7
4.2. Sketch synthesis
- Abstracted category-level synthesized sketches remained highly recognizable at the most abstract level and outperformed sketches with random segment removal.The experiment combines a synthesizer trained for each of nine categories with the abstraction model.Source: Table 2 · p. 7, §4.2 · p. 7
4.3. Photo to sketch synthesis
- For photo-to-sketch synthesis, the model produced more visually appealing examples than the [36] encoder-decoder baseline, whose outputs were blurry and appeared to suffer from mode collapse.The authors attribute the baseline's shortcomings to domain gaps and photo-sketch misalignment, and favor vectorized coordinates over pixel-matrix treatment.Source: Figure 7 · p. 8, §4.3 · p. 8
4.4. Fine-grained SBIR
- On Chair, FG-SBIR trained with the proposed synthesized sketches was only 7.12% lower on Top 1 accuracy than the annotated-pair upper bound and decisively beat the [36] synthesis baseline.The comparison also found an advantage over using simplified edge-maps directly as human-sketch replacements.Source: Table 3 · p. 8, §4.4 · p. 8, Figure 8 · p. 8
4.5. Human Study
- In a 10-user study, model-abstracted sketches were preferred to randomly thinned sketches in 64.3% of paired comparisons on average.The reported mean was 64.3 ± 4.59%; individual user results ranged from 58% to 70%.Source: §4.5 · p. 8
5. Conclusion
The paper introduces stroke-level sketch abstraction that predicts which strokes can be removed while retaining recognizability. It uses a rank-based reinforcement-learning reward and demonstrates applications including photo-only FG-SBIR training; extending to edge-maps in the wild and end-to-end variable-abstraction sampling remain future work.
5. Conclusion
- The model predicts which sketch strokes can be safely removed without affecting overall recognizability, using reinforcement learning with a rank-based reward.The paper demonstrates applications including FG-SBIR training with photos only.Source: §5 · p. 8
- Future work includes extending the model to edge-maps in the wild and developing end-to-end abstraction that directly samples sketches at variable levels.Source: §5 · p. 8