CLIPasso: Semantically-Aware Object Sketching

1. Introduction

• The paper asks whether computer renderings can imitate the human process of turning a concrete photograph into an abstract sketch, a task more difficult than producing realistic sketches with geometric operations.The challenge is to select features that capture the object’s essence in a minimal but descriptive rendering.Source: §1 · pp. 1–2
• CLIPasso generates dataset-free photo-to-sketch abstractions through optimization, avoiding explicit sketch datasets that prior data-driven methods require and whose styles constrain outputs [25] [30].Those methods use datasets of the desired abstraction level to guide sketch form and style.Source: §1 · p. 2
• CLIP supplies semantic understanding while the input photograph grounds sketch geometry; abstract depictions must preserve the subject’s underlying structure and meaning.The paper frames this semantic understanding as more complex than applying geometric operations to an image.Source: §1 · p. 2
• The method represents sketches as thin black B´ezier strokes on white, controls abstraction by stroke count, and optimizes control points with a differentiable rasterizer and CLIP loss.It combines final and intermediate activations of a pretrained CLIP model for geometric and semantic simplification, and uses saliency-guided initialization based on vision-transformer attention maps.Source: §1 · p. 2
• The resulting sketches combine semantic and visual features while remaining minimal and providing category- and instance-level object-recognition clues.The introduction describes abstraction as selecting key visual features while omitting others, as illustrated by Picasso’s progressively simplified bull.Source: §1 · pp. 1–2, Figure 2 · p. 2

Related Work

• Unlike dataset-selected sketch styles, CLIPasso’s optimization can produce multiple abstraction levels without suitable sketch datasets or lengthy retraining.Existing methods choose styles based on datasets, from category-prompted abstract sketches to contour-guided concrete ones.Source: §2 · p. 2

Photo-Sketch Synthesis

• Prior photo-sketch methods rely on sketch-specific datasets and span explicit facial models, deformable strokes [25], and sketch-adapted learning approaches [20] [41] [32].Because sparse, abstract sketches challenge direct domain-translation methods, prior work makes sketch-specific adjustments.Source: §2 · pp. 2–3

Vector Graphics

• Vector sketching uses diverse models, including RNNs [16], BERT [26], Transformers [35], CNNs, GANs, and reinforcement learning [12, 27, 46].Differentiable rendering enables vector synthesis with raster-based losses; CLIPasso uses Li et al.’s method for varied curves, including Bézier curves.Source: §2 · p. 3

Sketches Abstraction

• Prior abstraction methods either replace edge-map curves with artist strokes for portraits or remove recognizable strokes using a classifier trained on nine QuickDraw classes [16] [30].The portrait method needs new datasets for each abstraction level, while the classifier-based method requires fine-tuning for new classes.Source: §2 · p. 3

CLIP-based Image Abstraction

• CLIPDraw [11] optimizes random Bézier curves from text, while CLIPasso adds image-conditioned appearance control through geometric loss and saliency-guided initialization.Tian and Ha [39] use CLIP text or L2 geometry separately, whereas CLIPasso integrates semantic and geometric guidance.Source: §2 · p. 3

3. Method

• Sketches comprise black Bézier strokes whose control-point positions are optimized through a differentiable rasterizer, with stroke count controlling abstraction.Stroke degree, width, and opacity remain fixed during optimization.Source: §3 · pp. 3–4

3.1. Loss Function

• CLIPasso combines a semantic distance between final-layer sketch and image embeddings with an L2 geometric distance between intermediate CLIP activations, weighted by ws = 0.1.The geometric term uses ResNet101 CLIP layers 3 and 4.Source: §3.1 · pp. 4–5

3.2. Optimization

• Both sketch and target image are augmented before CLIP evaluation to discourage adversarial, human-unmeaningful sketches; optimization typically takes about 2000 iterations to converge.Gradients of the loss with respect to stroke parameters are computed with an off-the-shelf gradient-based solver.Source: §3.2 · p. 5

3.3. Strokes Initialization

• Saliency-based initialization significantly improves final sketch quality over random initialization by sampling stroke locations from salient image regions.The map combines CLIP ViT-B/32 attention relevancy with XDoG edges and softmax normalization; examples typically use three initializations and select the lowest-loss result.Source: §3.3 · p. 5

4.1. Qualitative Evaluation

• CLIP guidance without sketch-dataset training lets CLIPasso handle varied input categories without category definitions during optimization.The paper illustrates this robustness with infrequent categories.Source: Figure 9 · p. 5, §4.1 · p. 6
• As stroke count decreases, CLIPasso removes details while retaining defining features, such as a flamingo's pose, neck, and legs from 16 to 4 strokes.Stroke degree and an overlaid brush style also change the output sketch's style.Source: §4.1 · p. 6, Figure 10 · p. 6

4.2. Comparison with Existing Methods

• Compared with Muhammad et al. [30], CLIPasso retained recognizable geometry at 4 strokes; compared with Berger et al., it was more geometrically coherent while allowing abstraction.Berger et al.'s results followed a particular artist's style, but their method was limited to faces and its gathered dataset.Source: §4.2 · p. 6, Figure 11 · p. 6
• Unlike sketch-data-dependent methods, CLIPasso handles varied image categories and abstraction levels by changing stroke count, and can provide both semantic and geometric sketching styles.The horse example captured movement minimally without perfectly aligning its legs as Li et al. [23] did.Source: §4.2 · p. 7
• Unlike CLIPDraw [11], CLIPasso's geometric grounding preserves the subject's overall structure while rendering recognizable parts.The comparison constrained CLIPDraw's primitives to match CLIPasso's strokes; all sketches used 16 strokes.Source: §4.2 · p. 7, Figure 13 · p. 7

4.3. Quantitative Evaluation

• At 8 strokes, sketches reached 95% instance-level recognizability, while 4-stroke category recognition fell to 36%, marking a breaking point.Both category- and instance-level recognizability declined as abstraction increased; at 8 strokes and above, both rates were high.Source: §4.3 · p. 7, Table 2 · p. 8
• With the CLIP classifier, sketches achieved 78% accuracy at 16 strokes and 91% at 32 strokes.The evaluation used CLIP ViT-B/32 with text prompts, distinct from the CLIP model used to train the method.Source: §4.3 · p. 8, Table 3 · p. 8

5. Limitations

• For images with backgrounds, performance declines at higher abstraction levels; an automatic mask can address this, and a remedial loss term is a potential development.The passage proposes including the remedial term within the loss function.Source: §5 · p. 8
• All strokes are optimized simultaneously rather than sequentially, and stroke count must be chosen in advance; a possible extension is to learn the count for each image.Different images may require different stroke counts to reach similar abstraction levels.Source: §5 · p. 8

• The method produces photo-sketches at different levels of abstraction without training on specific sketch datasets.Source: §6 · p. 8
• It generalizes to various categories and copes with challenging levels of abstraction.Source: §6 · p. 8
• Its sketches maintain semantic visual clues that support instance-level and class-level recognition.Source: §6 · p. 8

A. Implementation Details

• The method is implemented in PyTorch [31] with the differentiable “diffvg” rasterizer.Source: A. Implementation Details · p. 11
• Initialization samples stroke starts from a ViT-32/B CLIP attention map multiplied by XDoG edges, then places each curve’s remaining three control points within radius 0.05.The attention map averages the 12 self-attention maps and uses attention between the final class embedding and 49 patches.Source: A. Implementation Details · p. 11
• RandomPerspective and RandomResizedCrop augment sketch and target inputs to CLIP, while unaugmented evaluation and three-seed selection guide optimization.Adam uses learning rate 1; optimization typically converges in about 2,000 iterations, taking 6 minutes on one Tesla V100, though recognizable sketches often appear after 100 iterations. Augmentation reduces susceptibility to adversarial samples and improves sketch quality; U2-Net can remove backgrounds before optimization.Source: A. Implementation Details · p. 11

C. User Study

• The user study generated 100 animal and 40 face sketches for category- and instance-level tests, sampling five images per animal class and five male and five female portraits.It used five animal classes—cat, dog, elephant, giraffe, and horse—SketchyCOCO [13], and faces from; five questionnaires each included 20 category-level and 28 instance-level questions at four abstraction levels.Source: C. User Study · p. 12

B. Dealing With Background

• With salient-object backgrounds, U2-Net masking yields satisfactory sketches; landscapes look pleasing at 32 or 16 strokes but fail at 8, and background performance is reduced.The method is most suitable for images without backgrounds; its zero-shot setup and geometric loss contribute to including background details in sketches.Source: B. Dealing With Background · pp. 11–12, Figure 14 · p. 12

C. User Study

• The study surveyed 121 people: 60 evaluated the method’s sketches, 38 evaluated Kampelmühler and Pinz, and 24 evaluated Li et al..The 60 participants evaluating this method were split across five questionnaire versions; responses contributed equally to overall accuracy, and version difficulty differences were reported as not significant.Source: C. User Study · p. 12

C.1. Category-Level Recognition

• Cats’ triangular ears and giraffes’ long necks may aid category recognition, and the method captures these cues through saliency-based initialization and CLIP-based loss.Source: §C.1 · p. 12
• With six choices including None, 53% of answers for highly abstract 4-stroke sketches were None; this share fell to 7% for 8-stroke sketches and virtually none for 16- and 32-stroke sketches.Source: §C.1 · p. 13

C.1. Category-Level Recognition · C.2. Instance-Level Recognition

• Excluding “None” answers raised category recognition for 4-stroke sketches from 36% to 76%, while instance recognition reached 85% among confident guesses.Source: §C.1 · p. 13, §C.2 · pp. 13–14

C.1. Category-Level Recognition

• Category-level accuracy averaged 36%; cats and giraffes performed above average, while dogs and horses fell below average.Source: §C.1 · p. 12, Figure 15 · p. 13

C.2. Instance-Level Recognition

• Instance-level accuracy rose from 60% at 4 strokes to 97% at 32 strokes, averaging 60% across classes and ranging from 39% for face to 97% for giraffe.Source: Figure 16 · p. 13, §C.2 · pp. 13–14
• Instance recognition depends more on geometric fidelity and matching than on class-level semantics, because the alternatives share the sketch’s class.Source: §C.2 · p. 13
• Face sketches had the lowest 4-stroke instance accuracy at 39%, likely because similar subject poses made detailed facial features more important.Source: §C.2 · p. 13

D.1. Initialization Ablation

• Changing the random seed can produce multiple sketches of the same abstraction level, with examples sorted by final loss.The red dots mark initial stroke locations sampled for three random seeds.Source: §D.1 · p. 14, Figure 21 · p. 15
• With 16 strokes, random initialization often converges to suboptimal sketches that miss semantic components such as eyes; the study compares ViT2 with XDoG, ViT-only, and random initialization.The paper attributes initialization sensitivity to the non-convex optimization objective and convergence to different local minima.Source: §D.1 · pp. 14–15, Figure 22 · p. 15

D.2. CLIP Attention

• CLIP ViT-B/32 relevancy maps average 12 attention heads within each layer, then average attention maps across layers.The analysis visualizes layer attention maps across several image classes.Source: §D.2 · p. 16, Figure 28 · p. 17
• The averaged attention map highlights salient regions, including the heads of horses, cats, and dogs, as well as the body.Source: §D.2 · p. 16

E. Loss Ablation and Analysis

• The loss analysis examines alternative CLIP architectures and intermediate layers, ablates loss components and weights, and compares the chosen loss with L2 and LPIPS.The paper states that its semantic and geometric losses are based on CLIP ResNet101.Source: E. Loss Ablation and Analysis · p. 18
• Because no suitable automatic metric is available, configurations are selected by visual judgment to balance geometric fidelity with semantic features and instance-level recognizability.Too much geometric fidelity can yield simple contours, while focusing only on semantics can produce incoherent depictions or reduce recognizability.Source: E. Loss Ablation and Analysis · p. 18

E.1. CLIP Layers and Architecture Analysis

• ResNet-101 layers 3 and 4 combine globally coherent geometry with recognizable semantics, while its sketches are cleaner and more stable than ResNet-50’s.These layers were selected for the geometric-fidelity loss; layers 1 and 2 produced non-recognizable results, while the fully connected layer lost spatial coherence.Source: Figure 29 · p. 18, §E.1 · p. 19
• Vision transformers encode stronger shape bias, especially in shallower layers, but ViT16 sketches are messier than ViT32, leading the authors to favor ResNet-101.The authors describe ResNet-101 as balancing geometric fidelity and semantics.Source: §E.1 · p. 19, Figure 30 · p. 19, Figure 31 · p. 20

E.2. Loss Term Weights

• Omitting semantic loss preserves geometry but weakens semantic fit, whereas omitting geometric loss loses instance recognition and can make face sketches unrecognizable.Removing geometric-loss layer 3 shifts results toward geometry and away from semantics; removing layer 2 has the opposite effect.Source: §E.2 · p. 20, Figure 32 · p. 20
• A semantic-loss weight of 0.1 balances geometry and semantics, while weights of 0.5 or 1 can cause overly extreme geometric changes.At weight 0.1, cat whiskers are added while general geometry is maintained, and the horse’s head varies slightly while retaining its pose.Source: §E.2 · pp. 20–21, Figure 33 · p. 21

E.3. Comparison to Other Loss Functions

• Compared with L2 and LPIPS, the proposed loss produces sketches with semantic attributes, while L2 lacks semantic information and LPIPS remains too geometric.The comparison also includes an XDoG edge-map baseline; all loss-based results use the same saliency-guided initialization.Source: §E.3 · p. 21, Figure 34 · p. 21

F. Quantitative Comparison

• The quantitative evaluation reports per-class accuracies from CLIP ViT-B/32 and ResNet34 classifiers on SketchyCOCO sketches synthesized by different methods.The accuracy figure distinguishes the two classifiers and the sketch synthesis methods but the supplied passages provide no accuracy values.Source: F. Quantitative Comparison · p. 21, Figure 35 · p. 22
• The evaluation also illustrates ResNet34 and CLIP ViT-B/32 misclassifications for sketches by Kampelmühler and Pinz [21] and Li et al. [23].The figures mark the correct class and each classifier’s prediction on the example sketches.Source: F. Quantitative Comparison · p. 21, Figure 36 · p. 22, Figure 37 · p. 22

G. Additional Qualitative Comparisons

• Unlike text-, image-, or jointly guided CLIPDraw [11] variants, CLIPasso retains input-image geometric grounding and offers appearance control; thin black curves encourage critical semantic and structural details.The comparisons show that those CLIPDraw settings do not produce drawings consistent with the input image’s geometry.Source: G. Additional Qualitative Comparisons · p. 23, Figure 38 · p. 23
• Additional shoe-image comparisons include SketchLattice [32] and Song et al..These comparisons are shown in Figures 39 and 40.Source: G. Additional Qualitative Comparisons · p. 23, Figure 39 · p. 23, Figure 40 · p. 23
• SketchyDatabase comparisons span teapot, horse, duck, and bicycle images against Kampelmühler and Pinz, Li et al. [25], Li et al., and CLIPDraw [11].The classes and images were chosen according to Li et al.’s [25] publicly available results.Source: G. Additional Qualitative Comparisons · p. 23, Figure 41 · p. 24, Figure 42 · p. 25
• On NPR benchmark portrait faces, Mo et al. produce mostly geometric line drawings that do not reflect the input faces’ semantic characteristics.The comparison also includes Li et al. [25] and Li et al..Source: G. Additional Qualitative Comparisons · p. 23, Figure 45 · p. 28

H. Additional Qualitative Results

• Using 4, 3, or 2 control points per stroke produces different styles, with 2-point strokes forming straight lines.The paper notes that curve degree can be changed to achieve different styles and abstraction levels.Source: H. Additional Qualitative Results · p. 28, Figure 46 · p. 29, Figure 47 · p. 30
• The additional examples vary abstraction using sketches with 64, 24, or 16 total control points.These totals are indicated for the examples in Figures 46–48.Source: H. Additional Qualitative Results · p. 28, Figure 46 · p. 29, Figure 47 · p. 30
• Additional examples include NPR benchmark portraits at 64, 32, 16, and 8 strokes, and 100 random cats from SketchyCOCO [13].The portrait sketches use four control points per stroke.Source: H. Additional Qualitative Results · p. 28, Figure 49 · p. 32, Figure 50 · p. 33