Source-linked AI summary
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt
Hangyu Lin, Yanwei Fu, Yu-Gang Jiang, Xiangyang Xue
TL;DR
Sketch research has often used pixel-based models, while vector-sketch work such as SketchRNN primarily targeted generation rather than recognition and retrieval. Sketch-BERT adapts transformer representation learning to sketches and pre-trains with sketch gestalt; the authors report improved performance across downstream sketch tasks.
Problem
Much sketch understanding has treated sketches as pixel images, while SketchRNN's vector representation primarily served generation rather than recognition or retrieval.
Source: §1 · p. 1Method
Sketch-BERT adapts a weight-sharing bidirectional transformer to sketch sequences and uses sketch gestalt pretraining to predict masked positions and states.
Source: §1 · p. 2, §3.2 · p. 3, §3.3 · p. 4Results
The authors report that sketch-gestalt pretraining improves Sketch-BERT performance on downstream sketch recognition, retrieval, and gestalt tasks.
Source: §1 · p. 2, §4.2 · p. 6, §4.3 · p. 6Takeaways & Limitations
Experiments on QuickDraw and TU-Berlin support Sketch-BERT's performance on classification and retrieval, and the paper also reports its ability on generative representation learning.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Previous researches of sketches often considered sketches in pixel format and leveraged CNN based models in the sketch understanding. Fundamentally, a sketch is stored as a sequence of data points, a vector format representation, rather than the photo-realistic image of pixels. SketchRNN studied a generative neural representation for sketches of vector format by Long Short Term Memory networks (LSTM). Unfortunately, the representation learned by SketchRNN is primarily for the generation tasks, rather than the other tasks of recognition and retrieval of sketches. To this end and inspired by the recent BERT model, we present a model of learning Sketch Bidirectional Encoder Representation from Transformer (Sketch-BERT). We generalize BERT to sketch domain, with the novel proposed components and pre-training algorithms, including the newly designed sketch embedding networks, and the self-supervised learning of sketch gestalt. Particularly, towards the pre-training task, we present a novel Sketch Gestalt Model (SGM) to help train the Sketch-BERT. Experimentally, we show that the learned representation of Sketch-BERT can help and improve the performance of the downstream tasks of sketch recognition, sketch retrieval, and sketch gestalt.
1. Introduction
Sketch understanding has largely treated drawings as pixel images, although sketches are sequential point data and SketchRNN primarily learned representations for generation. Sketch-BERT extends transformer-based representation learning to sketches and introduces sketch gestalt as a self-supervised task for recovering masked points.
1. Introduction
- Sketches are sequential data points grouped into strokes, whereas earlier sketch understanding commonly treated them as 2D pixel images.The paper describes each point using two continuous position values and a three-dimensional one-hot state value.Source: §1 · p. 1
- SketchRNN learned vector-sketch representations primarily for generation, while sketch understanding also calls for recognizing semantics and abstracting patterns.The introduction motivates a more general representation that can support recognition and masked-sketch completion.Source: §1 · p. 1
- Sketch gestalt recovers masked sketch points to complete the shape, predicting both continuous positions and discrete point states.The task is presented as a way to learn general sketch patterns.Source: §1 · p. 1, Figure 1 · p. 2
- Sketch-BERT adapts transformer representation learning to sketches with a three-level embedding and self-supervised learning by sketch gestalt.The embedding covers point, positional, and stroke information; sketch gestalt is introduced as a new task.Source: §1 · pp. 1–2
2. Related Works
Prior sketch representation work includes hand-crafted features, pixel-based neural models, and vector-based approaches, with recognition and retrieval as common goals. Related work also covers sketch generation, image inpainting, transformers, and self-supervised prediction of withheld data parts.
2. Related Works
- Sketch representation methods include hand-crafted features, sketch-specific CNNs, auxiliary-task CNNs, and vector-form approaches.Earlier work pursued discriminative features through classification or retrieval.Source: §2 · p. 2
- Prior sketch generation includes image-to-sketch CNN methods and vector-sketch generation with SketchRNN, while this paper introduces sketch gestalt.Unlike image inpainting, sketch gestalt targets abstract shape completion rather than pixels and visual texture consistency.Source: §2 · p. 2
- Transformer models such as BERT use self-supervised prediction of withheld data parts before downstream fine-tuning, motivating a sketch-specific pretraining task.The passage contrasts this with prior self-supervised models designed mostly for images.Source: §2 · p. 3
3. Methodology
Sketch-BERT represents sketches as point sequences, processes them with a weight-sharing transformer, and uses masked-point recovery for self-supervised training. The resulting model is applied to recognition, retrieval, and sketch completion tasks.
3.1. Embedding Sketches
- Sketches are represented as sequences of five-value points and embedded with learned point, positional, and stroke components before refinement for the transformer.The five values encode position offsets and point state; the embedding network refines the combined representation.Source: §3.1 · p. 3
3.2. Weight-sharing Multi-layer Transformer
- The bidirectional transformer shares encoder-layer weights, and self-attention updates sketch features for downstream tasks.The authors state that weight sharing makes Sketch-BERT converge faster.Source: §3.2 · p. 3
3.3. Self-Supervised Learning by Sketch Gestalt
- Sketch Gestalt pre-training reconstructs masked point positions and states, masking 15% of each respectively and accounting for class imbalances in mask sampling.The reconstruction network uses L1 loss for position offsets and cross-entropy loss for state categories.Source: §3.3 · p. 4
3.4. Learning Tasks by Sketch-BERT
- Recognition uses a [CLS] token and classifier, retrieval uses [RET] with triplet and category losses, and sketch gestalt reuses the pretrained SGM.The retrieval objective minimizes distances within a class and maximizes distances across classes.Source: §3.4 · pp. 4–5
4. Experiments and Discussion
Experiments evaluate Sketch-BERT on QuickDraw and TU-Berlin, comparing recognition and retrieval baselines and testing sketch-gestalt completion and pre-training choices.
4.2. Results on Sketch Recognition Task
- Sketch-BERT outperforms pixel- and vector-based classification baselines by about 2% on QuickDraw.Without self-supervised training, QuickDraw top-1 accuracy drops by 5%.Source: Table 1 · p. 5, §4.2 · p. 6
4.3. Results on Sketch Retrieval Task
- Sketch-BERT with self-supervised learning improves retrieval by about 2% over TCNet, while its version without sGesta performs much worse than other baselines.The paper reports retrieval results using Top-1/5 accuracy and mean Average Precision.Source: §4.3 · p. 6, Table 2 · p. 6
4.4. Results on Sketch Gestalt Task
- On masked-sketch completion, Sketch-BERT captures shape and detail more reliably than SketchRNN, especially for complicated sketches such as flashlights and tigers.Figure 3 compares ground truth, a jointly masked input, and completions from SketchRNN and Sketch-BERT across nine QuickDraw classes.Source: §4.4 · p. 6, Figure 3 · p. 7
4.5. Pre-training Task Analysis
- The full sketch-gestalt mask strategy improves performance over Single by >1.7%, Position by >1%, and State by >1.4%; position matters more than state.The comparisons cover classification and retrieval after different pre-training tasks.Source: §4.5 · p. 7, Table 3 · p. 7
- On QuickDraw, pre-training reduces convergence from about 50 to 5 epochs, with 50 iterations per epoch.The paper reports this convergence comparison for Sketch-BERT with and without pre-training.Source: §4.5 · pp. 7–8, Figure 4 · p. 7
- Increasing pre-training samples per class improves top-1 accuracy by 3%, while increasing the number of pre-training categories shows no obvious improvement.The study varies class count and samples per class in its QuickDraw pre-training settings.Source: §4.5 · p. 8, Table 4 · p. 8
- Among tested architectures, 8−12−768 balances complexity and performance; at hidden size 256, increasing depth from 6 to 12 layers improves classification and retrieval by 2%.The 12−16−1024 variant performs slightly worse than other variants and is difficult to converge.Source: §4.5 · p. 8, Table 5 · p. 8
- A Gated Convolution model struggles to reconstruct complicated masked sketches and produces artifacts; image-inpainting methods are not tailored to sketch gestalt.The paper notes that the irregular masks used for image inpainting differ fundamentally from those in sketch gestalt.Source: §4.5 · p. 8, Figure 5 · p. 8
5. Conclusion
The paper presents Sketch-BERT for sketch representation learning using self-supervised sketch gestalt. Results on QuickDraw and TU-Berlin show superiority on classification and retrieval, while sketch gestalt experiments demonstrate generative representation learning ability.
5. Conclusion
- Sketch-BERT uses self-supervised learning by sketch gestalt for sketch representation learning.Source: §5 · p. 8
- Results on QuickDraw and TU-Berlin show Sketch-BERT's superiority on classification and retrieval, and sketch gestalt experiments demonstrate generative representation learning ability.Source: §5 · p. 8
- The authors identify sketch-based image retrieval and sketch generation as possible future extensions.Source: §5 · p. 8