Source-linked AI summary
Sketchformer: Transformer-based Representation for Sketched Structure
Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, Moacir Ponti
TL;DR
Sketch representation must capture complex, casually drawn stroke sequences for classification, retrieval, reconstruction, and interpolation. Sketchformer applies Transformer-based continuous and tokenized representations, with dictionary-learning tokenization delivering stronger results than LSTM-based baselines and improved reconstruction and interpolation for complex sketches.
Problem
Representing complex free-hand sketches requires models that capture variable-length stroke sequences and their temporal structure across interpretation and generation tasks.
Method
Sketchformer learns a multi-purpose sketch embedding with Transformer variants that process continuous or dictionary-tokenized stroke-sequence inputs.
Results
Sketchformer improves classification, retrieval, reconstruction, and interpolation, with dictionary-learning tokenization producing a 6% gain over previous LSTM autoencoder models.
Takeaways & Limitations
Transformer-based stroke-sequence embeddings provide a promising basis for sketch understanding, generative modeling, and sketch-based visual search.
Abstract
from arXiv · showhide
Sketchformer is a novel transformer-based representation for encoding free-hand sketches input in a vector form, i.e. as a sequence of strokes. Sketchformer effectively addresses multiple tasks: sketch classification, sketch based image retrieval (SBIR), and the reconstruction and interpolation of sketches. We report several variants exploring continuous and tokenized input representations, and contrast their performance. Our learned embedding, driven by a dictionary learning tokenization scheme, yields state of the art performance in classification and image retrieval tasks, when compared against baseline representations driven by LSTM sequence to sequence architectures: SketchRNN and derivatives. We show that sketch reconstruction and interpolation are improved significantly by the Sketchformer embedding for complex sketches with longer stroke sequences.
1. Introduction
Sketchformer applies Transformer-based sequence modeling to free-hand stroke sequences, targeting sketch interpretation and manipulation. Its tokenized representation improves recognition, generative modeling, and matching across complex sketches.
- Transformer modeling addresses the challenge of representing complex, casually drawn sketches as variable-length stroke sequences rather than rasterized pixels.Stroke sequences preserve temporal order, while Transformer slot masking supports longer-term temporal structure.
- Sketchformer is the first Transformer-based network proposed to learn a deep representation for free-hand sketches.It processes stroke sequences in continuous and tokenized forms.
- The approach targets classification, generative sketch modeling, and sketch-based matching using learned sketch embeddings.The paper reports improvements in object recognition, reconstruction, interpolation, and search-related tasks.
- Sketchformer enhancements in object understanding, generative modeling, and matching support Transformer-based stroke-sequence modeling as a promising direction.
2. Related Work
Prior sketch representation work largely used rasterized inputs or recurrent stroke-sequence models. Sketchformer extends this sequence-modeling line by applying Transformers to learn a multi-purpose sketch embedding.
- Earlier sketch-based image retrieval methods primarily used hand-crafted or CNN-learned features over rasterized sketches.Rasterization discards stroke temporal order and requires models to recover higher-level spatial structure.
- The paper builds on a broader shift from rasterized sketch retrieval toward direct modeling of vector stroke sequences.
- SketchRNN introduced LSTM sequence-to-sequence modeling over vector stroke sequences, providing a major recurrent baseline for sketch representation.
- Sketchformer is presented as the first application of Transformer architecture to sketch modeling and aims to exceed state-of-the-art performance across several representation tasks.
3. Sketch Representation
Sketchformer adapts the Transformer to encode stroke-sequence sketches into compact embeddings for reconstruction, classification, and cross-modal retrieval. It explores continuous and tokenized inputs, modifies the bottleneck with self-attention, and trains embeddings with classification and reconstruction objectives.
- Applications: Sketchformer learns a multi-purpose sketch representation from stroke-sequence input for synthesis, classification, and sketch-based image retrieval.The learned embedding supports reconstruction and interpolation, classification, and cross-modal retrieval through task-specific network components.
- Pre-processing and Tokenization: Sketchformer processes both continuous stroke-5 sequences and tokenized representations, including dictionary and spatial-grid quantization schemes.Dictionary tokenization is compact and translation-invariant, whereas grid tokenization avoids accumulated quantization error but is translation-sensitive and uses a larger vocabulary.
- Transformer Architecture: The model adapts the language-modeling Transformer with self-attention over encoder outputs to produce a compact sketch embedding.This bottleneck self-attention learns timestep importance and aggregates the encoder sequence into embedding z; the decoder then expands z toward the original sequence shape.
- Training Losses: Training combines a classification loss that preserves semantic information with a reconstruction loss that requires the embedding to reproduce the input sequence.For continuous inputs, reconstruction uses an L2 transition loss and a three-way pen-state classification term.
- Cross-modal Search Embedding: For SBIR, an auxiliary four-layer network maps Sketchformer vector embeddings and raster image representations into a common space using triplet training.The vector branch uses Sketchformer, while raster representations come from a pretrained GoogLeNet-based network; shared layers unify the domains.
4. Experiments and Discussion
Experiments evaluate Sketchformer across classification, reconstruction, interpolation, and sketch retrieval. The dictionary-tokenized transformer is strongest for complex reconstruction, interpolation, and sketch-based image retrieval, while retrieval gains vary by modality.
- 4.1. Evaluating Sketch Classification: TForm-Tok-Dict yields the highest classification accuracy among the transformer variants and LSTM baselines on QuickDraw.Table 1 compares three transformer embeddings with SketchRNN and LiveSketch.
- 4.2. Reconstruction and Interpolation: For longer sketches, TForm-Tok-Dict is preferred for reconstruction, whereas LiveSketch performs better on short or medium sequences.The reconstruction study uses five-worker preference judgments across short, medium, and long stroke sequences.
- 4.2. Reconstruction and Interpolation: TForm-Tok-Dict outperforms transformer variants and baselines for both intra-class and inter-class interpolation, with narrower separation for shorter strokes.Interpolations are generated by encoding sketches, applying slerp in the embedding, and decoding the interpolated point.
- 4.3. Cross-modal Matching: R-V retrieval outperforms V-R for both class- and instance-level sketch matching, while transformer gains over RNN baselines are 1–3% in V-R retrieval.Performance is more equivocal for S-S retrieval when the search index contains rasterized sketches.
- 4.3. Cross-modal Matching: TForm-Tok-Dict delivers the best Sketch2Image retrieval performance on Stock10M against vector- and raster-query baselines.The evaluation reports mAP and precision@k using crowdsourced annotations for the top 15 results.
5. Conclusion
Sketchformer uses Transformer-based sketch representations across classification, reconstruction, interpolation, and visual search. Dictionary-learning tokenization improves performance over LSTM autoencoder baselines, especially for complex sketches.
- 5. Conclusion: Sketchformer is a Transformer-based sketch representation whose dictionary-learning tokenization improves performance by 6% over prior LSTM autoencoder models.The conclusion reports plausible interpolation, improved reconstruction for complex sketches, and effectiveness for sketch and image indexing.