Source-linked AI summary
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval
Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, Zhanyu Ma, Jun Guo
TL;DR
Sketch retrieval research lacked large-scale data, while sketch abstraction and stroke order make fine-grained retrieval challenging. The paper combines CNN and RNN features with a sketch-specific hashing loss, and reports strong retrieval, zero-shot generalization, and recognition performance.
Problem
Sketch research lacked freely available large-scale data, and prior methods had not fully accommodated sketches’ abstraction and dynamic stroke order.
Source: §1 · p. 1Method
The model combines CNN visual features with RNN stroke-order modeling and uses a sketch-specific hashing loss to form compact category clusters.
Source: §1 · p. 2, §3.2 · p. 3Results
The model outperforms shallow and deep alternatives and yields superior zero-shot generalization and sketch-recognition performance.
Source: §5 · p. 8Takeaways & Limitations
The reported results support sketch-specific design for retrieval, zero-shot hashing, and sketch recognition.
Source: §5 · p. 8
Abstract
from arXiv · showhide
We propose a deep hashing framework for sketch retrieval that, for the first time, works on a multi-million scale human sketch dataset. Leveraging on this large dataset, we explore a few sketch-specific traits that were otherwise under-studied in prior literature. Instead of following the conventional sketch recognition task, we introduce the novel problem of sketch hashing retrieval which is not only more challenging, but also offers a better testbed for large-scale sketch analysis, since: (i) more fine-grained sketch feature learning is required to accommodate the large variations in style and abstraction, and (ii) a compact binary code needs to be learned at the same time to enable efficient retrieval. Key to our network design is the embedding of unique characteristics of human sketch, where (i) a two-branch CNN-RNN architecture is adapted to explore the temporal ordering of strokes, and (ii) a novel hashing loss is specifically designed to accommodate both the temporal and abstract traits of sketches. By working with a 3.8M sketch dataset, we show that state-of-the-art hashing models specifically engineered for static images fail to perform well on temporal sketch data. Our network on the other hand not only offers the best retrieval performance on various code sizes, but also yields the best generalization performance under a zero-shot setting and when re-purposed for sketch recognition. Such superior performances effectively demonstrate the benefit of our sketch-specific design.
1. Introduction
The paper introduces sketch hashing retrieval on a multi-million-scale dataset to address the limits of smaller sketch datasets and account for sketches’ abstraction and stroke order. Its sketch-specific model achieves strong retrieval, zero-shot generalization, and sketch-recognition results.
1. Introduction
- Earlier sketch research relied on sub-million datasets, including TU-Berlin (20k) and Sketchy (75k), which had begun to bottleneck progress.The cited passage notes sketch recognition had exceeded human-level performance on TU-Berlin and SBIR was approaching human performance on Sketchy.Source: §1 · p. 1
- Human sketches combine high abstraction with dynamic stroke order, traits that earlier work only partly addressed using static pixelmaps and coarse temporal segmentation.The paper identifies these traits as requiring large, diverse stroke-level data; prior recognition work modeled temporal order by splitting sketches into three pixelmaps.Source: §1 · pp. 1–2
- Sketch hashing retrieval ranks a large gallery for a query sketch, requiring fine-grained features and compact binary codes beyond conventional sketch recognition.The proposed two-branch CNN-RNN models stroke ordering, while a hashing loss encourages compact category clusters in Hamming space.Source: §1 · p. 2
- QuickDraw-3.8M contains 3,829,500 sketches sampled across Google QuickDraw categories and is noisier than TU-Berlin because drawing time was limited to 20 seconds and no post-processing was applied.The paper uses this dataset to study large-scale sketch retrieval.Source: §1 · p. 2
- The proposed network outperforms state-of-the-art retrieval alternatives and also achieves state-of-the-art sketch recognition and zero-shot generalization.The authors attribute its performance to sketch-specific design.Source: §1 · p. 2
2. Related Work
Prior work spans sketch datasets, recognition, and deep hashing, but existing sketch hashing methods did not accommodate stroke order and abstraction at million scale. The paper positions its approach against these limitations.
Sketch Dataset
- Existing medium-scale sketch datasets typically contained hundreds to thousands of examples, too few for large-scale deep hashing; Ha and Eck introduced 50 million sketches across 345 categories, often abstract and noisy.Source: §2 · p. 2
Sketch Recognition
- Sketch recognition progressed from handcrafted features to deep CNNs and later stroke-level temporal augmentation; this work jointly models static visuals and temporal sketching information.The authors report that their approach is superior to existing models when repurposed for sketch recognition.Source: §2 · p. 3
Deep Hashing Learning
- Prior sketch hashing work [16] used cross-view similarity and cross-category semantic loss, but did not accommodate stroke ordering or drawing abstraction and was evaluated on Sketchy.The paper addresses these gaps with a larger human sketch dataset and sketch-specific solutions for million-scale retrieval.Source: §2 · p. 3
3. Methodology
The method maps sketches to binary codes for retrieval by combining raster and stroke-sequence encoders. Its sketch-center training uses fixed class centers estimated from entropy-filtered samples.
3.1. Problem Formulation
- Sketches are mapped to D-bit binary codes while preserving semantic and visual similarity across sample pairs.Each sample includes both a raster sketch and its corresponding segment sequence.Source: §3.1 · p. 3
3.2. Two-branch CNN-RNN Network
- A two-branch encoder combines CNN raster features with RNN stroke-order features through late-fusion concatenation.The fused feature is encoded as a low-dimensional real-valued hashing feature and transformed into a binary code.Source: §3.2 · pp. 3–4
3.3. Sketch Center Loss
- The hashing objective uses softmax cross-entropy and quantization loss, while sketch-center loss addresses visually similar but semantically different categories.Such abstract sketches can make distinct categories, such as dogs and pigs, look similar.Source: §3.3 · p. 4, §3.2 · p. 4
- Sketch-center loss uses fixed class centers computed from noise-removed sketches’ hashing features, providing gradients during final end-to-end fine-tuning.The centers are obtained using a pretrained CNN-RNN model.Source: §3.3 · p. 4
- The method retains each category’s middle 90% of sketches by image entropy and treats the remaining samples as outliers.The paper describes low-entropy sketches as overly abstract and high-entropy sketches as messy or scribble-like.Source: §3.3 · p. 5
- The full objective combines the losses, with λ_scl and λ_ql controlling their relative importance.The detailed training and optimization procedures are summarized in Algorithm 1.Source: §3.3 · p. 5
- Training proceeds by pretraining the CNN and RNN separately, fusing and fine-tuning them, then adding sketch-center loss before full-objective training.The separate branches are first trained with cross-entropy; the fused model is then fine-tuned before class centers are calculated.Source: §3.3 · p. 5
4. Experiments
Experiments evaluate the method on QuickDraw-3.8M across deep and shallow hashing comparisons, component analyses, sketch recognition, and zero-shot retrieval. The model leads deep hashing competitors on retrieval, while its sketch-specific design also shows benefits in recognition and unseen-category retrieval.
4.1. Datasets and Settings
- QuickDraw-3.8M contains 3,829,500 sketches, with 9,000 training and 1,000 validation sketches per category, plus 100 query and 1,000 gallery sketches per category.The dataset has 345 categories; raster sketches are scaled to 224 × 224 × 3, while vector sketches are processed for the RNN.Source: §4.1 · p. 5, Table 3 · p. 6
4.2. Competitors
- Comparisons include deep hashing methods DLBHC, DSH-Supervised, and DSH-Sketch, plus six unsupervised and two supervised shallow hashing methods.Deep competitors are evaluated with the same base network where applicable; shallow methods use 256-dimensional deep features because each task requires about 100–200 GB of memory.Source: §4.2 · p. 6
4.3. Results and Discussions
- At 16 bits, MAP/Precision@200 improve by 6.11/8.36 percent, and at 64 bits by 5.50/4.79 percent over the best deep hashing competitor.The comparison is on the QuickDraw-3.8M retrieval gallery.Source: Table 1 · p. 6, §4.3 · p. 7
- Cross-entropy alone with the two-branch CNN-RNN outperforms the best competitor, sketch center loss and quantization loss further improve performance, and common center loss harms it.Sketch center loss improves category structure across hash lengths, whereas common center loss undermines it.Source: §4.3 · p. 7
- For dog queries, the full model makes fewer semantic mistakes than DLBHC and DSH-Sketch; retrieval remains real-time on the 345,000-sketch gallery.The authors report that its false positives are more understandable, often reflecting ambiguity in the sketch itself.Source: Table 5 · p. 7, §4.3 · p. 8, Figure 6 · p. 8
4.4. Generalization to Sketch Recognition
- In sketch recognition, combining temporal ordering with static pixels yields 79.49% versus 68.71%, and sketch center loss brings additional benefits.The network is adapted with a 2048-dimensional fully connected layer before the 345-way classification layer.Source: §4.4 · p. 8, Table 6 · p. 8
4.5. Generalization to Zero-Shot Sketch Hashing
- On 20 held-out categories, the method reaches MAP 0.7547, versus 0.7094 for DLBHC and 0.5334 for DSH-Sketch, using 32-bit codes.The categories were randomly selected from QuickDraw-3.8M and excluded from training.Source: §4.5 · p. 8
5. Conclusion
The paper studies sketch hashing retrieval on a 3.8M-sketch dataset, using stroke ordering in a CNN-RNN architecture and a hashing loss designed for sketch abstraction. The model outperforms shallow and deep alternatives and shows strong zero-shot and sketch-recognition generalization.
5. Conclusion
- The work studies sketch hashing retrieval using a 3.8M-sketch dataset to examine sketch traits that prior work under-studied.It encodes stroke ordering with a CNN-RNN architecture and introduces a hashing loss designed for sketch abstraction.Source: §5 · p. 8
- The model outperforms shallow and deep alternatives and achieves superior generalization in zero-shot retrieval and when repurposed for sketch recognition.Source: §5 · p. 8