Source-linked AI summary
Generalisation and Sharing in Triplet Convnets for Sketch based Visual Search
Tu Bui, Leonardo Ribeiro, Moacir Ponti, John Collomosse
TL;DR
Category-level SBIR seeks photographs matching a sketched object concept, but deep SBIR had chiefly been explored for fine-grained retrieval. This paper evaluates triplet CNNs and training strategies for cross-category sketch–photo embeddings. Its best variant exceeds prior methods on Flickr15k and TU-Berlin benchmarks.
Problem
The paper asks whether triplet CNNs can generalize sketch–photo retrieval across object categories, beyond prior fine-grained SBIR settings.
Source: §1 · p. 1Method
The authors compare triplet CNN designs and training strategies for learning low-dimensional embeddings between sketches and photographs.
Source: §1 · p. 1, §3.1 · p. 3Results
36.2% mAP on Flickr15k and 37.4Tb on TU-Berlin exceed the state of the art by ∼20% mAP and ∼10Tb, respectively.
Source: §5 · p. 8Takeaways & Limitations
Half-sharing performs best for sketch–edgemap matching, while no-sharing is more suitable for sketch–photo learning.
Source: §5 · p. 8Generalization testing is limited to TU-Berlin-Class, the most diverse available dataset at 250 categories, and supervised deep learning requires expensive labeled data.
Source: §5.1 · p. 8
Abstract
from arXiv · showhide
We propose and evaluate several triplet CNN architectures for measuring the similarity between sketches and photographs, within the context of the sketch based image retrieval (SBIR) task. In contrast to recent fine-grained SBIR work, we study the ability of our networks to generalise across diverse object categories from limited training data, and explore in detail strategies for weight sharing, pre-processing, data augmentation and dimensionality reduction. We exceed the performance of pre-existing techniques on both the Flickr15k category level SBIR benchmark by $18\%$, and the TU-Berlin SBIR benchmark by $\sim10 \mathcal{T}_b$, when trained on the 250 category TU-Berlin classification dataset augmented with 25k corresponding photographs harvested from the Internet.
1. Introduction
The paper investigates category-level sketch-based image retrieval, focusing on whether triplet CNNs can learn sketch–photo embeddings that generalize across object categories. It examines data diversity, input preprocessing, network design, and low-dimensional embeddings for retrieval.
1. Introduction
- The paper extends deep-learning SBIR from fine-grained retrieval to category-level search, where sketches represent object kinds rather than specific instances.The target is retrieval across multiple object categories, consistent with sketches of unseen prototypical objects.Source: §1 · p. 1
- With visual concepts numbering about 10^5 and sketch datasets covering about 10^2 categories, the paper asks how well CNNs generalize to unseen sketched categories.It also asks whether category diversity and exemplar volume matter equally.Source: §1 · p. 1
- The study tests whether edge preprocessing helps SBIR embeddings and which losses, weight-sharing strategies, and embedding dimensions best support efficient retrieval.Its cross-domain framing learns a low-dimensional representation shared by sketches and photographs.Source: §1 · p. 1
2. Related Work
Prior SBIR work spans early blob-based and line-art matching, scalable shallow features, and deep networks for retrieval. This paper situates its category-level study among unresolved questions about cross-domain learning and weight sharing.
2. Related Work
- Early SBIR systems matched coarse color, shape, and texture or fitted line-art sketches to photographs, but optimization-based matching scaled at best linearly.Global descriptors later enabled more scalable indexing.Source: §2 · p. 2
2.1. SBIR with shallow features
- Shallow SBIR methods used edge-preprocessed sparse or mid-level features; Flickr15k systems often scored 10–15%, while Perceptual Edge reached 18%.Earlier systems often failed amid edge clutter; Perceptual Edge simplifies edge maps before HoG/BoVW.Source: §2.1 · p. 2
2.2. SBIR with deep networks
- Deep SBIR includes triplet networks for within-category or instance search, but cross-domain retrieval raises questions about architecture, loss, and weight sharing.Prior work reported full sharing for sketch-to-edgemap matching and no sharing for sketch-to-3D retrieval, motivating category-level investigation.Source: §2.2 · p. 2
3. Methodology
The method compares triplet-network designs for learning sketch–photo or sketch–edgemap embeddings. It combines domain-specific architectures, compact representations, staged training, and a modified loss to address convergence.
3. Methodology
- The study compares Siamese, Heterogeneous, and Hybrid triplet networks, varying branch sharing and the placement of fully connected layers and losses.The designs test cross-category generalization and retrieval accuracy.Source: §3 · pp. 2–3
3.1. Network architecture
- For sketch–photo matching, AlexNet processes photos while a SketchANet–AlexNet hybrid sketch branch shares layers 6–7 with photo branches.Those shared layers account for more than 90% of both networks’ parameters; sketch–edgemap designs instead use SketchANet across branches, sharing the top four layers in the Hybrid variant.Source: §3.1 · p. 3, Figure 1 · p. 4
3.2. Dimensionality reduction
- All designs use a 128-dimensional embedding; the classification-compatible FCR layer sits between FC7 and FC8 without an activation.The fixed embedding dimension allows the Flickr15K dataset to be encoded in 7MB.Source: §3.2 · p. 3, Figure 2 · p. 3
3.3. Training procedure
- Training sketch–photo networks proceeds from separate branch classification, through optional shared-layer training, to triplet-loss regression and optional auxiliary-data refinement.The early classification phases support convergence; later phases use regression to produce the SBIR embedding.Source: §3.3 · p. 4
- Random stroke removal augmentation keeps the first stroke group and randomly discards later groups; it improved Flickr15k mAP by about 1%.For sketches with at least 10 strokes, strokes are divided into four drawing-order groups.Source: §3.3 · p. 5
3.4. The gradient vanishing problem
- When triplet features converge too closely, standard loss can flatten at half the margin; the modified loss breaks this balance by scaling sketch features to half the edgemap-feature scale.At query time, sketch features are scaled by 2.0 to match photo-feature scale.Source: §3.4 · p. 5
4. Experiments
Experiments test generalization, weight sharing, and sketch-photo versus sketch-edgemap models using category-level retrieval benchmarks. The final sketch-photo models outperform reported baselines on Flickr15K and TU-Berlin-Retr.
4.1. Datasets
- Training paired 250 TU-Berlin-Class categories with 100 manually selected web photos each; Flickr15K and TU-Berlin-Retr tested retrieval, and Sketchy supplied fine-tuning data.TU-Berlin-Class provides 80 sketches per category; Sketchy has over 500 sketches per category.Source: §4.1 · p. 5
4.2. Experimental settings
- Flickr15K is scored by retrieval mAP and TU-Berlin-Retr by Kendall’s rank correlation; validation mAP queries validation sketches against training photos.Figure 3 contrasts standard triplet loss flattening at half the margin with a modified slope that enables successful half-share training.Source: §4.2 · p. 6, Figure 3 · p. 6
4.3. Generalisation and weight sharing
- Raising training categories from 20 to 250 lifts all models’ Flickr15K mAP from ~9% to 18–22%; with more categories, half-share leads alternatives by ~3% mAP.The Flickr15K comparison discards the four categories shared with the training set.Source: Figure 4 · p. 6, §4.3 · p. 6
- With 250 training categories, increasing sketches per category from 20 to 80 improves all models; half-share gains 2.5%.The number of training categories was held fixed for this comparison.Source: Figure 4 · p. 6, §4.3 · pp. 6–7
4.4. Modality: Sketch-photo vs. Sketch-edgemap
- On Flickr15K, sketch-photo models substantially outperform sketch-edgemap models, and no-share beats half-share by ~1% mAP in the sketch-photo setting.The authors suggest the larger sketch-photo domain gap may make sharing higher-level representations less suitable.Source: §4.4 · p. 7
4.5. Performance over SBIR benchmarks
- On Flickr15K, Tsp reaches 32.87% mAP versus extended GF-HoG’s 18.20%, and Tf surpasses Sgf at every recall; on TU-Berlin-Retr, sketch-photo models significantly beat SHoG.Table 1 covers Flickr15K mAP comparisons; Table 2 reports TU-Berlin-Retr Kendall’s rank correlation coefficient, Tb.Source: Figure 6 · p. 7, Table 1 · p. 7, §4.5 · p. 7
4.4. Modality: Sketch-photo vs. Sketch-edgemap
- Fine-tuning the best sketch-photo no-share model on Sketchy adds 3% mAP on the standard Flickr15K benchmark.This stage uses triplet regression with instance-level matching sketches.Source: §4.4 · p. 7
5. Conclusion
The best-performing triplet-convnet variant exceeded prior state of the art on both category-level SBIR benchmarks. The conclusion also notes limits from available sketch data and the cost of supervised training.
5. Conclusion
5.1. Limitations and Future Work
- Available sketch databases limited generalisation tests to TU-Berlin-Class, the most diverse dataset available at 250 categories.Source: §5.1 · p. 8
- The supervised deep-learning approach requires expensive labelled training data, unlike a free shallow-feature extractor.Source: §5.1 · p. 8