PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval

Ayan Kumar Bhunia, Subhadeep Koley, Abdullah Faiz Ur Rahman Khilji, Aneeshan Sain, Pinaki Nath Chowdhury, Tao Xiang, Yi-Zhe Song

arXiv:2203.14817v1cs.CV

TL;DR

Fine-grained sketch retrieval asks users for faithful drawings, yet noisy strokes can reduce retrieval quality. The paper trains a reinforcement-learning stroke selector with pretrained retrieval feedback; its reported comparison finds improved performance over existing state-of-the-art, with applications beyond retrieval.

  • Problem

    Fine-grained SBIR demands faithful sketches, but noisy strokes can impair retrieval and exhaustive subset enumeration is infeasible.

    Source: §1 · p. 1

  • Method

    A reinforcement-learning selector chooses stroke subsets using retrieval rank and triplet-loss feedback from a pretrained FG-SBIR model.

    Source: §4.2 · p. 5

  • Results

    The proposed stroke-subset selector significantly improves performance over existing state-of-the-art FG-SBIR approaches.

    Source: §6.1 · p. 7

  • Takeaways & Limitations

    Once trained, the selector can support other sketch applications, including stroke-importance guidance, interactive retrieval, and subset-based augmentation.

    Source: §1 · p. 2, §7 · p. 8

    Optimal stroke-subset selection is ill-posed without explicit labels, may admit multiple successful subsets, and brute-force annotation is computationally impractical.

    Source: §4.1 · p. 4

Abstract

from arXiv · show

Sketching enables many exciting applications, notably, image retrieval. The fear-to-sketch problem (i.e., "I can't sketch") has however proven to be fatal for its widespread adoption. This paper tackles this "fear" head on, and for the first time, proposes an auxiliary module for existing retrieval models that predominantly lets the users sketch without having to worry. We first conducted a pilot study that revealed the secret lies in the existence of noisy strokes, but not so much of the "I can't sketch". We consequently design a stroke subset selector that {detects noisy strokes, leaving only those} which make a positive contribution towards successful retrieval. Our Reinforcement Learning based formulation quantifies the importance of each stroke present in a given subset, based on the extent to which that stroke contributes to retrieval. When combined with pre-trained retrieval models as a pre-processing module, we achieve a significant gain of 8%-10% over standard baselines and in turn report new state-of-the-art performance. Last but not least, we demonstrate the selector once trained, can also be used in a plug-and-play manner to empower various sketch applications in ways that were not previously possible.

1. Introduction

Fine-grained SBIR can make users feel they cannot sketch because it demands faithful, diligent drawings. The paper argues that noisy strokes, rather than poor sketching ability alone, can damage retrieval, and proposes selecting useful stroke subsets to address this problem.

1. Introduction

  • Fine-grained SBIR has shifted from category-level retrieval toward instance-specific matching that reflects sketch pose and appearance details.The paper positions the fear-to-sketch problem as especially salient in fine-grained retrieval, where users are asked for more faithful drawings.Source: §1 · p. 1
  • Noisy or irrelevant strokes can hurt retrieval, even when users can draw a rough outline, motivating selection of strokes that contribute positively.The paper frames this as a way to address the fear-to-sketch problem in fine-grained SBIR.Source: §1 · p. 1
  • Considering every possible stroke subset is infeasible at O(2N), while random stroke dropping can produce noisy gradients from subsets too coarse to represent the paired photo.The paper instead seeks meaningful subsets that can sustain retrieval.Source: §1 · pp. 1–2
  • The proposed selector uses reinforcement learning to decide which strokes to include, with a pretrained retrieval model providing retrieval-based feedback.It represents sketches as vectors and rasterizes the selected subset before retrieval.Source: §1 · p. 2
  • The trained selector is also presented for stroke-importance guidance, faster interactive retrieval, and sketch-subset augmentation in a plug-and-play manner.The introduction describes these as secondary applications beyond noisy-stroke elimination.Source: §1 · p. 2

2. Related Works

Prior work spans category-level and fine-grained retrieval, partial-sketch modeling, and reinforcement learning for vision. The paper contrasts those lines with its premise that annotated fine-grained sketches may still contain noise.

2. Related Works

  • Category-level SBIR retrieves category-specific photos, while fine-grained SBIR targets instance-specific matches and has developed through graph-matching and deep-learning approaches [4].The cited work includes triplet-ranking networks and later attention, generation, text-tag, and reinforcement-learning variants.Source: §2 · p. 2
  • Earlier FG-SBIR work treated annotated sketch-photo pairs as perfect, whereas this paper assumes free-flow drawing can introduce noise even in carefully drawn sketches [41].The authors argue this concern matters because each fine-grained sketch stroke represents a part of its paired photo.Source: §2 · p. 2
  • Partial-sketch research has modeled completion, recognition, generation, and on-the-fly retrieval, while this paper asks whether a partial sketch is representative enough for faithful retrieval [17] [16].It also seeks the point at which a sequential sketch becomes representative for downstream tasks.Source: §2 · pp. 2–3
  • The paper situates its noisy-stroke removal objective alongside vision applications of reinforcement learning and prior work on learning from noisy labels [23] [52] [45] [18] [49].Unlike those noisy-label settings, it assumes each annotated sketch may fail to perfectly match its paired photo.Source: §2 · p. 3

3. Pilot Study: What’s Wrong with FG-SBIR?

The pilot study finds that retrieval can worsen as a sketch gains strokes, and that the best intermediate sketch can outperform the complete one. Exhaustive subset testing suggests further potential but is impractical for real-time inference.

3. Pilot Study: What’s Wrong with FG-SBIR?

  • The study uses a pretrained three-branch Siamese FG-SBIR baseline and ranks gallery images from rasterized sketches rendered after each new stroke.The baseline uses ImageNet-pretrained VGG-16 branches and triplet-loss training.Source: §3 · p. 3
  • As sketches progress, average ranking percentile improves overall, but individual sketches show sudden performance dips attributed to later irrelevant strokes.The pilot study observes these dips on ShoeV2.Source: §3 · p. 3
  • Best-in-episode retrieval reaches top@1(top@5) 42.54%(73.28%) versus 33.43%(67.81%) for complete sketches, while later strokes hurt 43.44% of instances.These results indicate that the complete sketch is not always the best retrieval query.Source: §3 · p. 4
  • Exhaustive stroke subsets achieve top@1(top@5) accuracy of 66.37%(88.31%), but evaluating every combination at inference is impractical.The study motivates a selector to construct a representative subset without exhaustive real-time search.Source: §3 · p. 4

4. Noisy Stroke Tolerant FG-SBIR

The method places a stroke-subset selector in vector space before a pretrained raster-based FG-SBIR model. It learns stroke selection from retrieval-model feedback, using reinforcement learning rather than explicit optimal-subset labels.

4.1. Stroke Subset Selector

  • Because optimal subsets lack explicit labels and brute-force annotation is impractical, the selector learns select-or-ignore actions using the pretrained FG-SBIR model as a critic.The paper notes that multiple subsets may support successful retrieval.Source: §4.1 · p. 4

4. Noisy Stroke Tolerant FG-SBIR

  • The selector preprocesses vector sketches, then rasterizes the selected strokes for retrieval with a pretrained FG-SBIR model.Figure 2 depicts the selector and retrieval model as a cascade.Source: Figure 2 · p. 4, §4 · p. 4

4.1. Stroke Subset Selector

  • A shared local stroke encoder and global relational network enrich each stroke feature with sketch-wide semantics before selection.Both networks are implemented as one-layer LSTMs with hidden size 128.Source: §4.1 · pp. 4–5

4.2. Training Procedure

  • Because rasterization is non-differentiable, the selector uses policy gradients; its sequential MDP samples subsets over five steps.The pretrained retrieval model supplies training feedback for sampled subsets.Source: §4.2 · p. 5
  • The reward combines paired-photo rank and triplet loss, aiming to improve both retrieval rank and feature-space separation.The reward uses inverse rank and negative triplet loss.Source: §4.2 · p. 5
  • Actor-critic PPO trains the selector, with a learned value function and entropy bonus accompanying the policy objective.The value function shares the actor network's parameters, replacing its final layer with a scalar-value prediction layer.Source: §4.2 · p. 5

5. Applications of Stroke-Subset Selector

The selector is also used to clean noisy training sketches, estimate when an evolving sketch is retrievable, and generate meaningful partial sketches for training.

5. Applications of Stroke-Subset Selector

  • The selector can clean noisy training sketches, which the authors report can boost FG-SBIR performance to some extent.They train the FG-SBIR model and selector in alternating stages, using selector-produced clean sketch labels.Source: §5 · p. 6
  • Its critic estimates retrieval ability during sketching, allowing on-the-fly retrieval to wait until a partial sketch has potential retrieval ability.This can avoid repeated inference on rasterized partial sketches, with less computational burden from the vector-based actor and critic.Source: §5 · p. 6
  • The selector creates retrievable partial sketches for supervised triplet-loss training, unlike random stroke dropping that can remove crucial details.The paper contrasts this with continuous RL training, described as time-intensive and allegedly unstable.Source: §5 · p. 6

6. Experiments

Experiments evaluate the stroke-subset selector on fine-grained sketch retrieval and related partial-sketch settings. The reported results cover standard benchmarks, synthetic noise, and on-the-fly retrieval, alongside analyses of the critic, augmentation, and model design.

6. Experiments

  • Experiments use QMUL-Shoe-V2 and QMUL-Chair-V2 with standard splits, assessing standard retrieval by acc.@1 and acc.@5 and early retrieval by r@A and r@B.The datasets include paired sketch-photo instances and sketch coordinates; the standard splits use 6,051 Shoe-V2 and 1,275 Chair-V2 training sketches.Source: §6 · p. 6

6.1. Performance Analysis

  • Table 1 reports significantly improved standard FG-SBIR performance; the selector improves B-Siamese by approximately 10.31% without complicated joint training.The selector is prepended to the untouched baseline and operates on vector sketches for efficient deployment.Source: Table 1 · p. 7, §6.1 · p. 7

6.2. Further Analysis and Insights

  • The critic’s predicted score correlates with ranking percentile for partial sketches, and top@5 accuracy reaches 80.1% for sketches above its score threshold.The authors also report similar behavior for partial-sketch classification using negative classification loss as reward.Source: §6.2 · p. 7, Figure 3 · p. 7
  • Selector-based stroke sampling improves retrieval training over standard supervised augmentation, especially in the high-data regime.The policy samples multiple augmented versions of a sketch using stroke-wise importance measures.Source: §6.2 · pp. 7–8
  • On ShoeV2, smartly augmented triplet-loss training reaches r@A(r@B) 85.78(21.1), versus 85.38 (21.24) reported for.Using retrieval-ability estimates to reduce rasterized-sketch inference cuts the number of feeds by 42.2%, with r@A(r@B): 85.07 (20.98).Source: §6.2 · p. 8, Figure 4 · p. 8
  • 37.2% top@1 (68.2% top@5), versus 13.4%(44.9%) without selection, under synthetic noisy input.The selector is trained with synthetic noisy patches and applied before retrieval at test time.Source: §6.2 · p. 8, Figure 6 · p. 8
  • The hierarchical LSTM outperforms a one-layer bidirectional LSTM by 4.9% top@1 and 6.7% top@5; Transformer replacement yields no meaningful improvement.The selector adds 22.4% multiply-add operations and 18.3% CPU time; PPO actor-critic beats actor-only by 1.7% top@1, and combined ranking and embedding rewards beat ranking-only by 1.2% top@1 on ShoeV2.Source: §6.2 · p. 8

7. Conclusion

The paper proposes an RL-based stroke subset selector that removes irrelevant strokes to boost FG-SBIR, and shows it can augment other sketch applications in a plug-and-play manner.

  • The selector detects and eliminates irrelevant, noisy strokes to boost the performance of off-the-shelf FG-SBIR frameworks.Source: §7 · p. 8
  • An RL-based framework learns an optimal stroke subset by interacting with a pretrained FG-SBIR model.Source: §7 · p. 8
  • The trained selector can augment other sketch applications in a plug-and-play manner.Source: §7 · p. 8

Supplementary material for Sketching without Worrying: Noise-Tolerant Sketch-Based

The supplementary material lists the paper's authors and their institutional affiliations.

8. Comparative Study with different RL methods

The study compares reinforcement-learning approaches and reports the best results with PPO actor-critic using a clipped surrogate objective.

  • PPO actor-critic with a clipped surrogate objective gives the best results among the compared RL approaches.Source: §8 · p. 11, Table 2 · p. 11
  • The comparison includes Vanilla Policy Gradient, Deep Q-Learning, TRPO, and variants of PPO.Source: §8 · p. 11
  • The critic network provides a byproduct by modelling the retrieval ability of partial sketches.Source: §8 · p. 11

9. Comparative Study with different reward functions

The paper finds that combining ranking and feature-embedding rewards through triplet loss gives the most optimum performance, and examines the selector's use for sketch classification.

9. Comparative Study with different reward functions

  • Combining ranking and feature-embedding-space rewards through triplet loss offers the most optimum performance.Source: §9 · p. 11, Table 3 · p. 11

10. Classification Ability and Data Augmentation for sketch classification

  • On TU-Berlin, critic-predicted scores correlate with classification accuracy for sketches evaluated by a pretrained ResNet50 classifier.Source: §10 · p. 11
  • The policy network can augment sketches for classification, with performance examined across varying training-data regimes on TU-Berlin.Source: §10 · p. 11

11. Motivation on removing “fear” for sketching

The section connects sketching confidence with retrieval from imperfect inputs, then examines critic scores on extreme sketches and alternative models of binary stroke selection.

11. Motivation on removing “fear” for sketching

  • The intended reduction in sketching “fear” is user confidence that retrieval can remain favorable even when a user cannot sketch well.Source: §11 · p. 11
  • Figure 7 plots critic-predicted retrieval ability against ranking percentile for partial sketches and selector-augmented performance across training-data sizes.Source: Figure 7 · p. 12

12. What happens with extreme cases?

  • The critic assigns random junk low retrieval-ability scores, while professional sketches receive higher scores and reach the retrieval threshold earlier.Source: Figure 8 · p. 12, §12 · p. 12

13. Clarity of binary stroke selection scheme

  • Categorical stroke selection converged faster and more stably than Bernoulli selection, with 1.41% higher Acc@1 on ShoeV2.Source: §13 · p. 12

14. Training-time comparison with baselines

The section compares training and evaluation choices, reports a soft-attention baseline against the final method, and identifies cross-dataset generalization as future work.

14. Training-time comparison with baselines

  • Baselines use random stroke dropping with at least 80% of sketch vector length retained; training stops after 20K validation iterations without improvement, with all methods taking 12–14 hours.Source: §14 · p. 12

15. Clarity on Training dataset

  • Training uses 6051+1800 images for retrieval-model pretraining and selector training, while testing uses 679+200 images held out from both stages.Source: §15 · p. 13

16. Comparison with soft-attention

  • Triplet-Attn-HOLEF+Augment achieves Acc@1(Acc@5) of 34.6%(68.9%) on ShoeV2, slightly exceeding Augment but falling significantly behind the final results.Source: §16 · p. 13

17. Limitations

  • Cross-dataset generalization for the stroke-subset selector remains future work, so its transfer across datasets is not established here.Source: §17 · p. 13

Related papers

Loading 2203.14817v1…