Source-linked AI summary
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval
Aneeshan Sain, Ayan Kumar Bhunia, Yongxin Yang, Tao Xiang, Yi-Zhe Song
TL;DR
Sketch-based image retrieval must handle substantial differences in how users draw the same object, including styles absent from training. StyleMeUp uses a meta-trained cross-modal VAE to disentangle shared semantics from sketch style, and the authors report state-of-the-art performance for category-level and instance-level SBIR.
Problem
Sketches of the same object can vary substantially across users, and existing SBIR models are poorly equipped to generalize to unseen user styles.
Source: §1 · p. 1Method
A cross-modal VAE disentangles shared semantic content from modality-specific style, with meta-learned encoder transformations and a semantic-code regularizer for adaptation.
Source: §1 · p. 2Results
Takeaways & Limitations
The framework is designed to adapt its disentanglement to unseen sketch styles and object categories or instances.
Source: §1 · p. 2
Abstract
from arXiv · showhide
Sketch-based image retrieval (SBIR) is a cross-modal matching problem which is typically solved by learning a joint embedding space where the semantic content shared between photo and sketch modalities are preserved. However, a fundamental challenge in SBIR has been largely ignored so far, that is, sketches are drawn by humans and considerable style variations exist amongst different users. An effective SBIR model needs to explicitly account for this style diversity, crucially, to generalise to unseen user styles. To this end, a novel style-agnostic SBIR model is proposed. Different from existing models, a cross-modal variational autoencoder (VAE) is employed to explicitly disentangle each sketch into a semantic content part shared with the corresponding photo, and a style part unique to the sketcher. Importantly, to make our model dynamically adaptable to any unseen user styles, we propose to meta-train our cross-modal VAE by adding two style-adaptive components: a set of feature transformation layers to its encoder and a regulariser to the disentangled semantic content latent code. With this meta-learning framework, our model can not only disentangle the cross-modal shared semantic content for SBIR, but can adapt the disentanglement to any unseen user style as well, making the SBIR model truly style-agnostic. Extensive experiments show that our style-agnostic model yields state-of-the-art performance for both category-level and instance-level SBIR.
1. Introduction
SBIR models commonly match sketches and photos through shared semantic content, but human sketching styles vary substantially and can undermine generalization to unseen users. StyleMeUp addresses this with a cross-modal disentanglement model whose semantic representation supports retrieval and whose style handling adapts to new users.
1. Introduction
- User style diversity can make sketches of the same object look drastically different, while existing SBIR models are poorly equipped to generalize to unseen user styles.Prior work primarily learns a joint embedding of shared photo-sketch semantics, which can itself vary with users’ choices about what to depict.Source: §1 · p. 1
- The proposed framework explicitly models style diversity and is designed to adapt dynamically to unseen user styles without model retraining.The introduction presents style-agnostic SBIR as a new framework that disentangles modal-invariant semantic content from a modality-specific part.Source: §1 · p. 2
- A cross-modal disentanglement model separates photo or sketch content into shared semantic content for retrieval and a modality-specific part that represents sketcher style.The introduction notes that sketch styles vary more than styles in domains such as art schools or building styles, and may manifest differently across object instances.Source: §1 · p. 2
- A cross-modal translation VAE supports reconstruction and translation, while MAML meta-learns feature-transformation layers and a semantic-code regularizer for adaptation to new styles and categories.The two components are meta-learned for fast adaptation during episodic training.Source: §1 · p. 2
- The authors report state-of-the-art performance from the style-agnostic design across category-level and instance-level SBIR.The contribution summary attributes the result to the proposed framework’s style-agnostic design.Source: §1 · p. 2
2. Related Works
Prior SBIR work developed category-level and instance-level retrieval methods, while related research explored disentangled representations and meta-learning. The paper positions its approach as combining cross-modal disentanglement with adaptation to unseen sketch styles and object instances.
2. Related Works
- Category-level SBIR methods range from handcrafted descriptors to deep joint embeddings, while FG-SBIR targets instance matching with approaches including attention, image generation, textual tags, and reinforcement learning.These models generally preserve shared semantic content but do not explicitly model sketcher style, limiting generalization to unseen objects and styles.Source: §2 · p. 2
- Prior disentanglement research includes autoencoder-adversarial methods, InfoGAN [7], β-VAE, and other task-specific approaches, but none is described as adapting sketch disentanglement to new users and instances.The passage also states that none of these methods had worked toward disentangling sketch features for SBIR.Source: §2 · p. 3
- MAML learns initial parameters that adapt quickly to new tasks through a few gradient updates, and the paper applies meta-learning to cross-modal disentanglement for unseen styles and instances.The related-work discussion distinguishes this goal from prior meta-learning work on domain adaptation or domain generalization.Source: §2 · p. 3
3. Methodology
StyleMeUp uses a cross-modal VAE to separate retrieval-relevant shared semantics from sketch-specific style, then meta-learns components that adapt this disentanglement across tasks and unseen styles.
3. Methodology
- The framework maps sketches and photos to a modal-invariant semantic embedding for cross-modal matching while explicitly modelling sketch-specific style.At inference, the encoder and invariant component produce the style-agnostic embedding.Source: §3 · p. 3
3.1. Disentanglement by Cross-modal Translation
- The VAE combines invariant semantics with a variable latent component, supporting reconstruction, cross-modal translation, and same-object cross-style translation.A triplet loss trains the invariant component to distinguish matching from non-matching photos.Source: §3.1 · p. 4
3.2. Meta-Learning for Adaptive Disentanglement
- Meta-training samples category-based tasks, uses inner-loop updates on training pairs, and evaluates updated parameters on validation pairs in an outer loop.Hard negatives are drawn from other categories.Source: §3.2 · pp. 4–5
- Encoder feature-transformation layers use sampled affine changes after batch normalization and are optimized episodically to reduce sketch-style variance.Outer-loop validation loss updates the feature-transformation hyperparameters.Source: §3.2 · pp. 4–5
- A meta-learned weighted ℓ1 regularizer adaptively controls how much semantic knowledge is retained in the invariant representation across tasks.The regularizer contributes to inner-loop updates, while outer-loop loss updates its parameters for generalization to unseen styles.Source: §3.2 · p. 5
4. Experiments
Experiments evaluate category-level and instance-level retrieval, with evaluation styles held out from training. The proposed method leads the compared methods across both settings and all four datasets, and its ablations examine style-diversity modeling and adaptation components.
4. Experiments
- Evaluation styles are held out from training; category-level splits use 70:10:20 partitions, while FG-SBIR uses separate meta-train, meta-validation, and retrieval sets.The FG-SBIR datasets contain multiple sketches per photo instance, supporting the study of style diversity.Source: §4 · p. 5
4.2. Performance Analysis
- The method outperforms all compared methods on both SBIR settings and all four datasets.Tables 1 and 2 report the FG-SBIR and category-level comparisons, respectively.Source: Table 1 · p. 6, Table 2 · p. 6, §4.2 · p. 7
4.1. Competitors
- Comparisons include state-of-the-art retrieval systems, disentanglement models, and baselines varying in cross-modal learning, multi-sketch input, or meta-learning.The baselines include triplet-loss methods, VAE-based approaches, and a vanilla-MAML Siamese model.Source: §4.1 · p. 6
4.2. Performance Analysis
- On ChairV2 and ShoeV2, the proposed method has lower average retrieval rank and rank variance than B-Basic-SN across sketches of the same photo.Figure 4 presents Ravg as bar height and Vavg as a variance line; lower values are better.Source: Figure 4 · p. 7, §4.2 · p. 7
4.3. Ablation Study
- Without diversity modeling, the FG-SBIR baseline performs much worse; on QuickDraw, the method also surpasses [57] in sketch retrieval and scores 75.81% versus Sketch-A-Net’s 68.71% in classification.The QuickDraw sketch-retrieval mAP (P@200) is 0.748 (0.792), exceeding [57] by 0.096 (0.103).Source: §4.3 · p. 8
- Removing FT layers reduces ShoeV2 performance by 3.19% to 33.28%, while fixed FT parameters also yield inferior results to meta-learning them.The authors report that the FT layers are intended to impart style-agnostic encoding behavior.Source: §4.3 · p. 8
- Removing Regψ lowers ShoeV2 Acc@1 to 32.57%, a 3.9% drop; the study also finds one inner-loop update and d = 64 optimal, with retrieval time near B-Basic-SN.Retrieval takes 0.18/0.42 ms per query on ChairV2/ShoeV2, versus 0.14/0.37 ms for B-Basic-SN.Source: §4.3 · p. 8, Figure 6 · p. 8
5. Conclusion
The conclusion presents a style-agnostic SBIR model that disentangles shared semantic content from modality-specific information and is designed to generalize to unseen sketch styles.
5. Conclusion
- The model addresses the challenge that different people sketch the same object differently, aiming to generalize to unseen sketching styles.Source: §5 · p. 8
- A cross-modal VAE separates photo and sketch representations into modal-invariant and modal-specific parts.Source: §5 · p. 8
- Two new components are meta-learned to make the disentanglement adaptive to unseen sketch styles, and experiments report significant outperformance over existing alternatives.Source: §5 · p. 8
Additional explanations
The additional explanations describe how feature transformations and episodic meta-learning address sketch-style variation, alongside training, evaluation, and feature-fusion details.
Additional explanations
- Feature transformation layers simulate varying intermediate encoder distributions, while episodic meta-learning minimizes style variance to form a style-agnostic embedding.Source: Additional explanations · p. 11
- Table 4 presents Inception-V3 quantitative analysis; rerun DSH and GDH results are described as consistent with the authors’ conclusions on Sketchy and TU-Berlin.Source: Additional explanations · p. 11, Table 4 · p. 11
- Training suppresses LKL’s impact initially and later increases it linearly; meta-learning occurs during training, with no adaptation step at inference.Source: Additional explanations · p. 11
- Element-wise addition performed better than concatenation when fusing modal-invariant and modal-specific features, possibly because it gives them a clearer boundary.Source: Additional explanations · p. 11