Source-linked AI summary
Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences
Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, Serge Belongie
TL;DR
The paper asks how to learn visual clothing compatibility across categories without relying on extensive fine-grained domain knowledge or labels. It combines Siamese CNNs with co-purchase data and heterogeneous-dyad sampling to learn a style space; the resulting framework captures clothing-style semantics and generates compatible outfits.
Problem
Learning outfit compatibility across visually distinct clothing categories requires evidence beyond visual similarity, while fine-grained labels and domain knowledge are difficult to obtain and generalize poorly.
Method
The framework uses Amazon co-purchase relationships, category labels, strategic sampling of cross-category heterogeneous dyads, and Siamese CNNs to learn a latent style space for compatibility.
Results
The learned style space captures semantic information about visual clothing style, generates compatible cross-category outfits, and outperforms the cited baseline feature and training approaches.
Takeaways & Limitations
The approach supports compatibility prediction and outfit retrieval across categories, with style features transferring to unseen clothing categories.
Takeaways & Limitations
The survey indicates that users’ decisions reflect subjective preferences as well as compatibility, motivating studies that separate hedonics from compatibility perception.
Abstract
from arXiv · showhide
With the rapid proliferation of smart mobile devices, users now take millions of photos every day. These include large numbers of clothing and accessory images. We would like to answer questions like `What outfit goes well with this pair of shoes?' To answer these types of questions, one has to go beyond learning visual similarity and learn a visual notion of compatibility across categories. In this paper, we propose a novel learning framework to help answer these types of questions. The main idea of this framework is to learn a feature transformation from images of items into a latent space that expresses compatibility. For the feature transformation, we use a Siamese Convolutional Neural Network (CNN) architecture, where training examples are pairs of items that are either compatible or incompatible. We model compatibility based on co-occurrence in large-scale user behavior data; in particular co-purchase data from Amazon.com. To learn cross-category fit, we introduce a strategic method to sample training data, where pairs of items are heterogeneous dyads, i.e., the two elements of a pair belong to different high-level categories. While this approach is applicable to a wide variety of settings, we focus on the representative problem of learning compatible clothing style. Our results indicate that the proposed framework is capable of learning semantic information about visual style and is able to generate outfits of clothes, with items from different categories, that go well together.
1. Introduction
The paper introduces a framework that learns cross-category clothing compatibility from images, category labels, and co-occurrence data, then retrieves compatible outfits. It addresses limitations of fine-grained, domain-dependent approaches through strategic heterogeneous-pair sampling and Siamese CNN feature learning.
- Motivation: Fine-grained recognition and hand-built matching graphs require substantial domain knowledge, difficult labels, and poor generalization to new subcategories.These challenges are especially pronounced in clothing because fashion collections change each season.
- Approach: The framework maps item images into a latent style space where compatible cross-category items are close and incompatible items are far apart.It targets bundles containing visually distinct categories such as shirts, shoes, and pants.
- Approach: Training uses heterogeneous dyads: frequently co-occurring item pairs whose members belong to different categories.The input combines item images, category labels, and links describing co-occurrences.
- Approach: A Siamese CNN learns the image-to-style transformation, while robust nearest-neighbor retrieval generates compatible outfits despite strong label noise.The framework combines learned style representations with retrieval for outfit construction.
- Evaluation: Amazon co-purchase data provides the compatibility signal, and experiments report that the learned style space captures semantic information about visual clothing style.The evaluation uses co-purchase behavior as a measure of product compatibility.
- Evaluation: The learned transformation quantitatively outperforms vanilla ImageNet features and Siamese CNNs trained without strategic sampling.The paper also compares the framework against baselines in a user study.
2. Related work
Prior work learns visual similarity, predefined style classes, or outfit similarity, whereas this paper learns continuous cross-category compatibility from images without discrete attributes or specified style classes.
- Metric learning and attributes: The paper learns a feature transformation into a similarity metric without relying on discrete, predefined attributes.
- Convolutional neural networks for learning distance metrics and image retrieval: Siamese CNNs directly learn transformations from image space to latent spaces of metric distances for retrieval and correspondence tasks.
- Convolutional neural networks for learning distance metrics and image retrieval: Unlike visual-similarity approaches, the proposed method targets compatibility among bundles containing items from different categories.
- Learning clothing style: Unlike methods requiring manually specified or predefined style classes, this framework learns a continuous high-dimensional style space.
- Learning clothing style: Unlike outfit-similarity methods, it learns compatibility between outfit parts and can build outfits from compatible clothing items.
- Learning clothing style: Compared with prior Amazon-based work, the approach fine-tunes the entire network with Siamese learning and strategic sampling, and evaluates transfer to unseen categories.
3. Dataset
The dataset combines product images, category labels, and Amazon co-purchase relations, using co-purchase as a noisy proxy for compatibility and strategic training examples across categories.
- Training requires positive and negative clothing pairs, with positive pairs constrained to items from different categories.
- The Amazon dataset contains product images, product categories, and co-purchase information; experiments use Clothing, Shoes, and Jewelry subcategories.
- Compatibility is defined by Amazon co-purchase relations such as items frequently bought together or bought by customers with related purchases.
- Amazon relations reflect recommendation-based collaborative filtering rather than direct user-provided compatibility judgments.
- Unlabeled item pairs are not necessarily incompatible because the behavior data is sparse and noisy.
4. Learning the style space
The method learns a cross-category style space by strategically sampling heterogeneous co-occurring pairs for Siamese CNN training, then retrieves compatible neighbors and evaluates transfer to unseen categories.
- Problem formulation: The retrieval goal is to find a compatible item from a different category by mapping images into a style space where compatible items are close.
- Learning the style space: The framework uses co-purchases and heterogeneous dyads to construct training data, then trains a Siamese CNN to transform images into latent style space.
- Training examples: Heterogeneous dyads pair item images whose categories differ, while experimental co-occurrence means co-purchase.
- Outfit generation: The learned style space supports outfit generation by retrieving nearest neighbors to a query item.
- Strategic sampling: The strategic sampler forces positive pairs to be heterogeneous, counteracting CNNs’ tendency to place visually similar same-category items nearby.
- Transfer evaluation: Training and test variants include holdout categories to evaluate whether the learned style transfers to categories absent from training.
- Results: Fine-tuned GoogLeNet separates positive and negative pair distances with a greater margin, indicating learned separation of matching and non-matching clothing.
5. Generating outfits
The framework generates outfits by projecting a query item into style space and retrieving compatible items from selected categories. Clustering and nearest-neighbor sampling help reduce the effect of false labels.
- Authors handpick meaningful category combinations for generated outfits, excluding pairs such as dresses and shirts that overlap in body coverage.
- A query image is projected into style space, then nearest neighbors are retrieved for the remaining outfit categories.For a shirt query, the example searches shoes and pants.
- The method clusters target-category items into 20 centroids and selects the centroid nearest to the query item.
- It samples the five closest items to that centroid and chooses the item nearest to the query for the outfit.
- The procedure robustly samples outfits while ignoring falsely labeled images, and Figure 4 shows generated outfits.
- Fine-tuned GoogLeNet separates matching and non-matching clothing with a greater distance margin.
6. Visualizing the style space
The learned 256-dimensional style space is visualized in two dimensions and is organized largely by clothing categories while also expressing compatibility across categories.
- t-SNE projects the 256-dimensional embedding into two dimensions, which are visualized by selecting images from grid cells.
- Although category labels are not used by the algorithm, the embedding is mostly organized by object categories.
- The embedding also captures compatibility across clothing categories.
- For category pairs, clustering retrieves the closest clusters as likely outfit matches and the most distant clusters as combinations that should not be worn together.
7. Evaluation
The evaluation tests link prediction, transfer to unseen categories, related-work comparisons, and user judgments. Strategic sampling outperforms the baselines on quantitative link prediction, while user-study results vary across conditions.
- Test set prediction accuracy: The first experiment evaluates test-set link prediction against strategic-sampling GoogLeNet and AlexNet, naively sampled GoogLeNet, and vanilla ImageNet GoogLeNet.
- Test set prediction accuracy: Strategic sampling clearly outperforms all baselines on the ROC evaluation, and GoogLeNet is retained for later experiments because it outperforms AlexNet.
- Feature transferability: Feature transferability is tested by holding out shoes, jeans, or shirts during training and evaluating links containing the unseen category.
- Quantitative comparison: Table 1 reports AUC scores across experiments and shows that GoogLeNet with strategic sampling outperforms the other methods.
- Comparison to related work: The related-work comparison is not directly comparable because the tasks, datasets, models, and category composition of test sets differ.
- User study: In the user study, the proposed method significantly outperforms the baseline in studies (a) and (b), but not in (c) and (d).The study presents users with a query item and two cross-category predictions, shown in random order.
- User study: Users may choose functionally suitable, visually similar, or personally preferred items rather than the item that best matches stylistically.
8. Conclusion
The framework learns cross-category clothing compatibility from co-occurrence information and category labels, then generates outfits through nearest-neighbor retrieval. Evaluation reports stronger performance than state-of-the-art approaches, alongside user-study comparisons.
- The framework recovers a clothing style space from co-occurrence information and category labels, learning compatibility between items from different categories.
- The user study compares GoogLeNet strategic sampling with random choice, GoogLeNet naïve sampling, AlexNet strategic sampling, and GoogLeNet Vanilla using a 95% binomial hypothesis test.The dashed line marks cases where differences are not statistically significant when both bars fall below it.
- It combines the learned framework with nearest-neighbor retrieval to generate outfits containing items from different categories that go well together.
- The method outperforms state-of-the-art approaches in predicting whether two items from different categories fit together.