Source-linked AI summary
Learning Type-Aware Embeddings for Fashion Compatibility
Mariya I. Vasileva, Bryan A. Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, David Forsyth
TL;DR
Fashion outfit representations must capture both item similarity and cross-type compatibility without collapsing distinct relationships. This paper learns type-aware embeddings with pair-specific compatibility projections, improving established outfit tasks and supporting geometric queries.
Problem
Type-agnostic embeddings can compress variation and create improper compatibility relationships, limiting diverse fashion queries.
Method
The model jointly learns a general embedding for item descriptions and pair-specific projected spaces for compatibility, with simplified diagonal projection variants.
Results
4% improvement in a fill-in-the-blank outfit completion experiment and 5% improvement in outfit compatibility prediction over the prior state-of-the-art.
Takeaways & Limitations
Type-aware embeddings represent both interchangeable-item similarity and compatibility relationships while supporting geometric queries that previous models could not readily address.
Takeaways & Limitations
The approach assumes a predefined taxonomy of T item types.
Abstract
from arXiv · showhide
Outfits in online fashion data are composed of items of many different types (e.g. top, bottom, shoes) that share some stylistic relationship with one another. A representation for building outfits requires a method that can learn both notions of similarity (for example, when two tops are interchangeable) and compatibility (items of possibly different type that can go together in an outfit). This paper presents an approach to learning an image embedding that respects item type, and jointly learns notions of item similarity and compatibility in an end-to-end model. To evaluate the learned representation, we crawled 68,306 outfits created by users on the Polyvore website. Our approach obtains 3-5% improvement over the state-of-the-art on outfit compatibility prediction and fill-in-the-blank tasks using our dataset, as well as an established smaller dataset, while supporting a variety of useful queries.
1 Introduction
The paper argues that outfit representations must model both item similarity and cross-type compatibility. It proposes type-respecting embeddings that preserve general similarity while using pairwise type-specific spaces for compatibility, improving benchmark performance and enabling richer queries.
- Outfit composition requires representations that capture both interchangeability among similar items and compatibility among potentially different item types.
- Conventional shared embeddings force compatible items across types to be close in one space, compressing useful variation.
- The proposed model learns a shared image embedding for similarity and learned pairwise projections for type-specific compatibility scoring.
- 4% improvement was reported for fill-in-the-blank completion and 5% for outfit compatibility prediction over the prior state of the art.
- The collected Polyvore dataset addresses limitations in existing fashion datasets, including missing compatibility annotations and restricted scale or annotation types.
2 Related Work
Related work uses embedding methods to learn difficult visual relationships, while fashion research also studies search, synthesis, style, attributes, and popularity. These approaches establish relevant foundations but do not cover the paper’s full type-aware compatibility setting.
- Embedding methods learn relationships from positive and negative item pairs using siamese networks or triplet losses.
- Prior fashion work includes product search and matching, garment synthesis, style categorization, latent fashion concepts, and metadata-based style identification.
- Other studies model compatibility through clothing attributes or address attribute recognition, relative attribute strength, and clothing popularity.
3 Data
The paper introduces Polyvore outfit data with richer item information and carefully designed train-test splits. The dataset variants vary in whether garments may recur across splits, affecting evaluation difficulty and scale.
- Polyvore items provide product images, text descriptions, tags, popularity scores, and type information.
- The Maryland Polyvore dataset is relatively small, lacks item types and detailed text descriptions, and contains test-set inconsistencies affecting quantitative evaluation.
- The easier split contains 53,306 training, 10,000 testing, and 5,000 validation outfits, while items may recur across splits.
- The difficult split prevents garments from appearing in multiple splits through graph segmentation, yielding 32,140 outfits and 175,485 items.
- The dataset comparison emphasizes that the proposed variants contain more outfits, detailed descriptions, and fine-grained semantic categories than related datasets.
4 Respecting Type in Embedding
The model learns a general image embedding for similarity and projects it into type-pair-specific spaces for compatibility. Training combines compatibility, similarity, visual-semantic, and regularization objectives with type-dependent projections.
- Embedding formulation: Each item image is mapped by a nonlinear function f(x_i; θ) to a d-dimensional general embedding y_i.
- Triplet construction: Triplets use a compatible cross-type pair as positive evidence and an unseen same-type replacement as the negative item.
- Triplet construction: The standard triplet loss requires compatible pairs to have smaller distance than incompatible pairs by a margin µ.
- Type-specific compatibility: For each type pair, a projection maps items into a type-specific space M(u,v) where compatibility is measured.
- Type-specific compatibility: The paper investigates diagonal learned projections and fixed binary projections as simplified alternatives to full matrix projections.
- Auxiliary objectives: Text descriptions regularize compatibility learning through a text embedding network and visual-semantic constraints.
- Auxiliary objectives: The final loss combines compatibility, similarity, visual-semantic, L2, and L1 terms while preserving dependence on type-specific projections.
5 Experiment Details
The experiments evaluate type-aware embeddings on compatibility and fill-in-the-blank tasks, compare design choices and sampling protocols, and visualize how type-specific spaces organize fashion items. Results show consistent gains from type-specific and progressively enriched embeddings, while dataset splitting and negative sampling materially affect evaluation.
- Evaluation tasks: AUC measures outfit compatibility, while fill-in-the-blank evaluates selecting the most compatible item from four candidates.The experiments use both tasks to assess compatibility prediction and fashion recommendation.
- Compared methods: The evaluated variants include SiameseNet, pairwise type-dependent projections, shared projections, similarity training, and a learned metric.The learned metric generalizes distance by transforming element-wise products of type-specific embeddings through a fully connected layer.
- Maryland Polyvore: 2-3% improvement comes from type-specific embeddings over a single embedding under category-aware negative sampling.Adding visual-semantic and explicit similarity objectives yields small additional gains, while the learned metric produces a 3-4% improvement over type-specific embeddings alone on both tasks.
- Polyvore Outfits: Type-specific embeddings consistently outperform a single embedding on the authors’ Polyvore dataset, with further improvements as the remaining model components are added.Performance is higher on the full dataset than on the disjoint split, which the authors attribute to additional training data for each type-specific embedding.
- Polyvore Outfits: Similar performance across the two dataset splits suggests that unseen outfits matter more than preventing shared items between training and testing.The authors therefore regard the larger dataset version as a fair option for reproducing the experiments.
- Embedding visualization: The global embedding emphasizes color, whereas type-specific spaces can emphasize pair-specific features such as shape.For example, the scarf-jewelry space focuses on pendant shape more than color, allowing local spaces to specialize while the global space represents appearance more evenly.
- Embedding visualization: The type-respecting embedding supports geometric queries that previous models cannot or are not straightforwardly equipped to answer.The paper presents this query capability as a demonstrated advantage of its representation.
6 Conclusion
Respecting item type improves fashion embedding performance and enables representations that capture both interchangeable items and compatible outfit combinations. The learned representation also supports diverse clothing-search and outfit-generation queries.
- Type-aware embeddings perform better on established tasks and improve results on the paper’s larger, richer dataset.
- The representation encodes similarity relationships between interchangeable garments and compatibility relationships between items that form coherent outfits.
- Type-specific spaces specialize in different appearance variations, helping explain improvements on fill-in-the-blank and outfit compatibility tasks.
- The model supports item replacement search, varied outfit completion, and recursive garment swapping to create novel outfits.
A Model Visualization
The model combines type-specific compatibility projections with a generalized distance metric and visual-semantic training from item text descriptions.
- Type-specific projections transform a general image embedding into pairwise compatibility spaces for different item-type combinations.
- A generalized distance metric compares triplets while scoring compatibility conditional on item types.
- Text-description features train a visual-semantic embedding that regularizes the general embedding and learned similarity and compatibility relationships.
B Ablation study
The ablation results compare projection architectures, distance metrics, compatibility-space sharing, and model components, while qualitative examples illustrate learned relationships and diversification.
- Ablation study: A fully connected type-specific projection provides a small performance improvement over a learned diagonal projection, but increases computation cost.
- Ablation study: The learned metric performs slightly better than cosine distance in the reported ablation.
- Ablation study: Table 7 reports additional ablations involving shared compatibility spaces, fully connected projections, cosine distance, and model components.
- Compatibility relationships: Compatibility visualizations compare model-suggested items with randomly sampled items from the queried type.
- Similarity relationships: Similarity visualizations show same-type alternatives grouped around each query item.
- Item replacement: Diversification visualizations replace a heldout outfit item with alternatives equally compatible with the unchanged garments, alongside random same-type alternatives.
F Outfit Generation by Recursive Item Swaps
Recursive outfit generation repeatedly replaces items in a valid human-curated outfit with different same-type items that remain equally compatible with the rest of the outfit.
- Each recursive swap replaces one starting-outfit item with a different item of the same type.
- Replacement items are selected to be equally compatible with the outfit’s remaining items.
- The figure presents a valid outfit in the top row and one replacement step per subsequent row.
- The full recursive item-swap visualization appears in the appendix.
G Interpretability of Learned General Embedding
Figure 10 visualizes the learned general embedding space for the Polyvore dataset using a t-SNE plot.
- Figure 10 uses t-SNE to visualize the learned general embedding space.
- The visualization is based on the paper’s Polyvore dataset.
- The plotted representation is the learned general embedding space rather than a type-specific space.
H Interpretability of Learned Type-Specific Embedding
Figures 11–20 visualize learned type-specific embedding spaces on the Polyvore dataset with t-SNE plots covering multiple item-type pairings.
- Tops and bags: Figure 11 visualizes the type-specific embedding space for tops and bags.
- Each figure uses a t-SNE plot of the learned type-specific embedding space on the Polyvore dataset.
- Cross-type visualizations: Figures 12–14 cover bottoms with outerwear, hats with scarves, and tops with jewelery.
- Cross-type visualizations: Figures 15–18 visualize bags with jewelery, sunglasses with accessories, hats with outerwear, and hats with accessories.
- Shoes-related visualizations: Figures 19–20 visualize bags with shoes and bottoms with shoes.