Source-linked AI summary
Learning Fashion Compatibility with Bidirectional LSTMs
Xintong Han, Zuxuan Wu, Yu-Gang Jiang, Larry S. Davis
TL;DR
Fashion recommendation requires modeling compatibility across an entire outfit, not just visual similarity. This paper jointly learns a visual-semantic embedding and outfit compatibility with a sequence-based Bi-LSTM; on Polyvore, the authors report effective compatibility learning and clear-margin gains over several alternatives.
Problem
Fashion recommendation requires modeling compatibility across different item categories and outfit-level coherence, beyond simply measuring visual similarity.
Source: §1 · p. 1Method
The approach jointly learns a visual-semantic embedding and item compatibility by treating outfits as sequences and using a Bi-LSTM to predict the next item.
Source: §1 · p. 2Results
Takeaways & Limitations
On Polyvore, the framework is evaluated for fill-in-the-blank recommendation, multimodal outfit generation, and outfit compatibility prediction.
Source: §1 · p. 2Earlier pairwise methods do not capture coherence across all outfit items and can incur high computational cost for large sets.
Source: §1 · p. 2
Abstract
from arXiv · showhide
The ubiquity of online fashion shopping demands effective recommendation services for customers. In this paper, we study two types of fashion recommendation: (i) suggesting an item that matches existing components in a set to form a stylish outfit (a collection of fashion items), and (ii) generating an outfit with multimodal (images/text) specifications from a user. To this end, we propose to jointly learn a visual-semantic embedding and the compatibility relationships among fashion items in an end-to-end fashion. More specifically, we consider a fashion outfit to be a sequence (usually from top to bottom and then accessories) and each item in the outfit as a time step. Given the fashion items in an outfit, we train a bidirectional LSTM (Bi-LSTM) model to sequentially predict the next item conditioned on previous ones to learn their compatibility relationships. Further, we learn a visual-semantic space by regressing image features to their semantic representations aiming to inject attribute and category information as a regularization for training the LSTM. The trained network can not only perform the aforementioned recommendations effectively but also predict the compatibility of a given outfit. We conduct extensive experiments on our newly collected Polyvore dataset, and the results provide strong qualitative and quantitative evidence that our framework outperforms alternative methods.
1 INTRODUCTION
The paper frames fashion recommendation as modeling outfit-wide compatibility to support item recommendation and outfit generation from multimodal inputs. It proposes jointly learning compatibility and a visual-semantic embedding with a Bi-LSTM, and evaluates the model on three tasks.
1 INTRODUCTION
- The work addresses item recommendation for an existing set and outfit generation from user inputs, and evaluates compatibility prediction as a third task.The inputs may combine text and images; evaluation uses fill-in-the-blank recommendation, outfit generation, and compatibility prediction.Source: §1 · pp. 1–2
- Prior recommendation methods either omit outfit composition, support only one recommendation form [8], or lack multimodal inputs, while outfit popularity methods [8] do not generate outfits.Pairwise compatibility methods [15] and estimate relationships between pairs rather than modeling item dependencies in the entire outfit.Source: §1 · pp. 1–2
- A compatible outfit should share visual style while forming a complete, nonredundant ensemble, which motivates modeling dependencies across the whole outfit rather than pairwise compatibility.Pairwise approaches require costly attribute annotation or voting over pairs, which can be computationally expensive and miss coherence among all items.Source: §1 · pp. 1–2
- The end-to-end model uses Inception-V3 image features and a 512-unit Bi-LSTM to predict outfit items sequentially, treating ordered items as time steps.The sequence follows an order such as top to bottom and then accessories; predicting transitions is used as a proxy for learning compatibility.Source: §1 · p. 2
- A jointly learned visual-semantic embedding projects image features into description semantics, adding attribute and category information as LSTM regularization and enabling multimodal outfit generation.The framework jointly learns the embedding and compatibility relationships end-to-end.Source: §1 · p. 2
- Experiments on a newly collected Polyvore dataset show the network outperforms several alternative methods with clear margins.The introduction does not specify the comparison metrics or values.Source: §1 · p. 3
2 RELATED WORK
Prior fashion work addresses item recognition and retrieval, outfit recommendation, and visual compatibility, while this paper uses a Bi-LSTM to model compatibility across outfit sequences. Its approach also learns visual-semantic embeddings to support item and outfit recommendation.
Fashion Recognition and Retrieval
- Fashion recognition and retrieval work parses garments, maps street photos to shop images, or predicts attributes and landmarks, whereas this work targets compatibility among outfit items.The cited examples include segmentation and pose-based parsing [27], street-to-shop mapping, deep image similarity, and FashionNet.Source: §2 · p. 3
Fashion Recommendation
- Earlier recommenders used labeled attributes, user-item interactions, or popularity prediction; the popularity-based RNN approach produced results no better than random [8].This method instead learns item compatibility with a visual-semantic embedding, enabling item and outfit recommendation.Source: §2 · p. 3
Visual Compatibility Learning
- Unlike prior methods measuring compatibility with outfit fashionability, clothing distances, or mid-level elements, this work models dependencies among items to assess an outfit’s overall visual compatibility.An outfit is treated as a stylish composition whose items share similar styles.Source: §2 · p. 3
Sequential Learning with LSTM
- LSTMs use gated memory to model long-range dependencies without vanishing gradients, and this work applies them to outfit items ordered top to bottom, followed by accessories.Each item image in the collection is treated as a time step; LSTMs have also been used for speech [4] and video [2] [18].Source: §2 · p. 3
3 POLYVORE DATASET
The authors collect a Polyvore dataset for fashion research because prior datasets using the site’s multimodal outfit information were not publicly available. Its fixed item ordering supports modeling outfits as sequences, while preprocessing limits item count and vocabulary size.
3 POLYVORE DATASET
- Because prior Polyvore datasets were not publicly available, the authors collected their own dataset for research release.Polyvore outfits include multimodal information such as item images and descriptions, outfit likes, and hashtags.Source: §3 · p. 3
- The dataset contains 21,889 outfits split into 17,316 training, 1,497 validation, and 3,076 test outfits, with no overlapping items between splits.It contains 164,379 items, each pairing a product image with a text description; outfits average 6.5 items.Source: §3 · p. 3
- Outfits retain a fixed top-to-bottom and accessory order, while items beyond the first eight are dropped and descriptions are filtered to a 2,757-word vocabulary.Words occurring fewer than 30 times are removed because descriptions are noisy and lower-ranked words have low visualness.Source: §3 · p. 3
4 APPROACH
The approach combines a bidirectional LSTM that models outfit-item compatibility with a visual-semantic embedding, jointly trained to incorporate semantic information and support multimodal recommendations.
4.1 Fashion Compatibility Learning with Bi-LSTM
- The Bi-LSTM treats an outfit as an ordered image sequence and predicts both next and previous items to model compatibility and overall outfit style.Each item is represented by CNN-derived features; the recurrent memory captures relationships across the sequence.Source: §4.1 · p. 4
4.2 Visual-semantic Embedding
- A contrastive visual-semantic embedding projects images and their weakly labeled text descriptions into a joint space, bringing matched pairs closer than non-matching pairs.Non-matching image-description pairs within each mini-batch provide the comparisons; semantically similar fashion items are thereby close in the embedding space.Source: §4.2 · pp. 4–5
4.3 Joint Modeling
- Joint training combines forward and backward Bi-LSTM objectives with the visual-semantic loss, using end-to-end back-propagation through time to regularize compatibility learning and enable multimodal recommendation.The CNN receives gradients from both sources, averaged during back-propagation.Source: §4.3 · p. 5
5 EXPERIMENT
Experiments evaluate the proposed approach on fill-in-the-blank recommendation, outfit compatibility prediction, and outfit generation. The results and examples demonstrate recommendation and generation from image and text specifications.
5.1 Implementation Details
- Implementation uses 2048D InceptionV3 features projected to 512D, 512 LSTM hidden units, 0.7 dropout, and a 512D visual-semantic space.Training uses margin 0.2, initial learning rate 0.2 halved every two epochs, and batches of 10 outfit sequences; ImageNet-pretrained layers are fine-tuned until validation loss stabilizes.Source: §5.1 · p. 5
5.2 Compared Approaches
- Compared methods include SiameseNet and SetRNN [8], alongside VSE-only, LSTM-only, directional LSTM+VSE ablations, and the full Bi-LSTM+VSE model.The first two are recent approaches; the remaining methods test component contributions, with hyperparameters selected on the validation set.Source: §5.2 · p. 5
5.3 Fill-in-the-blank Fashion Recommendation
- FITB asks users to choose a compatible item from multiple choices to complete a fashion-item sequence, a task not covered by existing datasets described here.The test-set protocol blanks one randomly selected outfit item and offers it alongside three items from other outfits; inference selects the candidate with the highest summed forward and backward prediction probabilities.Source: §5.3 · pp. 5–6
- Bi-LSTM+VSE achieved 68.6% FITB accuracy, 1.9 percentage points above Bi-LSTM alone, and F-LSTM outperformed B-LSTM.The authors report that combining both LSTM directions improves on either direction alone; GRU and LSTM also outperform basic RNN, and LSTM is more suitable for fashion compatibility on this task.Source: Table 1 · p. 6, §5.3 · p. 6
5.4 Fashion Compatibility Prediction
- Our method achieved the best outfit-compatibility AUC among the compared methods, clearly outperforming recent works [8]; one directional LSTM was sufficient.The model uses its Bi-LSTM prediction-loss terms as a compatibility indicator and was evaluated against 4,000 incompatible outfits formed by randomly selecting test-set items.Source: Table 1 · p. 6, §5.4 · p. 7
5.5 Fashion Outfit Generation
- The model generates visually compatible outfits from image queries and can incorporate text queries for semantically relevant recommendations or generate outfits from text alone.For image queries, it generates in both directions and handles multiple items by filling gaps or replacing nearest neighbors; text can update an initial outfit, specify shared attributes or style, or describe items to retrieve.Source: §5.5 · pp. 7–8
6 CONCLUSION
The paper jointly learns fashion-item compatibility with a sequence-based Bi-LSTM and a visual-semantic embedding that supplies category and attribute information. Experiments on fashion recommendation tasks using a newly collected Polyvore dataset show the method can learn outfit compatibility, while user-specific preferences remain future work.
6 CONCLUSION
- The method jointly trains a Bi-LSTM to predict the next item in an outfit sequence and a visual-semantic embedding that provides category and attribute information.Each outfit item is treated as a time step, and prediction is conditioned on previously seen items.Source: §6 · p. 8
- Experiments on different fashion recommendation tasks using the newly collected Polyvore dataset demonstrate that the method can effectively learn outfit compatibility.Source: §6 · p. 8
- Modeling user-specific compatibility and style preferences is identified as a future research direction because compatibility may vary between people.Source: §6 · p. 8