Source-linked AI summary
Learning Fashion Compatibility with Bidirectional LSTMs
Xintong Han, Zuxuan Wu, Yu-Gang Jiang, Larry S. Davis
TL;DR
Online fashion recommendation requires modeling compatibility among multiple fashion categories for both item completion and outfit generation. The paper jointly learns a visual-semantic embedding and Bi-LSTM compatibility model, and reports that it outperforms alternative methods across fashion recommendation experiments on a newly collected Polyvore dataset.
Problem
Online fashion recommendation must support item completion and multimodal outfit generation, but compatibility among different fashion categories goes beyond visual similarity.
Method
The framework jointly learns a visual-semantic embedding and Bi-LSTM compatibility relationships by modeling outfits as sequences and predicting each next item.
Results
The method outperforms several alternative methods with clear margins across experiments on a newly collected Polyvore dataset.
Takeaways & Limitations
The trained network supports fill-in-the-blank recommendation, multimodal outfit generation, and compatibility prediction.
Takeaways & Limitations
Prior pairwise approaches do not incorporate coherence among all outfit items, while the paper identifies user-specific compatibility and style preferences as future work.
Abstract
from arXiv · showhide
The ubiquity of online fashion shopping demands effective recommendation services for customers. In this paper, we study two types of fashion recommendation: (i) suggesting an item that matches existing components in a set to form a stylish outfit (a collection of fashion items), and (ii) generating an outfit with multimodal (images/text) specifications from a user. To this end, we propose to jointly learn a visual-semantic embedding and the compatibility relationships among fashion items in an end-to-end fashion. More specifically, we consider a fashion outfit to be a sequence (usually from top to bottom and then accessories) and each item in the outfit as a time step. Given the fashion items in an outfit, we train a bidirectional LSTM (Bi-LSTM) model to sequentially predict the next item conditioned on previous ones to learn their compatibility relationships. Further, we learn a visual-semantic space by regressing image features to their semantic representations aiming to inject attribute and category information as a regularization for training the LSTM. The trained network can not only perform the aforementioned recommendations effectively but also predict the compatibility of a given outfit. We conduct extensive experiments on our newly collected Polyvore dataset, and the results provide strong qualitative and quantitative evidence that our framework outperforms alternative methods.
1 INTRODUCTION
The paper addresses fashion recommendation tasks that require modeling outfit-level compatibility beyond visual similarity. It jointly learns compatibility relationships and visual-semantic representations with a Bi-LSTM framework.
- Motivation: Fashion recommendation includes matching an item to an existing set and generating an outfit from users’ text or image inputs.The paper also evaluates compatibility prediction for a given outfit.
- Motivation: Existing approaches may ignore item composition, support only one recommendation category, or fail to model compatibility across an outfit as a whole.Pairwise methods can incur high computational cost for large sets and fail to capture collection-wide coherence.
- Approach: A Bi-LSTM models an outfit as an ordered sequence and predicts each next item from previously seen items in both directions.The sequence proceeds from top to bottom and then to accessories, with each item treated as a time step.
- Evaluation: The network is evaluated on fill-in-the-blank recommendation, outfit generation, and compatibility prediction using a newly collected Polyvore dataset.The experiments compare the proposed method with alternative methods.
- Approach: The framework jointly learns fashion-item compatibility and a visual-semantic embedding end to end.The embedding incorporates image descriptions, including attribute and category information, as regularization for compatibility learning.
- Evaluation: The experiments report that the network outperforms several alternative methods with clear margins.The paper presents this as evidence for effective fashion recommendation across the studied tasks.
2 RELATED WORK
Related work covers fashion recognition and retrieval, while this paper focuses on visual compatibility relationships among items within an outfit. LSTM sequence modeling provides the paper’s approach to capturing these dependencies.
- Fashion recognition and retrieval: Prior fashion research commonly addresses garment parsing, clothing recognition, and clothing retrieval.These works target identifying or retrieving fashion items rather than modeling outfit compatibility.
- Fashion recommendation: The paper distinguishes its goal of learning visual compatibility relationships among fashion items in an outfit from retrieval-focused methods.The distinction is between item retrieval and compatibility modeling for outfit composition.
- Sequence modeling: LSTM models capture temporal dependencies in sequences through memory cells and functional gates.In this work, clothing items are ordered from top to bottom and then accessories, with images represented as time steps.
3 POLYVORE DATASET
Polyvore provides user-created outfits containing multimodal item information. The paper collected and split its own Polyvore dataset for fashion recommendation experiments.
- Dataset source: Polyvore is a fashion website where users create and upload outfits containing images, descriptions, likes, and hashtags.These multimodal data have been used for several fashion tasks, although prior datasets were not publicly available.
- Dataset construction: The authors collected 21,889 Polyvore outfits and divided them into 17,316 training, 1,497 validation, and 3,076 testing outfits.They used graph segmentation to prevent overlapping items between splits.
- Dataset construction: Outfits with too many fashion items were truncated to their first 8 items, producing a dataset with 164,379 items.Each item contains a product image and corresponding description.
4 APPROACH
The approach models fashion outfits as ordered image sequences and learns compatibility with a bidirectional LSTM, while jointly learning a visual-semantic embedding from image descriptions. This end-to-end framework uses sequential prediction and multimodal semantic information to support compatibility learning and fashion recommendation.
- 4. APPROACH: The framework combines a bidirectional LSTM for fashion compatibility modeling with a visual-semantic embedding for multimodal information.The two components are trained jointly in an end-to-end framework.
- 4.1 Fashion Compatibility Learning with Bi-LSTM: Outfits are represented as ordered sequences, with each fashion image treated as a time step and transitions used to estimate compatibility.The forward LSTM predicts the next item from previous items, while a backward LSTM predicts a previous item from later items.
- 4.1 Fashion Compatibility Learning with Bi-LSTM: The LSTM recursively maps input sequences to hidden states whose gates regulate inputs, forgetting, memory, and outputs.The model uses input, forget, memory-cell, and output activation vectors, with weight matrices and bias terms defining their transformations.
- 4.1 Fashion Compatibility Learning with Bi-LSTM: A softmax layer computes the probability of the next fashion item conditioned on previously seen items.During training, the candidate set contains images from the current batch rather than the full image vocabulary to reduce computational cost.
- 4.2 Visual-semantic Embedding: The visual-semantic module projects image features and word representations into a shared space and uses contrastive loss to separate matching from non-matching pairs.The loss encourages an image and its description to be closer than unmatched descriptions or images by a margin, bringing items with similar semantic attributes and styles closer together.
5 EXPERIMENT
The experiments evaluate the proposed framework on fill-in-the-blank recommendation, compatibility prediction, and multimodal outfit generation using a Polyvore dataset. Results compare the method with alternative approaches and provide qualitative evidence across recommendation and generation settings.
- Experiment settings: The evaluation covers fill-in-the-blank recommendation, outfit compatibility prediction, and multimodal outfit generation on a newly collected Polyvore dataset.These tasks test item completion, whole-outfit compatibility, and generation from image or text specifications.
- Fill-in-the-blank Fashion Recommendation: The fill-in-the-blank dataset masks one item from each test outfit and contrasts the ground-truth item with three items sampled from other outfits.The task requires choosing the compatible item from four candidates.
- Fill-in-the-blank Fashion Recommendation: During fill-in-the-blank inference, forward and backward LSTMs independently score each candidate, and the candidate with the highest total probability is selected.The objective combines predictions conditioned on items before and after the blank.
- Fill-in-the-blank Fashion Recommendation: The method outperforms random-guess-level SetRNN and VSE, SiameseNet, and single-direction LSTM variants on the fill-in-the-blank task.The paper attributes the advantage over pairwise methods to modeling compatibility across the entire outfit; combining both LSTM directions also improves accuracy.
- Fashion Compatibility Prediction: For compatibility prediction, the model uses its bidirectional LSTM prediction loss as an outfit-compatibility indicator and achieves the best performance among the compared methods.The evaluation creates 4,000 incompatible outfits and uses ROC AUC; qualitative examples distinguish stylistically coherent, mismatched, redundant, and incomplete outfits.
- Fashion Outfit Generation: For outfit generation, the model produces visually compatible and complete outfits from one to three image inputs and semantically relevant outfits from multimodal or text-only queries.Text inputs can specify shared attributes or styles, or describe items that the generated outfit should contain.
6 CONCLUSION
The paper jointly trains a Bi-LSTM and visual-semantic embedding to learn fashion compatibility across recommendation tasks. Experiments on a newly collected Polyvore dataset show that the method effectively learns outfit compatibility, while user-specific preferences remain future work.
- The method jointly trains a Bi-LSTM and visual-semantic embedding for fashion compatibility learning.The embedding supplies category and attribute information during Bi-LSTM training.
- Outfits are modeled as sequences, with the Bi-LSTM predicting each next item from previously seen items.
- Experiments on a newly collected Polyvore dataset cover different fashion recommendation tasks and demonstrate effective outfit-compatibility learning.
- Modeling user-specific compatibility and style preferences is identified as a future research direction because fashion compatibility may vary across people.