Source-linked AI summary
Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data
Yuncheng Li, LiangLiang Cao, Jiang Zhu, Jiebo Luo
TL;DR
Fashion outfit composition is difficult because it combines subtle visual attributes, contextual information, domain expertise, and creative item matching. The paper trains an end-to-end scorer using multimodal item data and online outfit engagement, then uses it for composition. The system achieves 77% accuracy on a constrained composition task, while the supplied passages do not state the abstract’s 85% scoring AUC.
Problem
Fashion outfit composition requires modeling subtle, subjective fashion concepts and diverse visual and contextual attributes that existing retrieval and recommendation studies do not address.
Method
The paper uses an end-to-end data-driven system that fuses fashion-item images and metadata with information across items, supervised by online outfit engagement.
Results
77% accuracy was achieved for the constrained fashion outfit composition task.
Takeaways & Limitations
Multi-modal inputs and suitable pooling of instance-level features provide the best reported performance for the outfit scoring system.
Abstract
from arXiv · showhide
Composing fashion outfits involves deep understanding of fashion standards while incorporating creativity for choosing multiple fashion items (e.g., Jewelry, Bag, Pants, Dress). In fashion websites, popular or high-quality fashion outfits are usually designed by fashion experts and followed by large audiences. In this paper, we propose a machine learning system to compose fashion outfits automatically. The core of the proposed automatic composition system is to score fashion outfit candidates based on the appearances and meta-data. We propose to leverage outfit popularity on fashion oriented websites to supervise the scoring component. The scoring component is a multi-modal multi-instance deep learning system that evaluates instance aesthetics and set compatibility simultaneously. In order to train and evaluate the proposed composition system, we have collected a large scale fashion outfit dataset with 195K outfits and 368K fashion items from Polyvore. Although the fashion outfit scoring and composition is rather challenging, we have achieved an AUC of 85% for the scoring component, and an accuracy of 77% for a constrained composition task.
I. INTRODUCTION
Fashion outfit composition requires balancing dressing codes, visual complexity, contextual information, and creative contrasts. The paper proposes an end-to-end, data-driven system that learns outfit quality from online engagement and combines image and metadata signals.
- Fashion outfit composition requires both appropriate dressing codes and creativity in balancing color and style contrasts.
- Existing clothes retrieval and recommendation studies do not address automatic fashion outfit composition.The paper attributes this gap to subjective fashion concepts and the difficulty of obtaining exhaustive attribute labels.
- Fashion images contain many visual attributes, while outfits also encode brand, designer, group, cultural, personality, and interest contexts.
- The proposed system uses deep convolutional visual encoding and multi-modal learning over image, title, and category information.The multi-modal approach is reported to outperform a single modality for outfit scoring and composition.
- The paper contributes a data-driven composition approach, an end-to-end system fusing item images, metadata, and across-item information, and a large-scale fashion dataset.
II. RELATED WORK
Prior fashion research primarily addresses image retrieval, attribute learning, and item compatibility, whereas this paper targets automatic outfit composition. The work also builds on set classification and multimodal multimedia understanding.
- Prior fashion studies focus mainly on image retrieval, attribute learning, and compatibility between fashion items.
- This paper differs by modeling complete outfits through both compatibility and aesthetics rather than only retrieving images or matching individual clothing items.
- The proposed techniques belong to set classification, a setting previously applied largely to face-recognition frame sequences and multi-view object recognition.
- The work is motivated by multimedia-understanding methods that combine visual analysis with text and audio modalities.
III. METHODOLOGY
The methodology first defines an outfit-composition algorithm and then develops a multi-modal, multi-instance model for scoring fashion outfits.
- The paper presents fashion outfit composition through an outfit scoring model followed by a multi-modal, multi-instance scoring model.
A. Fashion Outfit Composition
Fashion outfit composition is formulated as iterative item selection guided by a scoring function. The scorer evaluates candidate additions to a seed outfit until a target length is reached.
- Each outfit is a set of fashion items, and each item is associated with multiple modalities such as image, title, and category.
- The composition algorithm takes a scoring model, candidate items, a seed outfit, and a target outfit length as inputs, producing new fashion outfits.
- The scoring function f(S_i; θ) indicates how well an outfit S_i is composed.
- Composition proceeds by iteratively selecting items from a database candidate set to extend a user-selected seed outfit.For example, database skirts can be selected to complete an outfit containing shoes, a hat, and a top.
- The algorithm supports flexible applications, including finding the best item of a specified type for an existing outfit.
B. Fashion Outfit Scoring Model
The scoring model encodes fashion-item images, titles, and categories, fuses these modalities, pools item representations into an outfit vector, and classifies outfit quality end to end.
- Architecture: The model uses feature encoders for each modality, a fusion model for item representations, an outfit pooling model, and a classification model.The paper uses three modalities and connects the components so gradients can be backpropagated through the trainable system.
- Feature Encoding: Item images are encoded with a non-pretrained AlexNet variant, while titles and categories provide complementary context information.The image encoding has dimension 4096; titles are represented with averaged 300-dimensional GloVe vectors, and category embeddings have dimension 256.
- Multi-modality Fusion Model: The fusion model reduces and concatenates modality features, then applies a two-layer MLP with ReLU and dropout to obtain the final item representation.The resulting representation is F_i,j ∈ R^d.
- Multi-instance Pooling Model: Element-wise reductions such as mean, add, max, or product pool variable-length, orderless outfits into holistic embeddings.The paper also adapts recurrent neural networks as an alternative pooling model for variable-length fashion-item sequences.
- Outfit Classification Model: A sigmoid classifier predicts whether an outfit is good, and cross-entropy loss uses user engagement as the ground-truth quality signal.The classifier output is used as the outfit score in the composition algorithm for new fashion outfits.
IV. EXPERIMENTS
The experiments section covers dataset collection, scoring and composition evaluation, and further analysis; Figure 4 describes category frequencies and examples.
- Experimental Scope: The experiments evaluate the scoring model, the composition model, and additional analyses.The section introduction identifies these as the main experimental components.
- Category Analysis: Figure 4 places item categories on the x-axis and reports category examples, actual dataset frequencies, and percentage frequencies in overlaid tables.The right-hand table's third column lists percentage frequency.
A. Dataset
The dataset is collected from Polyvore, where users create and engage with curated fashion outfits, then filtered and split to support evaluation.
- Data Collection: Polyvore provides tools and interfaces for users to create fashion outfits that are viewed, favored, and recreated by visitors.Some user-curated outfits attract high volumes of attention, providing engagement information for the dataset.
- Data Filtering: The collection removes sparse categories, repeatedly used items, and outfits outside selected like-count percentile ranges.Categories with fewer than 500 items and items appearing in more than 5 outfits are removed; outfits between the 1st–40th and 90th–99th like-count percentiles are retained.
- Dataset Splits: Training, development, and testing splits are constructed without shared items by segmenting a graph of outfits connected through common items.This construction ensures testing items are never seen during training.
B. Evaluate Fashion Outfit Quality Scorer
The fashion outfit quality scorer is evaluated as a classification model against existing methods and across modality, pooling, fusion, and embedding choices. Results favor direct outfit-quality classification, multimodal inputs, mean pooling, and an embedding dimension of d = 8.
- Evaluation protocol: AUC and Average Precision evaluate the scorer on a held-out testing split, with development data selecting hyperparameters.The train, development, and testing splits respectively fit parameters, tune choices such as d, and provide evaluation ground truth.
- Comparison with existing methods: The proposed framework outperforms LEML and a Siamese network when comparing image-only outfit scoring models.The comparison attributes the advantage over LEML to end-to-end representation learning and over Siamese to direct classification aligned with outfit quality.
- Modality combinations: Fashion item title performs best alone, while image and category provide complementary information; category alone performs poorly.The paper links weak category performance partly to high-level labels and weak image-only performance partly to limited training data for large ConvNets.
- Pooling models: Mean reduction achieves the best result among mean, max, and RNN pooling models.RNN pooling performs poorly partly because it imposes an order on fashion items even though outfits are orderless.
- Embedding dimension: Performance peaks at embedding dimension d = 8, while smaller or larger embeddings hurt performance.The embedding maps fashion items into a fixed-length vector before pooling and classification.
C. Evaluate Outfit Composition Algorithm
The constrained composition evaluation tests whether a scorer can select a compatible item for a three-item seed outfit from five candidates. Expert labels provide a stricter reference because randomly sampled candidates may also be compatible.
- Constrained Composition Setup: The constrained setup fixes three-item seed outfits, five candidate items, and a final outfit length of four.It models selecting the best additional item for an existing three-item outfit.
- Evaluation Dataset: The automatic evaluation treats the original item as positive and four randomly sampled database items as negatives.Randomly sampled items can nevertheless fit the seed outfit, motivating a separate expert-labeled evaluation set.
- Expert Evaluation: 93.7% of original target items were labeled positive by experts, supporting user engagement as an outfit-quality indicator.Experts labeled an average of two positive items per evaluation set, showing that multiple candidates can be acceptable.
- Composition Results: 77% accuracy was achieved by the best composition model in simulating expert results.The authors caution that the constrained process and limited evaluation sample make the human–machine comparison restricted.
- Composition Results: Combining images and meta-data produced the best performance on both evaluation sets, while Qauto performance correlated well with Qgold.Qgold uses fashion-expert labels, whereas Qauto is built from the original item and randomly sampled alternatives.
D. Model Analysis
The analysis examines item importance, modality-specific cues, and scoring errors. It shows how replacing low-importance items can guide outfit curation while revealing different roles for image, title, category, and combined models.
- Fashion Item Importance: Larger score decrements after replacing an item indicate higher predicted item importance.Algorithm 2 estimates each item’s contribution by replacing it with an arbitrary database item and measuring the resulting quality change.
- Fashion Item Importance: Replacing the least important item is reported as safe for all three analyzed outfits and may improve outfit quality.One example identifies an overlapping item as least important.
- Modality Analysis: The image model tends to identify color tone and shape, while title and category models capture item semantics.The white dress example is judged mismatched in color by the image model; title and category errors are semantically consistent.
- Modality Analysis: The full model’s item rankings cannot be explained by combining single-modality rankings, indicating nonlinear and complex modality interaction.The analysis also notes that the full model and image model often agree on importance ordering.
- Error Analysis: The full modality configuration produces the fewest errors among the top 1000 predictions.The reported modality-specific error rates are 0.353 for full, 0.415 for image, 0.371 for title, and 0.605 for categories.
E. Implementation Details
The framework is trained with Adam and a fixed pretrained GloVe title encoder.
- Optimization and Encoders: Training uses Adam with batch size 50, learning rate 0.01, 40K iterations, and learning-rate halving every 15K iterations.The title encoder uses fixed GloVe embeddings pretrained on a 42-billion-token webpage corpus with 1.9 million unique words.
V. CONCLUSIONS
The paper frames fashion outfit composition as a challenging problem and proposes an end-to-end quality-scoring approach. Combining multiple modalities with appropriate pooling of instance-level features gives the best performance, while broader data and improved evaluation remain future goals.
- The paper addresses fashion outfit composition, including the challenge of matching domain-expert knowledge and modeling fashion diversity.
- The proposed generic composition algorithm is based on an end-to-end trainable outfit quality scorer.
- Combining multi-modalities with proper pooling of instance-level features leads to the best performance.
- Future work includes collecting more data and finding better ways to evaluate the composition algorithm.