Source-linked AI summary
End-to-End Image-Based Fashion Recommendation
Shereen Elsayed, Lukas Brinkmeyer, Lars Schmidt-Thieme
TL;DR
Fashion recommendation can benefit from item images, while comparisons between image-based and attribute-aware recommenders are often overlooked. This paper adds image features to an attribute-aware model using a calibrated ResNet50 component and reports that it outperforms state-of-the-art image-based models on two real-world datasets.
Problem
Method
The paper adds latent image features extracted by a calibrated ResNet50 component to an attribute-aware recommender and compares three incorporation techniques.
Source: §1 · p. 2Results
ImgRec-EtE was best on the Amazon fashion and men datasets, improving over DVBPR by 2.5% on fashion and 4.8% on men.
Source: §4.4 · p. 8Takeaways & Limitations
The proposed attribute-aware image model outperforms complex state-of-the-art image-based models in experiments on two benchmark datasets.
Source: §1 · p. 2
Abstract
from arXiv · showhide
In fashion-based recommendation settings, incorporating the item image features is considered a crucial factor, and it has shown significant improvements to many traditional models, including but not limited to matrix factorization, auto-encoders, and nearest neighbor models. While there are numerous image-based recommender approaches that utilize dedicated deep neural networks, comparisons to attribute-aware models are often disregarded despite their ability to be easily extended to leverage items' image features. In this paper, we propose a simple yet effective attribute-aware model that incorporates image features for better item representation learning in item recommendation tasks. The proposed model utilizes items' image features extracted by a calibrated ResNet50 component. We present an ablation study to compare incorporating the image features using three different techniques into the recommender system component that can seamlessly leverage any available items' attributes. Experiments on two image-based real-world recommender systems datasets show that the proposed model significantly outperforms all state-of-the-art image-based models.
1 Introduction
The introduction motivates fashion recommendation with item images and proposes a hybrid attribute-aware model that incorporates image features. It reports experiments, an ablation study, and better performance than complex state-of-the-art methods.
1 Introduction
- Fashion recommendation uses item images alongside user-item interaction data, with images described as helping recommend items that look similar or compatible.Source: §1 · p. 2
- The proposed image-aware item-recommendation model uses item image features extracted by a fine-tuned ResNet50 component.The model is presented as simple and attribute-aware.Source: §1 · p. 2
- Experiments on two benchmark datasets report that the proposed model outperforms more complex state-of-the-art methods.Source: §1 · p. 2
- An ablation study compares three approaches for incorporating ResNet50-extracted image features into the recommender model.Source: §1 · p. 2
2 Related work
Prior image-based recommenders use pretrained or jointly trained image networks, while attribute-aware models can incorporate external item information. The paper identifies attribute-aware models as underused in image-based item ranking and proposes using latent image features as item attributes.
2 Related work
- VBPR uses pretrained CNN features, while JRL combines reviews, images, and ratings using CaffeNet and PV-DBOW.Source: §2 · p. 2
- DeepStyle represents items through style and category factors, while SAERS combines global and region-level image features using user-preference attention.Source: §2 · p. 3
- DVBPR jointly trains a BPR recommender with a CNN for pixel-level image features and also uses GANs to generate images from user preferences.Source: §2 · p. 3
- Attribute-aware models incorporate external user and item attributes, and some can be extended to image settings by converting raw image features into latent attributes.Examples include CoNCARS and GraphRec.Source: §2 · p. 3
- Attribute-aware models had been largely set aside for image-based item ranking, motivating a simple image-aware model that uses latent image features as item attributes.The paper reports that this model outperforms complex image-based state-of-the-art models.Source: §2 · p. 3
3 Methodology
The method frames personalized ranking as estimating missing user-item preferences while considering item images. It combines an attribute-aware recommender with a partially fine-tuned ResNet50 image component and uses two-stage training.
3.1 Problem Definition
- The task is to rank personalized item shortlists by estimating missing interaction likelihoods while considering visual information in item images.Source: §3.1 · pp. 3–4
3.2 Proposed model
- The proposed model jointly optimizes an image-feature extractor and an attribute-aware recommender component.Source: §3.2 · p. 4
3.2.1 Recommender System Component
- The recommender embeds user one-hot vectors and item vectors concatenated with external features, then computes their dot product as the user-item score.The image extractor maps each raw item image to a latent feature vector of size F.Source: §3.2.1 · p. 4
3.2.2 Extraction of Image Features
- ImgRec-EtE uses ImageNet-pretrained ResNet50, updates its last 50 layers, fixes its first 126, and adds a jointly trained fully connected layer to compact image features.Source: §3.2.2 · p. 5
3.3 Training Strategy
- Training first fixes pretrained image-network parameters while optimizing the recommender, then jointly fine-tunes the last 50 image-network layers with it.The authors say this saves time and computational power while outperforming use of fixed pretrained parameters alone.Source: §3.3 · p. 5
4 Experiments
The experiments compare ImgRec-EtE with established image-based recommenders and test three ways to add image features. The reported comparisons favor ImgRec-EtE, while the ablation shows that performance varies with how image features are incorporated.
4.1 Datasets
- Amazon Fashion covers six clothing and footwear categories, whereas Amazon Men includes all subcategories.The authors report contacting the dataset authors to verify the Fashion user-item counts in Table 1.Source: §4.1 · p. 6, Table 1 · p. 6
4.4 Comparative study against state-of-the-art image-based models (RQ1)
- ImgRec-EtE performs best on both Amazon Fashion and Amazon Men, improving over DVBPR by 2.5% and 4.8%, respectively, and reaching 0.825 AUC on Fashion, 0.9% above SAERS.Table 2 uses 100 negative samples per user; the SAERS comparison uses 500.Source: Table 2 · p. 7, Table 3 · p. 7, §4.4 · pp. 8–9
4.2 Evaluation Protocol
- Evaluation uses leave-one-out splits and AUC; comparisons use 100 negatives for DVBPR and 500 for SAERS, with means over five trials.The different negative-sample counts accommodate the baselines’ published protocols and unavailable SAERS source code.Source: §4.2 · p. 7
4.3 Baselines
- The comparisons include published image-based models DVBPR and SAERS as well as established item-recommendation baselines.The paper also describes baselines spanning popularity ranking, matrix factorization, visual-content, and attribute-aware approaches.Source: §4.3 · p. 8
4.5 Ablation Study (RQ2)
- ImgRec-Dir reaches AUCs of 0.77 on Fashion and 0.736 on Men; ImgRec-FT improves performance by 3.2% and 1.6%, and ImgRec-EtE adds 1.6% over ImgRec-FT on both datasets.ImgRec-Dir concatenates pre-trained ResNet50 features with item one-hot vectors; ImgRec-FT adds a fine-tuning layer, while ImgRec-EtE jointly trains part of ResNet50 with the recommender.Source: §4.5 · p. 9, Table 4 · p. 9
4.6 Hyperparameters
- ImgRec-Dir and ImgRec-FT use a learning rate of 0.0001 and L2 regularization of 0.1; ImgRec-EtE uses different regularization values across its two phases.ImgRec-EtE uses L2 values of 0.000001 in the fixed-weights phase and 0.00005 in joint training.Source: §4.6 · pp. 9–10
5 Conclusion
The paper proposes an image-based attribute-aware ranking model that jointly trains a ResNet50 component with the recommender. It reports improved performance and superior results to image-based recommendation approaches, and compares three ways of incorporating image features.
5 Conclusion
- ImgRec-EtE jointly trains part of a ResNet50 component with the recommender to incorporate image features into personalized item ranking.Source: §5 · p. 10
- Adding image features significantly improves the model’s performance, and ImgRec-EtE outperforms image-based recommendation approaches.Source: §5 · p. 10
- The ablation compares direct feature concatenation, a fine-tuning fully connected layer, and joint training of part of the image network.Source: §5 · p. 10