PaperlayerView exact revision on arXiv

Source-linked AI summary

Visually-Aware Fashion Recommendation and Design with Generative Image Models

Wang-Cheng Kang, Chen Fang, Zhaowen Wang, Julian McAuley

arXiv:1711.02231v1cs.CVcs.AIcs.HCcs.IRcs.MM

TL;DR

Fashion recommendation is difficult because fashion styles are subjective and semantically complex, and existing visual methods commonly use off-the-shelf representations. The paper jointly learns fashion-aware image representations and recommendation, then combines the model with GANs to generate items conditioned on users and categories. It reports state-of-the-art AUC recommendation performance and synthesized images with higher preference scores than existing products.

  • Problem

    Fashion recommendation must model subjective, semantically complex styles, while prior visually-aware methods commonly rely on pre-trained image representations.

  • Method

    The paper jointly learns pixel-level fashion-aware visual representations and personalized ranking, then uses a category-conditioned GAN with user preference optimization to generate clothing images.

  • Results

    The system achieves state-of-the-art recommendation performance in AUC and generates images with higher preference scores than existing corpus items.

  • Takeaways & Limitations

    The framework extends recommendation beyond retrieving existing products toward exploring, modifying, and designing items tailored to individual users.

Abstract

from arXiv · show

Building effective recommender systems for domains like fashion is challenging due to the high level of subjectivity and the semantic complexity of the features involved (i.e., fashion styles). Recent work has shown that approaches to `visual' recommendation (e.g.~clothing, art, etc.) can be made more accurate by incorporating visual signals directly into the recommendation objective, using `off-the-shelf' feature representations derived from deep networks. Here, we seek to extend this contribution by showing that recommendation performance can be significantly improved by learning `fashion aware' image representations directly, i.e., by training the image representation (from the pixel level) and the recommender system jointly; this contribution is related to recent work using Siamese CNNs, though we are able to show improvements over state-of-the-art recommendation techniques such as BPR and variants that make use of pre-trained visual features. Furthermore, we show that our model can be used \emph{generatively}, i.e., given a user and a product category, we can generate new images (i.e., clothing items) that are most consistent with their personal taste. This represents a first step towards building systems that go beyond recommending existing items from a product corpus, but which can be used to suggest styles and aid the design of new products.

I. INTRODUCTION

Fashion recommendation is difficult because user preferences and product styles are complex, changing, and applied to long-tailed catalogs with continual cold-start. The paper jointly learns fashion-aware visual representations and recommendation, then uses the model with GANs to generate user-tailored clothing designs.

  • I. INTRODUCTION: Fashion recommendation faces long-tailed items, continual cold-start, shifting preferences, and highly complex fashion semantics.These factors complicate prediction from historical clicks, purchases, or ratings.
  • I. INTRODUCTION: The system combines personalized retrieval with GAN-based synthesis of new clothing items conditioned on a user’s personal style.Given a user and category, the model can explore potential items beyond the existing product corpus.
  • I. INTRODUCTION: The paper jointly trains pixel-level image representations and a recommender specifically for fashion recommendation.The approach combines a Siamese network with a BPR-based objective rather than relying only on pre-trained visual features.
  • I. INTRODUCTION: The generated clothing images are plausible and diverse, supporting exploratory recommendation and design of new styles.The paper presents this as extending recommendation from selecting existing products toward assisting design.
  • I. INTRODUCTION: The learned system optimizes pairwise preferences between purchased and non-purchased items using item images and latent item properties.This produces personalized preference models and rankings of likely purchases before synthesis is applied.

II. RELATED WORK

Related work spans implicit-feedback ranking, visually-aware recommendation, Siamese image models, and GAN-based generation. The paper differs by learning visual representations end-to-end for personalized ranking and using the resulting system generatively for fashion design.

  • Recommender Systems: BPR optimizes pairwise rankings for implicit feedback and has outperformed competitive baselines on implicit-feedback datasets.Its pairwise assumption requires observed feedback to be more preferable than non-observed feedback.
  • Visually-aware Recommender Systems: Visually-aware recommenders incorporate product-image signals, but closely related BPR extensions use pre-trained visual features.The paper targets richer, task-specific representations learned end-to-end instead.
  • Fashion and Clothing Style: Fashion vision research also studies style categorization and compatibility without necessarily modeling individual users.The paper instead combines fashion-aware visual learning with personalized recommendation.
  • Siamese Networks and Comparative Image Models: Siamese networks train CNNs comparatively by joining two copies through a function that compares their outputs.This architecture supports metric learning and retrieval, including comparative image modeling.
  • Siamese Networks and Comparative Image Models: The closest prior works address global item compatibility or a different comparative-image objective, and neither uses the model for generative image synthesis and design.This distinguishes the paper’s personalized, generative setting from those neighboring approaches.
  • Image Generation and Generative Adversarial Networks: GANs use competing generator and discriminator components to produce realistic-looking images and can be conditioned on additional inputs.The paper conditions generation on product category and uses activation maximization for a selected user’s style.

A. Key Differences

The paper’s key difference is an end-to-end visually-aware recommender that learns task-specific image representations for personalized ranking, combined with GAN-based generation. This extends visual recommendation toward exploring and designing new fashion items.

  • A. Key Differences: The paper’s central distinction is richer learned visual representations that achieve state-of-the-art personalized ranking and enable exploration and design of new items.This generative capability is presented as the most novel aspect of the system.
  • A. Key Differences: The method combines a Siamese CNN-based visual recommender with Bayesian Personalized Ranking and a GAN-based image-generation component.The two components separately support recommendation and synthesis.
  • A. Key Differences: The recommendation task ranks non-interacted items below observed items using implicit feedback rather than explicit ratings.Users, items, item images, and positive-feedback sets define personalized rankings.
  • A. Key Differences: The model uses latent user and item factors to score pairwise preferences between purchased and non-purchased items.These scores produce personalized rankings of likely purchases.
  • A. Key Differences: DVBPR replaces pre-trained visual features and their embedding matrix with a CNN that extracts task-specific features directly from item images.Its learned representation is optimized within the recommendation framework.
  • A. Key Differences: The CNN’s final layer has K dimensions intended to represent variation in users’ fashion preferences rather than general-purpose classification outputs.The architecture uses CNN-F as an efficient training backbone, with K replacing its 1000-dimensional final layer.

B. Learning the Model

The model adapts BPR to jointly learn fashion-aware image representations and personalized rankings from purchased versus non-purchased items, using Siamese CNNs with shared weights.

  • B. Learning the Model: BPR optimizes personalized rankings by comparing each user’s purchased item i against a non-observed item j.The predictor should assign a larger preference score to i than j.
  • B. Learning the Model: The training objective uses the difference between preference scores, the sigmoid function, model parameters, and regularization.The global and user bias terms can be discarded because they cancel between xu,i and xu,j.
  • B. Learning the Model: Sampling many non-observed items approximately optimizes each user’s AUC for ranking observed feedback.Optimization uses stochastic gradient ascent with Adam because all objective components are differentiable.
  • B. Learning the Model: The convolutional network processes two item images in a Siamese setup, with both CNN branches sharing weights.This incorporates image content into pairwise preference learning.

C. From Recommendation to Generation

The system extends personalized recommendation with conditional GANs that synthesize realistic category-specific images, while balancing preference optimization with image realism.

  • C. From Recommendation to Generation: Conditional GANs generate fashion images according to a specified product category, enabling exploration beyond retrieved corpus items.The generator receives random noise and a category, while the discriminator judges whether images are real.
  • C. From Recommendation to Generation: The generator and discriminator are trained alternately with least-squares losses until generated-image quality is acceptable.The discriminator predicts 1 for real images and 0 for fake images, while the generator seeks realistic outputs.
  • C. From Recommendation to Generation: Generated clothing images are distinct from training images yet similar in quality, likely because the domain uses canonical poses and a limited set of conditions.The paper contrasts these outputs with dream-like hallucinations produced for natural-image domains.

D. Personalized Design

Personalized design uses preference maximization to retrieve, synthesize, or modify items for a user and category, trading preference score against image quality and diversity.

  • D. Personalized Design: The system supports sampling new user-tailored items and making small modifications to existing items for personal style.These scenarios are intended to help interpret learned preferences and provide inspiration to designers.
  • D. Personalized Design: Preference maximization seeks images with the highest user-specific preference score for a given product category.The approach parallels activation maximization by finding inputs that maximize a learned model response.
  • D. Personalized Design: The GAN synthesizes images by optimizing a latent generator code while adding a discriminator-based realism term.The hyperparameter η controls the trade-off between preference score and image quality.
  • D. Personalized Design: The latent code is constrained through z = tanh(z′), converting bounded optimization over z into unconstrained optimization over z′.The paper solves the resulting objective with gradient ascent.
  • D. Personalized Design: Multiple generated candidates are ranked by preference value, but sampling is used because direct top-k selection produces poor diversity.The optimization is restarted from 64 random initial points, and the highest-scoring solution is selected.

IV. EXPERIMENTS

The experiments evaluate both ranking and image generation quantitatively, while qualitative studies demonstrate image synthesis and design capabilities.

  • IV. EXPERIMENTS: The evaluation combines quantitative tests of ranking performance and generated images with qualitative demonstrations of image synthesis and design.The paper states that data and code are available on the first author’s webpage.

A. Experimental Setting

The experiments use four implicit-feedback fashion datasets, AUC-based ranking evaluation, and image-generation metrics for preference, quality, and diversity. The study reports both quantitative comparisons and qualitative synthetic-image results, while noting limitations in image-quality and diversity metrics.

  • Datasets: Experiments use Amazon Fashion, Amazon Women, Amazon Men, and Tradesy datasets with implicit feedback.Amazon datasets cover clothing categories, while Tradesy contains clicks, purchases, sales, and other feedback; each item has one image and pre-extracted CNN features.
  • Preprocessing: Inactive users with fewer than 5 observations are discarded, with one action withheld for validation and one for testing.Remaining interactions form the training set, and qualitative evaluation focuses on Amazon Fashion.
  • Recommendation evaluation: AUC measures ranking quality by the fraction of observed items preferred over non-observed items in pairwise comparisons.The evaluation uses test positives paired with items absent from the training, validation, and test interactions.
  • Image-generation evaluation: Generated images are evaluated using preference score, inception score, and Opposite SSIM for user alignment, image quality, and diversity.The compared retrieval and synthesis methods seek high objective values for sampled users and categories.
  • Evaluation caveats: Image-quality and diversity metrics are subjective, so qualitative results are also reported for different image-generation scenarios.The paper evaluates the designed-product direction indirectly rather than directly with real users.

B. Baselines

The evaluation compares standard personalized-ranking methods with visual baselines and the proposed end-to-end learned visual representation. These comparisons isolate the value of personalization, visual signals, and task-specific image learning.

  • Content-unaware baselines: Ranking baselines include RAND, PopRank, WARP, and BPR-MF for implicit-feedback recommendation.RAND has AUC = 0.5 by definition, while BPR-MF uses matrix factorization with Bayesian Personalized Ranking.
  • Visual baselines: Visual baselines include VisRank, Factorization Machines, and VBPR using visual similarity, item-image interactions, or pre-trained CNN features.VBPR is a visually aware personalized-ranking method, while VisRank ranks by similarity to items bought by the user.
  • Proposed method: DVBPR is the proposed method, using learned image representations rather than relying on pre-trained visual features.The comparison is designed to test whether fashion-specific end-to-end representation learning improves recommendation.
  • Comparison goals: The baseline comparisons target personalization, direct incorporation of visual features, and learning representations specifically for fashion recommendation.Implementations use LightFM for WARP and FM, custom implementations for other baselines, and TensorFlow for DVBPR.
  • Training protocol: Hyperparameters for WARP, BPR-MF, FM, and VBPR are tuned by validation grid search, while DVBPR uses one setting across datasets.DVBPR uses minibatches of 128 for recommendation and 64 for GAN training, with K = 50 and λθu = 1.

C. Quantitative Evaluation

DVBPR achieves the strongest recommendation performance across datasets and remains effective for cold-start items. In image generation, personalized synthesis improves preference scores over retrieval while maintaining similar quality and slightly lower diversity.

  • Recommendation performance: Recommendation performance is reported with AUC for all items and cold items observed fewer than 5 times during training.The cold-item setting evaluates relative preference scores for sparsely observed products.
  • Recommendation performance: 5.13% average improvement over the second-best method occurs across all datasets, with 2.73% improvement in cold-start scenarios.The paper also reports that content-aware methods outperform content-unaware matrix-factorization methods on these data.
  • Generated-image evaluation: Generated images are assessed by mean objective value, inception score, and Opposite SSIM for preference matching, quality, and diversity.Table IV compares top-3 returned images for a given user and category over 1000 trials; larger values are better.
  • Generated-image evaluation: Random generation has poor preference scores, while generated images achieve comparable quality and diversity because the GAN does not condition random samples on users.This comparison distinguishes user personalization from the GAN’s ability to produce realistic and diverse images.
  • Generated-image evaluation: 6.8% improvement in preference score is achieved by synthesis over retrieval on Amazon’s comprehensive clothing catalog.Synthesis has similar quality and slightly lower diversity than retrieval, while η controls the preference–quality trade-off and η = 1 is selected.

D. Qualitative Evaluation

The qualitative evaluation shows that the GAN generates plausible, novel fashion images and can tailor them to users or modify prototypes toward their preferences.

  • Image sampling: Generated images are realistic and plausible while differing substantially from dataset images, sharing shape and color profiles but exhibiting different styles.
  • Personalized design: For six users, GAN-generated top-3 images differ from training items yet appear stylistically similar, indicating that individual user styles were captured.
  • Image sampling: Figure 4 compares generated samples from six clothing categories with their nearest dataset neighbors after images are rescaled to square form.
  • Prototype-based tailoring: Prototype-based optimization substantially increases preference scores over the original image while keeping L1-approximated images near the prototype.
  • Prototype-based tailoring: User-specific prototype modifications include color changes, sleeve-length adjustments, distressed pants, shortened pants, and other minor stylistic changes.

V. CONCLUSIONS AND FUTURE WORK

The paper presents an end-to-end visually aware recommender combined with GANs to generate fashion items aligned with individual preferences. It positions the framework as supporting both recommendation and design, while identifying improved image quality and fine-grained style control as future work.

  • The system combines end-to-end Siamese-CNN recommendation with GANs to generate novel fashion items that maximize users’ preference objectives.
  • The framework supports recommending existing items, exploring possible fashion items, modifying prototypes, and generating items tailored to individuals.
  • Future work targets higher-quality generated images, finer-grained style control, applications beyond fashion images, and tighter links between content recommendation and generation.
Loading 1711.02231v1…