PaperlayerView exact revision on arXiv

Source-linked AI summary

Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation

Yujie Lin, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Jun Ma, Maarten de Rijke

arXiv:1806.08977v3cs.IR

TL;DR

Outfit recommenders have largely focused on visual features and neglected user comments, despite their potential role in recommendation and explanation. NOR jointly matches outfits and generates abstractive comments, and the paper reports improved recommendation performance and strong comment-generation scores. Its comments are intended to explain recommended outfit pairings.

  • Problem

    Prior outfit recommendation work emphasizes visual features and neglects user comments, which can contribute to explanations and recommendation results.

    Source: §1 · p. 1

  • Method

    NOR jointly trains a CNN-based outfit matcher with mutual attention and a gated recurrent comment generator with cross-modality attention.

    Source: §1 · p. 2

  • Results

    NOR significantly improves over state-of-the-art baselines on MAP, MRR, and AUC, and achieves impressive ROUGE and BLEU scores against human-written comments.

    Source: §8 · p. 13

  • Takeaways & Limitations

    Generated comments can serve as explanations for why a recommended top-bottom outfit is matched.

    Source: §3.1 · p. 4

    Because most dataset comments are positive and many are short, NOR rarely generates negative explanations and tends to produce short comments.

    Source: §8 · p. 13

Abstract

from arXiv · show

Most previous work on outfit recommendation focuses on designing visual features to enhance recommendations. Existing work neglects user comments of fashion items, which have been proved to be effective in generating explanations along with better recommendation results. We propose a novel neural network framework, neural outfit recommendation (NOR), that simultaneously provides outfit recommendations and generates abstractive comments. NOR consists of two parts: outfit matching and comment generation. For outfit matching, we propose a convolutional neural network with a mutual attention mechanism to extract visual features. The visual features are then decoded into a rating score for the matching prediction. For abstractive comment generation, we propose a gated recurrent neural network with a cross-modality attention mechanism to transform visual features into a concise sentence. The two parts are jointly trained based on a multi-task learning framework in an end-to-end back-propagation paradigm. Extensive experiments conducted on an existing dataset and a collected real-world dataset show NOR achieves significant improvements over state-of-the-art baselines for outfit recommendation. Meanwhile, our generated comments achieve impressive ROUGE and BLEU scores in comparison to human-written comments. The generated comments can be regarded as explanations for the recommendation results. We release the dataset and code to facilitate future research.

1 INTRODUCTION

The introduction frames explainable outfit recommendation as recommending matching bottoms for a top while generating a sentence explaining each pairing. NOR jointly learns outfit matching and abstractive comment generation, and experiments report improved recommendation and comment-generation results.

1 INTRODUCTION

  • Fashion-oriented online communities provide outfit compositions and user comments, which offer valuable data for building more accurate and intelligent recommender systems.The introduction contrasts these crowdsourced resources with earlier small expert-annotated datasets.Source: §1 · p. 1
  • The task recommends a short list of bottoms for a given top and generates a sentence explaining why each top-bottom pairing matches.The introduction says explanations can make recommendations more transparent and trustful, helping users make faster and better decisions.Source: §1 · p. 1
  • NOR jointly trains outfit matching and abstractive comment generation, using mutual attention for visual compatibility and cross-modality attention to map visual features into concise text.A CNN encodes top and bottom images, a rating score predicts matching, and a gated RNN generates comments.Source: §1 · p. 2
  • NOR outperforms state-of-the-art models on outfit recommendation by MAP, MRR, and AUC, while generated comments achieve impressive ROUGE and BLEU scores.The introduction also reports evaluation on a collected real-world Polyvore dataset.Source: §1 · p. 2
  • The paper presents user comments as a way to improve outfit recommendation quality while also providing explanations.Source: §1 · p. 2

2 RELATED WORK

Prior outfit recommenders model compatibility, while explainable recommendation and text-generation work address related but distinct tasks. The paper positions NOR as combining outfit matching with readable comments and visual-to-text generation.

2.1 Outfit recommendation

  • Earlier outfit-recommendation work largely omits user comments and does not provide both recommendations and readable comments as NOR does.Prior work includes visual, contextual, and compatibility models, while many early studies relied on small manually annotated datasets.Source: §2.1 · pp. 2–3

2.2 Explainable recommendation

  • Related explainable recommenders use tags, factors, topics, or extracted phrases; this paper instead generates concise sentences based on user comments.The authors argue that simulating users' comments can better express shared experience and feelings, helping users understand and accept recommendations.Source: §2.2 · p. 3
  • Prior joint recommendation-and-text systems address personalized ratings or tips, whereas NOR targets non-personalized outfit matching across image and text modalities.The authors distinguish their cross-modality attention approach from simpler RNN-based mappings.Source: §2.2 · p. 3

2.3 Text generation

  • Earlier comment- and review-generation studies condition text on auxiliary attributes or learned representations, but do not jointly perform recommendation.Source: §2.3 · pp. 3–4

3 NEURAL OUTFIT RECOMMENDATION

NOR jointly recommends compatible outfits and generates comments that explain those recommendations. It encodes visual compatibility for matching and uses cross-modality attention to generate text.

3.1 Overview

  • NOR recommends ranked outfit items and generates a comment for each recommended top-bottom pair as an explanation.The comments describe the complete outfit, whether the recommendation is for a top or a bottom.Source: §3.1 · p. 4

3.2 Top and bottom image encoder

  • A shared two-layer CNN extracts image features, while mutual attention encodes compatibility between each top and bottom in their latent representations.The method also incorporates learned top and bottom latent factors alongside visual features.Source: §3.1 · p. 4, §3.2 · pp. 4–5

3.4 Generation decoder

  • A GRU generation decoder uses cross-modality attention over visual features to produce comments one word at a time.At each step, it uses the previous word embedding, context vector, and hidden state to compute the next hidden state and output.Source: Figure 3 · p. 5, §3.4 · p. 6

3.3 Matching decoder

  • The matching decoder maps the top and bottom latent representations into a shared space and predicts whether the pair matches.The resulting match probability supports recommendations of tops or bottoms.Source: §3.3 · p. 5

3.5 Multi-task learning framework

  • Matching and comment generation are jointly optimized with a multi-task objective that combines their losses with L2 regularization.The framework is trained end to end using back-propagation.Source: §3.5 · p. 6

4 EXPERIMENTAL SETUP

The experiments assess recommendation and comment generation using existing fashion datasets and a newly collected Polyvore dataset. The setup specifies the dataset splits, evaluation candidates, and model-training and decoding choices.

4.1 Datasets

  • FashionVC is used only for recommendation; existing datasets are too small for text generation, and ExpFashion models are not personalized because user information is absent.Since comments are not labeled by item, models generate comments for complete outfits.Source: §4.1 · pp. 6–7
  • ExpFashion contains 200,745 outfits, 29,113 tops, 20,902 bottoms, and 1,052,821 comments, split into validation, test, and training sets.For recommendation, each test top or bottom is evaluated against 100 randomly selected candidates plus its positive matches.Source: §4.1 · pp. 6–7

4.2 Implementation details

  • Training uses randomly sampled negative outfits, excludes comment-generation loss for negatives, and uses beam search with size 3 for comment decoding.The framework uses Adam, a mini-batch size of 64, and gradient clipping in the range [−5, 5].Source: §4.2 · p. 7

5 BOTTOM AND TOP ITEM RECOMMENDATION

NOR is evaluated against recommendation baselines on ExpFashion and FashionVC using MAP, MRR, and AUC. It leads the ExpFashion comparisons across all three metrics; on FashionVC it leads most task-metric comparisons, with bottom-item MRR as an exception.

5.1 Methods used for comparison

  • The comparison includes popularity, collaborative-filtering, and image-content models; content-based models can recommend new items when images are available.POP ranks by pairing popularity, while NRT uses latent factors; the content-based group includes DVBPR, SetRNN, IBR, and BPR-DAE.Source: §5.1 · pp. 7–8

5.2 Evaluation metrics

  • Recommendation quality is measured with MAP, MRR, and AUC, and significance is tested using a paired t-test at p < 0.05.The section defines the metrics for bottom-item recommendation and notes that top-item recommendation is evaluated similarly.Source: §5.2 · p. 8

5.3 Results on the ExpFashion dataset

  • NOR outperforms all baselines on ExpFashion in MAP, MRR, and AUC.The section reports significance testing at p < 0.05.Source: §5.3 · p. 8, Table 2 · p. 8
  • On ExpFashion, all methods recommend tops better than bottoms; the authors link the harder bottom task to fewer positive bottoms per top.The dataset averages about eight positive combinations per top.Source: §5.3 · p. 9

5.4 Results on the FashionVC dataset

  • On FashionVC, NOR-CG leads bottom-item MAP and AUC and top-item MAP, MRR, and AUC; it is slightly below BPR-DAE on bottom-item MRR.Only AUC differences are significant; the authors attribute difficulty detecting other differences to FashionVC's small test size.Source: Table 3 · p. 8, §5.4 · p. 9

6 COMMENT GENERATION

The paper compares NOR's generated comments with adapted extractive and topic-modeling methods and NRT. NOR performs strongly on several ROUGE F-measures and BLEU, while its short-comment setting is a disadvantage on ROUGE-2.

6.1 Methods used for comparison

  • Comment-generation baselines include LexRank, CTR, and RMR extraction procedures, plus NRT; the extraction methods are given an advantage because comments are unavailable in many cases.LexRank extracts a relevant sentence, while CTR and RMR are adapted to extract comments.Source: §6.1 · p. 9

6.2 Evaluation metrics

  • Comment quality is evaluated with ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-SU4 recall, precision, and F-measure, alongside BLEU.ROUGE scores n-gram overlap or longest common subsequence, while ROUGE-SU4 uses skip-bigram and unigram co-occurrence.Source: §6.2 · p. 9, §6.3 · p. 9

6.3 Results

  • NOR performs best on BLEU and ROUGE-1, ROUGE-L, and ROUGE-SU4 F-measures, though LexRank leads ROUGE precision and RMR leads ROUGE recall.The authors describe NOR as having more solid overall performance; the reported comparison is on ExpFashion.Source: §6.3 · pp. 9–10, Table 4 · p. 10
  • NOR performs relatively poorly on ROUGE-2; the dataset's comments average about seven words, beam search favors short outputs, and extraction methods favor longer sentences.The authors state that extraction-based methods' advantage becomes clearer as n increases in ROUGE-N.Source: §6.3 · p. 10

7 ANALYSIS AND CASE STUDY

The analyses test attention mechanisms, joint recommendation and comment learning, and latent factors; case studies illustrate recommendation and generation behavior. Results show trade-offs between recommendation and comment generation, alongside limitations in generated comments.

7.1 Attention mechanism analysis

  • NOR-MA outperforms NOR-NO on both bottom and top recommendation, while NOR-CA outperforms NOR-NO on comment generation.The authors conclude that mutual attention helps recommendation and cross-modality attention helps comment generation.Source: §7.1 · p. 10, Table 5 · p. 10, Table 6 · p. 11
  • NOR-(MA+CA) improves comment generation over NOR-MA but performs worse on recommendation; it beats NOR-NO on both tasks.The authors suggest joint training lets the attention mechanisms influence one another, creating a recommendation–generation trade-off.Source: §7.1 · pp. 10–11, Table 5 · p. 10, Table 6 · p. 11

7.2 Multi-task learning analysis

  • NOR improves over NOR-CG: bottom MAP 1.35%, MRR 1.73%, AUC 2.72%; top MAP 0.89%, MRR 1.66%, AUC 2.02%.The authors attribute the gains to joint learning of shared representations that use comment information for recommendation.Source: Table 7 · p. 11, §7.2 · p. 11

7.3 Latent factors analysis

  • NOR-LF performs poorly without content features, while visual features improve recommendation and comments further improve it.Removing latent factors also lowers MAP, MRR, and AUC; the authors describe these factors as complementary to visual features.Source: §7.3 · p. 11, Table 8 · p. 11

7.4 Recommendation case studies

  • In ExpFashion test examples, most top-10 recommendations appear compatible, though positive items are not always ranked first.Unhighlighted recommendations should not be treated as negative; some higher-ranked alternatives are also judged suitable.Source: §7.4 · pp. 11–12, Figure 5 · p. 12

7.5 Generation case studies

  • Generated comments are generally grammatical and often express opinions or identify outfit features such as color, texture, and style.Examples include comments appreciating combinations and naming color matches or denim material.Source: §7.5 · p. 12, Table 9 · p. 13
  • Bad generations include user-to-user replies and references to items beyond tops and bottoms, as well as duplicate or meaningless text.The authors connect these errors to comments that include communication and other outfit items, and call for better filtering.Source: §7.5 · p. 12

8 CONCLUSIONS AND FUTURE WORK

The conclusion presents NOR as a joint outfit-recommendation and abstractive-comment framework, released with the real-world ExpFashion dataset. Experiments report improvements over state-of-the-art baselines and identify limits in comment polarity and length.

8 CONCLUSIONS AND FUTURE WORK

  • NOR jointly recommends outfits and generates abstractive comments as explanations, and the authors release the ExpFashion dataset.ExpFashion includes images, contextual item metadata, and user comments.Source: §8 · p. 13
  • NOR significantly improves over state-of-the-art baselines on MAP, MRR, and AUC, while generated comments achieve impressive ROUGE and BLEU scores.The authors also report that mutual attention and cross-modality attention are useful for recommendation and comment generation.Source: §8 · p. 13
  • Because most dataset comments are positive and short comments are common, NOR rarely generates negative explanations and tends to produce short comments.Source: §8 · p. 13

Related papers

Loading 1806.08977v3…