Source-linked AI summary
Image-based Recommendations on Styles and Substitutes
Julian McAuley, Christopher Targett, Qinfeng Shi, Anton van den Hengel
TL;DR
The paper asks how to model human judgments about visual relationships between objects, including complements and substitutes, without relying on fine-grained annotations. It uses large-scale indirectly related image data and a scalable relational model, showing that such relationships can be modeled beyond simple visual similarity. The resulting system supports appearance-based recommendations for items such as clothing and accessories.
Problem
The paper addresses the challenge of modeling human notions of whether object appearances complement or substitute for one another rather than merely measuring visual similarity.
Method
The authors exploit a large web-derived dataset of related images and learn visual relationships with a scalable model that captures feature interactions beyond similarity.
Results
The proposed method outperforms category-based and weighted-nearest-neighbor baselines across the evaluated cases, with performance improving from K = 10 to K = 100.
Takeaways & Limitations
The system can model visual relationships beyond similarity, particularly complementarity, and can recommend objects based on the appearance of another object.
Takeaways & Limitations
Evaluation uses Amazon recommendations rather than ground truth for human visual preferences, so the relationships reflect factors beyond object appearance.
Abstract
from arXiv · showhide
Humans inevitably develop a sense of the relationships between objects, some of which are based on their appearance. Some pairs of objects might be seen as being alternatives to each other (such as two pairs of jeans), while others may be seen as being complementary (such as a pair of jeans and a matching shirt). This information guides many of the choices that people make, from buying clothes to their interactions with each other. We seek here to model this human sense of the relationships between objects based on their appearance. Our approach is not based on fine-grained modeling of user annotations but rather on capturing the largest dataset possible and developing a scalable method for uncovering human notions of the visual relationships within. We cast this as a network inference problem defined on graphs of related images, and provide a large-scale dataset for the training and evaluation of the same. The system we develop is capable of recommending which clothes and accessories will go well together (and which will not), amongst a host of other applications.
1 Introduction
The paper models human judgments about whether objects visually complement or substitute for one another, rather than treating visual similarity as sufficient. It combines large-scale web-derived relationships with image-based modeling for visual recommendation.
- The task is to model human notions of complementarity and substitution between object appearances, not merely visual similarity.
- Object relationships support recommendations such as selecting matching clothing, while appearance-based inputs may help address cold-start limitations in metadata- and history-based systems.
- The approach avoids extensive manual annotation by exploiting a much larger, indirectly related web dataset whose connection to visual relationships is not explicit.
- The Styles and Substitutes dataset contains over 180 million relationships among almost 6 million objects, with images and category labels available for each object.
- Its recorded relationships include viewed, purchased, and co-purchased product connections, representing substitute and complement goods alongside Amazon’s estimated relationships.
- The proposed visual and relational recommender uses object appearance rather than metadata, reviews, or similar information to recommend one object based on interest in another.
2 The Model
The model learns distance transformations that place related products closer than unrelated products, using scalable low-dimensional style embeddings and optional user-specific style weights.
- 2 The Model: The model learns a parameterized distance where related objects receive lower distances than unrelated objects, with relationship probability increasing as distance decreases.Object features come from convolutional neural networks, and the model uses a shifted sigmoid to map distances to relationship probabilities.
- 2 The Model: A weighted nearest-neighbor distance emphasizes individual feature dimensions but cannot represent subtler compatibility relationships between visually dissimilar products.The limitation motivates modeling interactions between feature dimensions, such as style relationships between pants and shoes.
- 2 The Model: A Mahalanobis distance models relationships between pairs of feature dimensions, but a full-rank matrix is impractical at feature dimension F = 212 because it requires approximately 8 million parameters.The full-rank transform would also be prone to overfitting and cannot be fit tractably with existing solvers.
- 2.1 Style space: The model approximates the Mahalanobis matrix with a low-rank factorization, yielding O(FK) distance and derivative computation for scalable training.The resulting representation embeds products into a low-dimensional style-space where related products may be close despite visual dissimilarity.
- 2.2 Personalizing styles to individual users: Personalization learns user-specific weights over style-space dimensions, representing which style dimensions each user considers important.The construction requires users associated with edges, so the paper studies personalization using product pairs jointly reviewed by individual users.
3 Training
Training maximizes the likelihood of observed relationships alongside an equally sized randomly sampled negative set, optimizing the low-rank transform and sigmoid shift with gradient ascent.
- 3 Training: Training forms a negative set Q with |Q| = |R| and maximizes the likelihood of the observed relationship set together with these negative examples.The negative set contains relationships not present in the observed relationship set.
- 3 Training: The model optimizes Y and c by gradient ascent using hybrid L-BFGS, with likelihood and derivative computations parallelized across relationship pairs.Training the largest Amazon books dataset with rank K = 100 took around one day on a 12-core machine.
4 Experiments
Experiments evaluate image-based relationship prediction against category and weighted-nearest-neighbor baselines, using Amazon-derived relationships and image-only representations. The method consistently outperforms both baselines, produces coherent style-space structure, and supports recommendation and personalization analyses.
- Baselines: The experiments compare the proposed method with Weighted Nearest Neighbor and Category Tree baselines, alongside a review-topic WNN baseline.The Category Tree uses Amazon’s detailed category hierarchy, while the topic baseline uses 100-dimensional review-derived vectors.
- Baselines: 60% accuracy was achieved by the review-topic baseline on the largest Books dataset, making it noncompetitive with the alternatives.The authors associate this weakness with the scale of topic-model training and the scarcity of reviews for many products.
- Relationship prediction: The model estimates specified relationships purely from object images, and its improvement over WNN reflects visual relations beyond simple image similarity.The distinction between the proposed model and WNN is smallest for Books and greatest for Clothing Shoes and Jewelery.
- Style-space analysis: K-means style-space clusters show coherent clothing styles, meaningful book-category distinctions, and low-cost paths connecting visually consistent intermediate products.Book clusters appear to separate children’s, self-help, romance, and graphic novels; embedding visualizations also show gradual transitions among clothing categories.
- Personalization: The personalization experiment evaluates copurchase recommendations using a user personalization term on tuples of products and users who reviewed them.This task uses a separate copurchase dataset rather than the population-level graphs used in the earlier experiments.
5 Visualizing Style Space
The style-space embedding organizes products by learned visual relationships, revealing both obvious categories and subtler preference-linked clusters. Visualizations also show smooth paths between related items and meaningful structure beyond clothing, including books.
- Style-space clusters: K-means visualization of the K-dimensional embedding reveals clusters organized by colors, shapes, product categories, and cross-category visual themes.Examples include shoes, shirts, watches, colorful items, love hearts, and animals.
- Style-space clusters: Leather boots separate from ugg boots despite subtle visual differences, suggesting the embedding captures distinctions associated with different customer preferences.
- Beyond clothing: The model identifies meaningful visual information in book covers, achieving 70% accuracy when predicting book co-purchases.Children’s books, self-help books, romance novels, and comics form characteristic visual groups.
- Navigating style space: The learned distance measure produces visually smooth low-cost paths between items that are unlikely to be co-browsed directly.
- Navigating style space: A two-dimensional embedding shows sporting shoes drifting toward slippers and sandals, while underwear drifts toward shirts and coats.
6 Generating Recommendations
The model generates complementary-item recommendations by selecting visually related products from each desired category. Given a query item, it constructs outfits such as pants, tops, shoes, and accessories with generally consistent styles.
- Recommendation generation: Given a query item, the system recommends items from desired categories that are most likely to connect according to their learned visual style.
- Recommendation generation: For clothing demonstrations, an outfit combines pants, a top, shoes, and an accessory, while omitting the category containing the query item.
- Recommendation generation: For each category, the recommendation is the item with minimum distance under the model’s learned measure.
- Observed recommendations: The generated recommendations are generally reasonable, with items in each category usually exhibiting a consistent style.
7 Outfits in The Wild
The learned relationship model is applied to real-world outfits by scoring how likely their component pairs are to be related. It distinguishes coordination patterns among Top Gear presenters and provides small external validation through makeover outfits.
- Scoring outfits: Outfits are scored by the average log-likelihood of their component pairs under models trained on men’s or women’s co-purchases.Averaging removes bias toward outfits with more or fewer components.
- Top Gear outfits: Top Gear outfits show considerable coordination separation: Richard Hammond is typically least coordinated, James May most coordinated, and Jeremy Clarkson spans both extremes.
- Makeover outfits: Made-over outfits have higher log likelihood in 12 of 17 observed cases, with p ≃7%.The log-likelihoods are normalized to correct for the number of outfit components.
- Measurement boundary: The coordination measure is undefined for outfits containing only a single item.
- Makeover outfits: The makeover comparison provides external, albeit small, validation of the learned model independent of its training dataset.
8 Conclusion
The paper shows that large, indirectly related datasets can support modeling human visual relationships beyond simple similarity, especially complementarity. It presents this as a scalable contribution with applications to visual and relational recommendation.
- Conclusion: A suitably large dataset can model human notions of visual relatedness even when the relevant information is only tangentially contained in the data.
- Conclusion: The method models visual relationships beyond simple similarity, including what makes items complementary.
- Conclusion: The authors describe the approach as an early direct, large-scale attempt to model human preference for one object’s appearance given another’s.
- Conclusion: The work proposes visual and relational recommender systems and provides a large dataset for their training and evaluation.