Source-linked AI summary
PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest
Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, Jure Leskovec
TL;DR
Single user embeddings provide an incomplete representation of diverse interests, limiting personalized recommendation systems. PinnerSage clusters user actions into conceptual groups, represents each with multiple embeddings and medoids, and significantly improves offline relevance and online engagement at Pinterest.
Problem
Single user embeddings offer limited insight into users’ multiple interests, raising questions about effective multi-embedding recommendation at production scale.
Method
PinnerSage clusters user actions with Ward hierarchical clustering, represents clusters with medoids, and uses multiple embeddings for scalable recommendation.
Results
PinnerSage significantly improves retrieval relevance, ranking reciprocal rank, and Pinterest homefeed engagement over single-embedding baselines.
Takeaways & Limitations
Multi-embedding user representations can support more relevant and engaging personalized recommendations in a large-scale production system.
Takeaways & Limitations
PinnerSage assumes fixed pin embeddings generated by a black-box model and uses a two-pronged update strategy.
Abstract
from arXiv · showhide
Latent user representations are widely adopted in the tech industry for powering personalized recommender systems. Most prior work infers a single high dimensional embedding to represent a user, which is a good starting point but falls short in delivering a full understanding of the user's interests. In this work, we introduce PinnerSage, an end-to-end recommender system that represents each user via multi-modal embeddings and leverages this rich representation of users to provides high quality personalized recommendations. PinnerSage achieves this by clustering users' actions into conceptually coherent clusters with the help of a hierarchical clustering method (Ward) and summarizes the clusters via representative pins (Medoids) for efficiency and interpretability. PinnerSage is deployed in production at Pinterest and we outline the several design decisions that makes it run seamlessly at a very large scale. We conduct several offline and online A/B experiments to show that our method significantly outperforms single embedding methods.
1 INTRODUCTION
PinnerSage addresses the challenge of recommending relevant ideas to Pinterest’s hundreds of millions of users by representing each user with multiple embeddings. It uses hierarchical clustering and medoids in a production-deployed, scalable recommender system.
- Motivation: Pinterest’s personalized recommendations help users discover relevant ideas across billions of visual bookmarks and are pervasive throughout its products.Pinterest has 350M+ monthly active users and 2B+ pins.
- Challenge: User interests span diverse, evolving facets, making it difficult to encode each user effectively for recommendations from billions of candidate items.Some interests persist long term, while others last only briefly.
- Prior Work: Multi-embedding representations can improve recommendations, but prior work leaves unresolved how many embeddings to use, how to operate at scale, and how to select them.Prior work reported a 25% improvement in YouTube video recommendations, yet production deployment questions remain.
- Prior Work: Prior multi-embedding approaches often avoid production deployment or restrict users to very few embeddings, limiting their utility.These limitations leave large-scale industrial use insufficiently addressed.
- Present Work: PinnerSage is a production-deployed, scalable recommender system that represents users with multiple PinSage embeddings derived from clustered actions and summarized by medoids.The system uses hierarchical clustering to form conceptual clusters and medoids to represent them efficiently.
2 PINNERSAGE DESIGN CHOICES
PinnerSage uses fixed pin embeddings, unconstrained multi-embedding user representations, medoid-based cluster summaries, sampled medoid retrieval, dual real-time update mechanisms, and an optimized approximate nearest-neighbor system. These choices target interpretability, scalability, efficient retrieval, and adaptation to both long-term and current user interests.
- Design Choice 1: Pin Embeddings are Fixed: PinnerSage keeps pin embeddings fixed to avoid the complexity, slower inference, and real-time update difficulties of jointly learning user and item embeddings.The design also addresses side effects that jointly learned embeddings can introduce in large-scale applications.
- Design Choice 2: No Restriction on Number of Embeddings: Users have as many embeddings as their data supports, with Ward clustering grouping actions into conceptually coherent clusters.A light user may have 3-5 clusters, whereas a heavy user may have 75-100 clusters.
- Design Choice 3: Medoids based Representation of Clusters: Each cluster is represented by a medoid pin rather than a centroid, avoiding topic drift and outlier sensitivity while requiring only a pin ID for storage.Because the medoid is an originally interacted pin, it can also support cross-user and cross-application cache sharing.
- Design Choice 4: Medoid Sampling for Candidate Retrieval: For candidate retrieval, PinnerSage samples 3 medoids proportional to importance scores and recommends their nearest neighboring pins.Sampling limits retrieval cost and avoids bombarding users with too many different items.
- Design Choice 5: Two-pronged Approach for Handling Real-Time Updates: PinnerSage combines daily batch inference from long-term interaction history with online inference to handle both accurate user representation and real-time needs.The long-term history spans 60-90 days of activity.
- Design Choice 6: Approximate Nearest Neighbor System: An approximate nearest-neighbor system retrieves k pins closest to each medoid in embedding space, while production optimizations reduce cost to 1/10 of the original prototype.The improvements include filtering low-quality pins, selecting indexing techniques carefully, and caching medoids.
3 OUR APPROACH
PinnerSage infers a variable number of user embeddings from action pins in a fixed PinSage embedding space. It clusters recent actions into conceptually coherent groups, represents each group compactly, and scores their importance to capture multiple interests efficiently.
- Problem formulation: Each user’s actions are time-ordered clicks or repins, and fixed PinSage embeddings place similar pins nearby for retrieval.The fixed-embedding assumption simplifies model complexity because pin embeddings are generated by a black-box PinSage model rather than jointly inferred.
- Problem formulation: The goal is to infer a variable number of d-dimensional embeddings per user from the user’s actions and fixed pin embeddings.Different users may have different numbers of embeddings, while each embedding remains compatible with the pin-embedding space for retrieval.
- Design rationale: Single embeddings are too limiting, whereas clustering offers a practical trade-off between recommendation accuracy and storage requirements.The preceding next-action task exposes weak correlations between the latest pin and older relevant pins, causing limited-memory single-embedding models to fail.
- Design rationale: PinnerSage clusters users’ action pins from the last 90 days, computes a medoid representation for each cluster, and assigns each cluster an importance score.These three components provide compact, multiple representations of a user’s interests.
- Clustering: Ward hierarchical clustering combines conceptually similar pins using minimum-variance merges and automatically determines the number of clusters from distances.The method is adapted from the Lance-Williams algorithm for efficient clustering and has computational complexity O(m^2).
- Cluster importance: Cluster importance combines interaction frequency and recency, with λ = 0 emphasizing frequency, λ = 0.1 emphasizing recency, and λ = 0.01 providing a balance.Clusters interacted with frequently and recently receive higher importance.
4 PINNERSAGE RECOMMENDATION SYSTEM
PinnerSage combines importance-sampled medoid retrieval with large-scale ANN infrastructure, candidate refinement, and caching. Production updates use daily batch inference plus lightweight online inference to balance recommendation relevance with cost and latency.
- Medoid-Based Retrieval: PinnerSage samples at most 3 medoids per user, then uses them to retrieve candidates from the nearest-neighbor system.The number of inferred medoids can vary with the underlying data, but only three are sampled at a time for recommendation generation.
- Approximate Nearest-Neighbor Retrieval: Pinterest’s ANN system indexes billions of pin embeddings, and HNSW performed best among evaluated indexing schemes for cost, latency, and recall.The ANN infrastructure is designed to meet prescribed cost and latency limits at very large scale.
- Candidate Pool Refinement: Specialized in-house models remove near-duplicate and lower-quality pins from the candidate pool.The filtering targets duplicates and pins with issues such as low resolution or excessive text.
- Caching Framework: Medoid pin IDs enable caching and reduce repeated ANN calls, especially when popular pins serve as medoids for multiple users.Pin IDs are easier to cache than arrays of d floating-point embedding values.
- Productionization: Productionization depends critically on system optimization, while the two-pronged update strategy imposes limitations and preserves independently improvable components.The architecture separates components so they can be improved independently despite the update constraints.
5 EXPERIMENT
PinnerSage produces coherent multi-interest clusters and diverse, relevant recommendations, and large-scale experiments show gains over single-embedding baselines in retrieval, ranking, and engagement. Offline analyses further identify Ward clustering, medoid representations, and three embeddings as effective design choices.
- Qualitative assessment: PinnerSage groups contextually similar pins into coherent clusters and recommends a relevant mix spanning the user’s top interests.The illustrated recommendation set covers shoes, gadgets, and food based on the user’s prior interactions.
- A/B experiments: In randomized A/B experiments, users receiving PinnerSage recommendations are compared with users receiving decay-average single-embedding recommendations using equal recommendation counts.The experiments evaluate engagement volume, including repins and clicks, across Pinterest surfaces.
- Offline evaluation: Offline evaluation uses tens of millions of users’ activities from the previous 90 days, with chronological daily testing and model updates after each test batch.Baselines include single-embedding models, PinnerSage variants, multiple losses, and several negative-sampling strategies.
- Retrieval task: PinnerSage retrieves candidates from billions of pins using multiple user embeddings, combining nearest-neighbor results into a recommendation set of at most 400 pins.Retrieval is evaluated by relevance, based on cosine similarity ≥0.8, and recall of observed action pins.
- Retrieval and ranking results: PinnerSage outperforms retrieval baselines and ranking baselines, with its single-embedding version exceeding state-of-the-art single-embedding sequence methods.Ranking is measured using R-Precision and Reciprocal Rank, and the results support user embeddings as a stand-alone ranking feature.
- Design choices and diversity: Ward clustering outperforms K-means and complete-link methods, while medoid and sequence-based cluster embeddings perform similarly; λ = 0.01 outperforms λ = 0.Increasing the number of embeddings increases both relevance and diversity, with e = 3 identified as a diversity sweet spot and relevance gains tapering for e > 3.
6 RELATED WORK
Prior research has extensively learned user and item embeddings, drawing on scalable word-representation models such as Word2Vec for recommendation and other applications. Other work represents users with multiple embeddings to capture changing interests, multiple meanings, or nonlinear factorization effects.
- User and item embeddings: Extensive prior research learns embeddings for users and items across recommendation and other application settings.This line of work includes models inspired by word representation learning.
- User and item embeddings: Word2Vec provides scalable continuous bag-of-words and skip-gram language models that have influenced embedding-based research.Researchers have adopted word-representation models from several domains, including recommendation candidate ranking.
- Multiple user embeddings: Prior multi-embedding approaches model changing user interests, multiple meanings, or nonlinear factorization effects.One factorization-based approach reports a 25% improvement in YouTube recommendations.
7 CONCLUSION
PinnerSage is an end-to-end Pinterest recommendation system that replaces single-embedding user representations with a multi-embedding scheme. Its clustering approach is designed to provide fuller insight into users’ needs while supporting production deployment through additional design choices.
- PinnerSage is an end-to-end system powering personalized recommendation at Pinterest.
- Unlike prior production systems using a single embedding, PinnerSage represents users with multiple embeddings.
- Its clustering scheme is intended to provide fuller insight into users’ needs and improve understanding of them.
REPRODUCIBILITY SUPPLEMENTARY MATERIALS · APPENDIX A: Convergence proof of Ward clustering algorithm
Appendix A proves convergence properties of Ward clustering by establishing monotonicity of merged-cluster distances and preventing clusters from being repeatedly added to the stack. The proof combines distance inequalities, symmetry, and a contradiction argument.
- APPENDIX A: Convergence proof of Ward clustering algorithm: A merged cluster’s distance to any other cluster is at least the smaller of its two child clusters’ distances to that cluster.This is the central monotonicity lemma for Ward clustering.
- APPENDIX A: Convergence proof of Ward clustering algorithm: When clusters i and j merge, their pairwise distance satisfies dij ≤ dik and dij ≤ djk.The proof parameterizes dik and djk as d + γ and d + γ + δ, with γ, δ ≥ 0.
- APPENDIX A: Convergence proof of Ward clustering algorithm: Ward clustering cannot add the same cluster twice to the stack.Lemma 8.2 establishes this property as a prerequisite for convergence.
- APPENDIX A: Convergence proof of Ward clustering algorithm: A contradiction argument uses three stack-order inequalities and distance symmetry to show that repeated addition would require incompatible distance relations.The conditions imply dik ≤ dij while the first condition requires dij ≤ dik, forcing equality and preventing the subsequent addition.
- APPENDIX A: Convergence proof of Ward clustering algorithm: After cluster i first enters the stack, later mergers cannot add i again because Lemma 8.1 preserves a distance no smaller than the closest child distance.Thus, if the child cluster closest to i cannot add i, neither can the merged cluster.