Source-linked AI summary
Related Pins at Pinterest: The Evolution of a Real-World Recommender System
David C. Liu, Stephanie Rogers, Raymond Shiau, Dmitry Kislyuk, Kevin C. Ma, Zhigang Zhong, Jenny Liu, Yushi Jing
TL;DR
Web-scale recommenders must be built incrementally despite small teams, limited resources, noisy data, and no initial training set. This paper follows Related Pins over three years, documenting its evolving components and the trade-offs used to manage complexity. Related Pins now drives over 40% of saves and impressions, while the account also identifies operational constraints and system interdependencies.
Problem
Web-scale recommenders often begin with small teams, limited compute, no training data, billions of poorly labeled items, and noisy implicit feedback, while prior system accounts often omit incremental development.
Source: §1 · pp. 1–2Method
The paper presents a three-year longitudinal study of Related Pins’ evolving components, trade-offs, and solutions to real-world recommender challenges.
Source: §1 · p. 2Results
Takeaways & Limitations
Making more of the system real-time in candidate generation and ranking significantly increased experimentation velocity and improved responsiveness of results.
Source: §7 · p. 9Offline ranking covered 50% of user queries because of cluster limits and slowed experimentation because model changes required reranking all queries.
Source: §5.5 · p. 8
Abstract
from arXiv · showhide
Related Pins is the Web-scale recommender system that powers over 40% of user engagement on Pinterest. This paper is a longitudinal study of three years of its development, exploring the evolution of the system and its components from prototypes to present state. Each component was originally built with many constraints on engineering effort and computational resources, so we prioritized the simplest and highest-leverage solutions. We show how organic growth led to a complex system and how we managed this complexity. Many challenges arose while building this system, such as avoiding feedback loops, evaluating performance, activating content, and eliminating legacy heuristics. Finally, we offer suggestions for tackling these challenges when engineering Web-scale recommender systems.
1. INTRODUCTION
Related Pins illustrates how a recommender can grow from a constrained prototype into a major discovery mechanism, while making visible the challenges of incremental development at Web scale. The paper traces its three-year evolution and the trade-offs made along the way.
1. INTRODUCTION
- Web-scale recommenders must start with small teams, limited compute, no initial training data, noisy implicit feedback, and billions of poorly labeled items.Ad-hoc heuristics and trade-offs can make later changes difficult to reason about as the system grows.Source: §1 · p. 1
- Related Pins launched in 2013; in 2014, about 10% of pins saved across Pinterest were discovered through it, and it now drives over 40% of all saves and impressions through multiple product surfaces.Source: §1 · p. 2
- Earlier recommender-system papers often describe final systems rather than incremental construction; this paper gives concrete examples of challenges and solutions in Related Pins.The team prioritized simple, high-leverage products, beginning with a heuristic candidate generator built around user-curated boards and later adding memorization and learned ranking.Source: §1 · p. 2
2. RELATED PINS SYSTEM OVERVIEW
Related Pins recommends pins for a query pin using Pinterest’s human-curated content and serves recommendations across several product surfaces. Its pipeline narrows candidates, incorporates historical engagement, and ranks results for save propensity.
2. RELATED PINS SYSTEM OVERVIEW
- Related Pins uses human-curated pins and boards to recommend pins related to a query pin across the closeup view, Home Feed, and other Pinterest surfaces.User engagement includes closeups, link clicks, long clicks, and saves; the system targets Related Pins Save Propensity, the fraction of viewers who save a recommended pin.Source: §2 · p. 2
- Each image can appear in multiple pins across boards, so the system aggregates pin information by image signature and uses that aggregation for query and result pins..Source: §2 · p. 2
- Candidate generation narrows billions of pins to roughly 1,000; Memboost uses historical query-result engagement, and learned ranking orders candidates to maximize Save Propensity.Ranking combines query and candidate, user-profile, session-context, and Memboost features, trained on past engagement; the architecture evolved over time.Source: §2 · pp. 2–3
3. EVOLUTION OF CANDIDATES
Candidate generation evolved from board-based heuristics toward a more diverse set of sources, addressing relevance, coverage, rare pins, and content activation. The system combines graph, session, search, visual, and locale-specific approaches.
3. EVOLUTION OF CANDIDATES
- The original board-graph candidate source later shifted from precision toward recall, adding sources to address coverage gaps and provide diverse candidates.Board co-occurrence remains the primary generator, with methods changed to improve relevance and query coverage.Source: §3 · p. 3, §3.1 · p. 3
3.1.1 Heuristic Candidates
- The original heuristic generator sampled same-board pin pairs, added hand-tuned text and category relevance scores, and was built by two engineers in three weeks.The approach was chosen for simplicity and leveraged the strong signal in human-curated boards.Source: §3.1.1 · p. 3
3.1.2 Online Random Walk
- Online Pixie random walks over the pruned board-pin graph use roughly 100,000 steps and aggregated visit counts to approximate query-seeded Personalized PageRank.Higher board co-occurrence makes pins more likely to be visited, while multi-hop walks increase coverage for rare pins.Source: §3.1.2 · p. 3
3.2 Session Co-occurrence
- Board co-occurrence can group tangential pins on broad or drifting boards and miss related pins saved to separate narrow boards; Pin2Vec adds session co-occurrence.Pin2Vec learns embeddings from pins saved by the same user within a time window, then retrieves nearest neighbors for queries among its N popular pins.Source: §3.2 · p. 3
3.3 Supplemental Candidates
- Supplemental sources broaden candidate diversity: search uses query-pin annotations, while visual retrieval uses duplicate-image recommendations or nearest neighbors of visual embeddings.Search candidates are less specifically relevant than board-based candidates but offer more diverse, somewhat related results.Source: §3.3 · pp. 3–4
3.4 Segmented Candidates
- Locale-segmented candidate sets address content activation by generating recommendations from content in the viewer’s locale.For board co-occurrence, the input graph is filtered to pins of the given locale; the approach could extend to gender-specific or fresh content.Source: §3.4 · p. 4
4. EVOLUTION OF MEMBOOST
Memboost uses past query–result engagement to adjust Related Pins, evolving from a lightweight memorization layer into scoring and insertion mechanisms. Its benefits come with feedback-loop and experimentation costs.
4. EVOLUTION OF MEMBOOST
- Memboost was introduced to memorize strong result pins from engagement logs before attempting full-fledged learning because it was lightweight.The initial system already received substantial engagement.Source: §4 · p. 4
- Rank- and platform-specific click-through rates expose position bias, so Memboost uses clicks over expected clicks instead of raw historical click-through rates.Expected clicks accumulate fractional expectations from impressions at each rank and platform.Source: Figure 5 · p. 4, §4 · p. 4
4.1 Memboost Scoring
- The log of engagement actions relative to expected actions, with additive smoothing, adjusts existing pin scores, which are then sorted by the adjusted score.The action totals combine weighted engagement types; smoothing handles items with low action or impression counts.Source: §4 · p. 5, §4.1 · p. 5
- Because hand-tuned weights could become suboptimal when the model, scoring function, or time period changed, Memboost parameters are now jointly retrained and its intermediate values become ranker features.Source: §4.1 · p. 5
4.2 Memboost Insertion
- Memboost insertion restores the top n results by aggregate Memboost score when upstream candidate generation or ranking has removed them from incoming results.Source: §4.2 · p. 5
4.3 Discussion
- Memboost feedback loops can leak experimental gains into control and production and make launched experiments harder to retest, risks the authors currently accept given its significant positive impact.Source: §4.3 · p. 5
- Insertion may hide harmful experiments from negative A/B results, dilute new ranking effects, and indefinitely retain candidates no longer produced upstream.Source: §4.3 · p. 5
- Unlike item-id ranking features, which require a large model and training data, Memboost pre-aggregates engagement statistics so the main ranking model can train on one machine.Source: §4.3 · p. 5
5. EVOLUTION OF RANKING
The first learning-to-rank model increased users’ propensity to save and click results by over 30%. Ranking turns heterogeneous query and candidate data into features and explores choices in training data, objective, and model form.
5. EVOLUTION OF RANKING
- Over 30%, the first learning-to-rank model increased users’ propensity to save and click results.Candidate generation and Memboost were already operating before ranking was introduced.Source: §5 · p. 5
- Feature extractors transform query and candidate raw data into a vector, and the ranking model maps that vector to a score.Transformations include normalized Memboost data, one-hot categorical fields, and match scores; Table 1 gives examples of raw inputs.Source: §5 · pp. 5–6, Table 1 · p. 5
5.1 Choices; 5.1.1 Training Data Collection
- The system’s design choices span training-data collection, learning objective, and model type.The paper explores Memboost scores and individual Related Pins sessions as training-data sources.Source: §5.1 · p. 6, §5.1.1 · p. 6
5.1.2 Model Objective
- Pointwise, pairwise, and listwise objectives differ in whether their loss considers one, two, or many candidates at a time.The paper explores pointwise and pairwise approaches, compared in Table 2.Source: §5.1.2 · p. 6, Table 2 · p. 6
5.1.3 Model Formulation
- The model’s precise form determines its capacity to describe complex relationships between features and scores.Table 3 compares two model types used in the system.Source: §5.1.3 · p. 6
5.2 Evolution of our Decisions
Related Pins ranking evolved through changes to its training data, objective, and model. The progression addressed personalization and model expressiveness, with the final version using binary labels and pointwise classification to target save propensity.
5.2 Evolution of our Decisions
- Version 1 trained a linear RankSVM with pairwise loss on Memboost-derived relevance pairs, including a random popular pin to stabilize rankings.The paper presents the versions as combinations of training-data, objective, and model choices.Source: §5.2 · p. 6
- Version 2 switched from aggregated Memboost data to individual sessions to retain user and context information for personalization.Memboost data also provided reliable estimates mainly for popular content, motivating the switch.Source: §5.2 · p. 6
- Version 3 moved to a RankNet GBDT model because linear models required manual transformations and feature crosses to capture complex relationships.Decision trees can learn feature interactions, reducing the need for manual feature crosses and speeding development.Source: §5.2 · pp. 6–7
- Version 4 used pointwise classification, binary saved/not-saved labels, and a logistic GBDT, with positive examples reweighted to increase save propensity.The authors considered clicks and closeups potentially counterproductive because they may not reflect save propensity.Source: §5.2 · p. 7
5.3 Previous-Model Bias
Using engagement logs to train successive ranking models created previous-model bias: deployed rankings shaped later training examples. The system addressed this by collecting a small share of randomly ordered, unranked results, which produced better models than biased data.
5.3 Previous-Model Bias
- Training on engagement logs created a feedback loop: deployed rankings shaped future training data, and newer data did not let the same-feature model beat the deployed model.The authors hypothesized that training pins no longer matched the distribution of pins ranked at serving time.Source: §5.3 · p. 7
- A small random share of queries served randomly sampled, unranked candidates for unbiased training data, and models trained on it performed better than those trained on biased data.The lower-quality unranked results limited the intervention’s traffic share to avoid degrading users’ experience too much.Source: §5.3 · p. 7, Figure 6 · p. 7
5.4 Success Metrics
Online A/B experiments are the gold standard for evaluating ranking changes, but take days or weeks. Offline metrics provide a faster check: PR AUC predicts closeups and clickthroughs well, but save behavior remains difficult to predict.
5.4 Success Metrics
- Online A/B experimentation is the gold standard for testing changes, with ranking evaluated primarily by its impact on save propensity.Collecting data for these experiments takes days or weeks.Source: §5.4 · p. 7
- Offline evaluation rescores pins users actually saw in later-dated sessions and measures agreement between model scores and logged behavior.The authors reuse much of the training-data generator with a distinct date range and a slightly different sampling strategy.Source: §5.4 · p. 7
- PR AUC metrics were extremely predictive of closeups and clickthroughs in A/B experiments, but offline evaluation had difficulty predicting save behavior.The authors use offline metrics as a sanity check and rough estimate of potential impact.Source: §5.4 · p. 8
5.5 Serving Infrastructure
Related Pins moved from offline precomputation to online ranking to address limited coverage and slow experimentation. Its online serving architecture distributes scoring across nodes holding candidate data, reducing raw-data transfer.
5.5 Serving Infrastructure
- Related Pins serves many tens of thousands of queries per second at peak loads, using existing Pinterest systems to handle the scale.Source: §5.5 · p. 8
- Offline ranking covered only 50% of user queries and slowed experiments because each model change required reranking queries.The cluster could rank only a few million queries at a time.Source: §5.5 · p. 8
- The online system assembles candidate and user data, scores candidates on leaf servers, and returns the overall top results through a root server.The request combines pin data, offline user data, recent activity, and Memboost data.Source: §5.5 · p. 8
- Scoring candidates where their raw data is stored avoids transferring large amounts of pin data for each query.The architecture was chosen to increase data locality after Hadoop-based processing required substantial raw-data transfer.Source: §5.5 · p. 8
6. CHALLENGES
Related Pins’ challenges include interactions among system components and the difficulty of surfacing relevant local content. The team describes mitigations ranging from joint training and staged candidate integration to locale-aware results and pin substitution.
6.1 Changing Anything Changes Everything
- CACE means a component optimized for the system’s current state may perform worse after another component changes, motivating joint training and automated tuning.Memboost weights and ranking hyperparameters otherwise became outdated as the system changed.Source: §6.1 · p. 8
- Upstream data changes can harm a downstream model trained on an older feature definition; updating production currently requires manual retraining and coordinated deployment.The team says this is time-consuming and depends on learning of the upstream change.Source: §6.1 · p. 8
- Changing candidate generators can misalign training and serving distributions, so new candidates are added to training data before testing a newly trained model.Source: §6.1 · p. 8
- Unintended interactions multiply with system components, making simplification important.Source: §6.1 · p. 8
6.2 Content Activation
- Local pins were scarce in candidate sets and had less historical engagement, which led models to rank them lower even when candidates existed.The broader content-activation challenge is balancing fresh and rarely surfaced content with established high-quality content.Source: §6.2 · pp. 8–9
- Local results rose from 9% to 54%; swapping in a same-image pin in the viewer’s language increased local impressions and saves without changing relevance.Figure 7 shows the proportion of Related Pins impressions that are local for major international countries.Source: §6.2 · p. 9, Figure 7 · p. 9
- Promoting existing local pins to higher positions was ineffective when candidate sets contained few local pins.Source: §6.2 · p. 9
- Locale-filtered board-based candidates and blending local candidates into results were used to increase exposure to local pins.Source: §6.2 · p. 9
7. CONCLUSION
The paper traces how Related Pins grew from a constrained prototype into a complex, widely used recommender. It describes system interdependencies and practical responses, including automated joint training, content activation, and more real-time components.
7. CONCLUSION
- Related Pins grew organically from a system built by a small team with limited computational resources and noisy Web-scale data, illustrating how development can produce complexity.Source: §7 · p. 9
- The authors propose automated joint training to mitigate interdependencies and describe content activation and real-time candidate generation and ranking as ways to address system challenges.They report that making more of the system real-time significantly increased experimentation velocity and improved result responsiveness.Source: §7 · p. 9