Source-linked AI summary
E-commerce in Your Inbox: Product Recommendations at Scale
Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan Bhamidipati, Jaikit Savla, Varun Bhagwan, Doug Sharp
TL;DR
E-mail advertising lacks a consistently effective format, motivating personalized product ads based on purchase history extracted from receipts. The paper develops neural product-recommendation models, evaluates them against popularity and co-occurrence baselines at large scale, and reports improved live ad metrics before production launch.
Problem
E-mail clients lack a winning advertising formula, although inboxes contain commercial activity and purchase information relevant to ad targeting.
Method
The system learns product embeddings from purchase sequences, clusters products, and models cluster transitions to recommend products for Yahoo Mail users.
Results
9% lift in click-through rates over other mail ad formats was observed online, with comparable conversion-rate lift; the system was subsequently launched in production.
Takeaways & Limitations
Receipt-based recommendations let an e-mail provider use purchase history across commercial domains for more personalized cross-vendor targeting.
Takeaways & Limitations
User-to-product recommendations require frequent updates because they must reflect recent purchases, unlike product-to-product recommendations that may remain relevant longer.
Abstract
from arXiv · showhide
In recent years online advertising has become increasingly ubiquitous and effective. Advertisements shown to visitors fund sites and apps that publish digital content, manage social networks, and operate e-mail services. Given such large variety of internet resources, determining an appropriate type of advertising for a given platform has become critical to financial success. Native advertisements, namely ads that are similar in look and feel to content, have had great success in news and social feeds. However, to date there has not been a winning formula for ads in e-mail clients. In this paper we describe a system that leverages user purchase history determined from e-mail receipts to deliver highly personalized product ads to Yahoo Mail users. We propose to use a novel neural language-based algorithm specifically tailored for delivering effective product recommendations, which was evaluated against baselines that included showing popular products and products predicted based on co-occurrence. We conducted rigorous offline testing using a large-scale product purchase data set, covering purchases of more than 29 million users from 172 e-commerce websites. Ads in the form of product recommendations were successfully tested on online traffic, where we observed a steady 9% lift in click-through rates over other ad formats in mail, as well as comparable lift in conversion rates. Following successful tests, the system was launched into production during the holiday season of 2014.
1. INTRODUCTION
The paper develops personalized product ads for Yahoo Mail from purchase information extracted from e-mail receipts, addressing the difficulty of attracting attention in e-mail environments. Its neural recommendation system was evaluated at scale and improved purchase prediction and live ad performance over baseline approaches.
- Motivation: E-mail users spend substantial time on commercial activities, but persuading them to leave “email mode” and click ads remains difficult.The paper identifies relevance and personalization as central to overcoming users’ narrow focus on handling mail.
- Motivation: Receipt schemas expose purchased product names and prices, enabling targeting beyond mail retargeting based only on communication with an e-commerce site.Aggregated purchase data across commercial e-mail domains also supports recommendations spanning vendors.
- Motivation: Cross-vendor recommendations connect products bought from one vendor with products bought from another, extending the usual “Customers who bought X also bought Y” pattern.The paper presents this broader view as a more powerful targeting opportunity for e-mail providers.
- Approach: The proposed system embeds products with a neural language model over purchase sequences, then clusters products and models transitions between clusters to generate diverse recommendations.Products with similar purchase contexts receive nearby vectors, and candidates from probable clusters form final recommendations.
- Evaluation: More than 280 million purchases from 29 million users and 2.1 million products supported offline evaluation against global, cohort, and recent co-purchase baselines.Cohort back-fill recommendations addressed users without earlier purchases.
- Results: The model improved purchase prediction over baselines, substantially improved click and conversion metrics in live bucket tests, operated below 200ms latency, and launched in production.The live system was tested in comparable ad slots before deployment.
2. RELATED WORK
Prior work covers behavioral ad targeting, commercial e-mail analysis, purchase-based recommendation, and neural language models. The paper positions product embeddings learned from purchase sequences within this broader line of research.
- Ad targeting: Web platforms support behavioral ad targeting through detailed records of user information, activity, and interactions.These records include clicks, page views, searches, visits, social activity, and ad interactions.
- Commercial e-mail data: Publisher event features can predict clicks or purchases but are described as only weak proxies for users’ purchase intent.The paper contrasts these signals with commercial e-mails containing purchase-related information.
- Commercial e-mail data: Early work on commercial e-mail data applied Sparse Principal Component Analysis to received-mail counts to cluster commercial domains.That study demonstrated promise for commercial e-mail as a data source.
- Recommendation systems: E-commerce recommenders use collaborative filtering from similar users or context-based methods using a user’s historical interactions with products.The paper distinguishes its approach from these established e-commerce settings.
- Neural language models: Neural language models learn low-dimensional distributed word embeddings by modeling statistical dependence associated with word order.Distributed representations have also been extended to sentences, entities, attributes, images, and graph nodes.
3. PROPOSED APPROACH
The proposed approach learns low-dimensional product and user representations from purchase sequences in e-mail receipts, then uses those representations for product recommendations. It includes product-to-product, cluster-based, and user-to-product models tailored to receipt structure and purchase history.
- Low-dimensional product embeddings: Product recommendations use low-dimensional product embeddings learned from historical e-mail receipt logs, followed by nearest-neighbor search.The objective is to place similar products near one another in a D-dimensional vector space.
- Low-dimensional product embeddings: prod2vec treats purchase sequences as sentences and products as words, learning representations with a skip-gram language model.Products with similar neighboring-purchase contexts receive similar vector representations.
- Product-to-product predictive models: bagged-prod2vec modifies skip-gram training to model e-mail receipts as shopping bags, preventing products purchased in the same receipt from predicting one another.The model instead predicts products from neighboring receipts and can use only future products as context to capture temporal purchase patterns.
- Product-to-product predictive models: Product-to-product recommendations rank products by cosine similarity to a purchased product, either directly or after selecting related product clusters.The cluster model first ranks clusters by transition probability, then ranks products within top clusters by cosine similarity.
- User-to-product predictive models: user2vec jointly learns product and user vectors, recommending products by finding nearest products to a user in the shared embedding space.User vectors are trained to predict products from users’ purchase sequences, while the method can be extended to the bagged formulation.
- User-to-product predictive models: User-to-product recommendations are tailored to purchase history but require frequent updates because recent purchases can quickly change their relevance.This update requirement contrasts with product-to-product recommendations, which may remain relevant for longer periods.
4. EXPERIMENTS
The experiments analyze purchase behavior and evaluate popular-product and neural recommendation methods using large-scale e-mail receipt data. Offline and live tests show that tailored predictions improve recommendation performance and business metrics over baseline approaches.
- Data and evaluation: The data set contains more than 280.7 million purchases from 29 million users across 172 commercial websites, with 2.1 million products.Recommendations were evaluated on a held-out month using prediction accuracy and a daily budget of K = 20 distinct recommendations per user.
- Purchase behavior: Female shoppers consistently represent a higher percentage of online shoppers across age groups, while male users buy more expensive items on average.The analysis examines demographic and geographic purchasing differences to inform targeting.
- Predicted products: Time-decayed recommendation scoring combines predictions from multiple prior purchases and selects the top K products after sorting by decayed score.The decay factor α = 0.9 produced the best prod2vec accuracy in the reported evaluation.
- Predicted products: The bagged-prod2vec-cluster model provides the best prod2vec performance, while neural methods outperform cohort-based popular products and co-purchase predictions.User2vec performs best on day 1 but loses predictive power quickly, whereas prod2vec variants remain steady across the first 7 days.
- Live bucket results: In live bucket tests, predicted recommendations slightly exceed popular recommendations in CTR and significantly outperform them in yield rate, while popular recommendations become stale faster over time.Both recommendation buckets outperform standard ads, and seven-day updates are insufficient.
5. SYSTEM DEPLOYMENT
The Yahoo Mail recommendation system was productionized through continuously updated product-to-product predictions, scalable storage and processing, and asynchronous ad retrieval. Recommendations were displayed in the inbox and supplemented with popular-product back-fill for users without purchase history.
- Model and data updates: Near-real-time recommendations used product-to-product predictions from the bagged-prod2vec-cluster model, updated every 5 days with recent purchase data.Product vectors were stored on HDFS and updated through Hadoop-based parallel processing.
- Cold-start handling: Popular products served as back-fill for users without prior purchase history.The system recalculated popular products every 3 days using a 5-day lookback and randomized 100-product cohort lists.
- Serving architecture: A multi-tier architecture stored user profiles and prediction models in distributed key-value stores while batch updates avoided affecting live traffic.Separate processes queried purchases, retrieved predictions and offers asynchronously, and returned selected ads to the presentation layer.
- Ad serving: The system showed a new recommendation after each inbox action, placing product ads above the first e-mail in the pencil position.Examples of triggering actions included folder clicks, composing e-mail, and inbox searches.
6. CONCLUSIONS AND FUTURE WORK
The paper presents a large-scale Yahoo Mail product-ad framework based on neural language models for product-to-product and user-to-product prediction. After offline variant testing and online bucket tests, the selected system was launched in production, with implicit ad feedback identified for future improvement.
- The framework was launched on Yahoo Mail as product ads after large-scale recommendation development.
- Neural language models learned product embeddings for product-to-product predictions and user embeddings for user-to-product predictions.
- Offline model variants were tested, the best candidate entered an online bucket test, and encouraging results preceded production launch.
- Future work will use implicit feedback from ad views, clicks, and conversions to improve recommendation performance.