Source-linked AI summary
Wide & Deep Learning for Recommender Systems
Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, Hemal Shah
TL;DR
Sparse recommender systems must balance memorizing observed feature interactions with generalizing to unseen combinations without over-generalizing to irrelevant items. The paper jointly trains wide linear and deep neural components, and online experiments on Google Play showed significant app-acquisition improvements over wide-only and deep-only models.
Problem
Sparse, high-rank user-item interactions make embeddings prone to over-generalization, while memorization through cross-products requires more feature engineering.
Method
Wide & Deep jointly trains a linear model with cross-product features and a neural network using low-dimensional embeddings.
Results
+3.9% app acquisition rate on the main landing page relative to the wide-only control group, and +1% over the deep-only model.
Takeaways & Limitations
Wide & Deep combines memorization and generalization for large-scale recommendation and ranking with sparse input data.
Abstract
from arXiv · showhide
Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks can generalize better to unseen feature combinations through low-dimensional dense embeddings learned for the sparse features. However, deep neural networks with embeddings can over-generalize and recommend less relevant items when the user-item interactions are sparse and high-rank. In this paper, we present Wide & Deep learning---jointly trained wide linear models and deep neural networks---to combine the benefits of memorization and generalization for recommender systems. We productionized and evaluated the system on Google Play, a commercial mobile app store with over one billion active users and over one million apps. Online experiment results show that Wide & Deep significantly increased app acquisitions compared with wide-only and deep-only models. We have also open-sourced our implementation in TensorFlow.
1. INTRODUCTION
Recommender systems benefit from both memorizing observed feature interactions and generalizing to unseen combinations. Wide & Deep learning jointly combines these capabilities through a linear model and a neural network.
- The framework targets app recommendation in Google Play while being intended to apply to generic recommender systems.
- Wide linear models memorize sparse feature interactions through cross-product transformations, while deep networks generalize through dense embeddings.
- Embedding-based models can over-generalize when query-item interactions are sparse and high-rank, producing nonzero predictions for many irrelevant pairs.
- Wide & Deep jointly trains a linear model component and a neural network component to achieve memorization and generalization in one model.
2. RECOMMENDER SYSTEM OVERVIEW
A recommender system retrieves and ranks items for a user query containing user and contextual information. Because the app database is very large, retrieval narrows the candidates before ranking scores them.
- A recommendation query can include various user and contextual features, and the system returns apps on which users may click or purchase.
- Retrieval first reduces the candidate pool because exhaustively scoring over a million apps cannot meet serving-latency requirements.
- The ranking system scores the retrieved items, usually with P(y|x), the probability of a user action label given the features.
3. WIDE & DEEP LEARNING
The Wide & Deep framework combines a wide linear component that memorizes feature interactions with a deep neural component that learns dense embeddings for generalization. Joint training combines their output log odds and optimizes both components through one common loss.
- The Wide Component: The wide component is a generalized linear model using raw features and transformed cross-product features to capture feature interactions.Cross-product transformations are nonlinear feature transformations over sparse inputs.
- The Deep Component: The deep component converts sparse categorical features into low-dimensional dense embeddings, then feeds the embeddings into a feed-forward neural network.The embeddings are trained to minimize the final loss function, and their dimensionality is usually O(10) to O(100).
- Joint Training: The wide and deep components are combined through a weighted sum of their output log odds before the shared logistic loss.The combined model produces the prediction for a logistic regression problem.
- Joint Training: Joint training optimizes the wide and deep parameters simultaneously, unlike an ensemble that trains separate models and combines predictions only at inference.The wide component can therefore complement the deep component with a smaller set of cross-product transformations.
4. SYSTEM IMPLEMENTATION
The apps recommendation implementation uses separate data-generation, model-training, and model-serving stages. It combines sparse and dense inputs in a Wide & Deep model and uses parallelized serving to meet latency requirements.
- Pipeline: The apps recommendation pipeline consists of data generation, model training, and model serving.These three stages are presented as the implementation structure for the recommendation system.
- Data Generation: Training examples represent app impressions and use app acquisition as the binary label, while vocabularies map categorical feature strings to integer IDs.Continuous real-valued features are normalized to the [0, 1] interval.
- Model Training: The experimental model uses cross-products of installed and impression apps, 32-dimensional categorical embeddings, approximately 1200 concatenated dimensions, and 3 ReLU layers.The model structure combines the wide cross-product with the deep embedding and dense-feature representation.
- Model Training: The system warm-starts retraining by initializing new models with embeddings and linear weights from the previous model.This addresses the computational cost and serving delay associated with retraining from scratch.
- Model Serving: Before deployment, the model undergoes a dry run and empirical validation against the previous model as a serving and quality sanity check.The validation occurs before loading the model into live model servers.
- Model Serving: Serving receives retrieved app candidates and user features, scores each candidate with a forward pass, and ranks apps by score.The serving system uses multithreaded smaller batches to target latency on the order of 10 ms.
5. EXPERIMENT RESULTS
Live experiments evaluated Wide & Deep for app acquisitions and serving performance, finding significant acquisition gains over wide-only and deep-only models and lower serving latency with multithreading.
- Evaluation: Wide & Deep was evaluated in live experiments using app acquisitions and serving performance as the main measures.The evaluation included online experiments and operational serving metrics.
- App Acquisitions: +3.9% relative app acquisition gain was achieved over the wide-only control on the app store’s main landing page.The result was statistically significant in a three-week A/B test.
- App Acquisitions: +1% app acquisition gain was achieved over the deep-only model with the same features and neural network structure.This comparison was also statistically significant.
- Offline and Online Metrics: Wide & Deep had a slightly higher offline AUC, while its online impact was more significant.The paper attributes this possible difference to fixed offline impressions and labels versus exploratory online recommendations and new user responses.
- Serving Performance: 14 ms client-side latency was achieved with multithreading, compared with 31 ms for single-threaded scoring of all candidates in one batch.The 14 ms figure includes serving overhead.
6. RELATED WORK
Related work combines linear and neural approaches across recommender systems, language models, and computer vision; this paper applies joint wide-and-deep training to sparse user and impression data.
- Hybrid Models: Factorization machines combine linear modeling with generalization by factorizing pairwise interactions into low-dimensional embedding dot products.Wide & Deep instead expands embedding interactions through neural networks with highly nonlinear interactions.
- Joint Training: Joint training of neural networks and linear models has also been used for language modeling, residual learning, and generic sparse-input ranking.These approaches create direct or shortcut connections alongside neural representations.
- App Recommendation: Wide & Deep differs from prior collaborative-filtering and content-based app recommenders by jointly training on user and impression data.Earlier examples include collaborative deep learning and AppJoy’s use of collaborative filtering on app usage records.
7. CONCLUSION
The conclusion presents Wide & Deep as a framework that combines memorization and generalization, with significant app-acquisition improvements over wide-only and deep-only models on Google Play.
- Core Findings: Wide linear models memorize sparse feature interactions, while deep neural networks generalize to unseen interactions through low-dimensional embeddings.The conclusion identifies both capabilities as important for recommender systems.
- Core Findings: Wide & Deep jointly combines these memorization and generalization strengths in one recommender model.The framework was productionized and evaluated on Google Play.
- Core Findings: Wide & Deep significantly improved app acquisitions over both wide-only and deep-only models.This conclusion is based on online experiment results in Google Play’s commercial recommender system.