Source-linked AI summary
Product-based Neural Networks for User Response Prediction
Yanru Qu, Han Cai, Kan Ren, Weinan Zhang, Yong Yu, Ying Wen, Jun Wang
TL;DR
User response prediction supports decisions in recommender systems, web search, and online advertising. PNNs combine embeddings, product-based interaction modeling, and fully connected layers, and outperform compared models across datasets and metrics.
Problem
User response prediction estimates positive responses such as clicks or purchases, supporting downstream decisions including document ranking and ad bidding.
Method
PNN uses an embedding layer, a product layer for inter-field feature interactions, and fully connected MLPs to model high-order patterns.
Results
PNNs perform best on the Criteo and iPinYou datasets across AUC, log loss, RMSE, and RIG, with significant improvements over baseline models under log loss.
Takeaways & Limitations
PNN provides an efficient and effective approach for learning feature interactions and high-order latent patterns in multi-field categorical data.
Abstract
from arXiv · showhide
Predicting user responses, such as clicks and conversions, is of great importance and has found its usage in many Web applications including recommender systems, web search and online advertising. The data in those applications is mostly categorical and contains multiple fields; a typical representation is to transform it into a high-dimensional sparse binary feature representation via one-hot encoding. Facing with the extreme sparsity, traditional models may limit their capacity of mining shallow patterns from the data, i.e. low-order feature combinations. Deep models like deep neural networks, on the other hand, cannot be directly applied for the high-dimensional input because of the huge feature space. In this paper, we propose a Product-based Neural Networks (PNN) with an embedding layer to learn a distributed representation of the categorical data, a product layer to capture interactive patterns between inter-field categories, and further fully connected layers to explore high-order feature interactions. Our experimental results on two large-scale real-world ad click datasets demonstrate that PNNs consistently outperform the state-of-the-art models on various metrics.
I. INTRODUCTION
User response prediction estimates positive responses in personalization tasks, but multi-field categorical data is highly sparse and traditional or deep models face representation challenges. PNN addresses this by combining embeddings, product-based interactions, and fully connected layers to model latent patterns.
- Motivation: User response prediction estimates the probability of clicks or purchases in recommender systems, web search, and online advertising.These probabilities inform decisions such as document ranking and ad bidding.
- Data Representation: Multi-field categorical data is commonly converted into high-dimensional sparse binary features through one-hot encoding.An example represents weekday, gender, and city as concatenated one-hot field vectors.
- Limitations: Traditional models depend heavily on feature engineering to capture high-order latent patterns from sparse categorical features.Linear logistic regression, gradient boosting decision trees, and factorization machines are established approaches for these inputs.
- Prior Work: Deep neural networks can learn expressive representations, but applying them directly to high-dimensional categorical input remains challenging.Prior approaches include factorization-machine-based embeddings and convolutional click prediction models.
- Contribution: PNN uses an embedding layer, a product layer for inter-field interactions, and fully connected MLPs for high-order feature patterns.The proposed variants use inner-product or outer-product operations and avoid embedding pre-training.
- Results: PNNs consistently outperform state-of-the-art user response prediction models on various metrics across two large-scale real-world datasets.The experiments use CTR estimation as the working example.
III. DEEP LEARNING FOR CTR ESTIMATION
The paper formulates CTR estimation over multi-field categorical data and introduces PNN as an embedding-and-product architecture. Its product layer generates interaction signals that feed fully connected hidden layers before supervised prediction.
- Task Formulation: CTR estimation predicts the probability that a user clicks a specific advertisement in a given context.Each sample contains categorical fields from user, publisher, and advertisement information.
- Task Formulation: One-hot encoding creates enormous sparsity and curse-of-dimensionality challenges while field dependencies and hierarchical structures remain present.These properties motivate learning representations and interactions directly from categorical fields.
- PNN Architecture: PNN embeds each categorical field and applies a product layer to automatically explore feature interactions before subsequent neural processing.The model has inner-product IPNN and outer-product OPNN variants.
- PNN Architecture: The product layer combines linear signals and quadratic interaction signals, which are then passed to the first fully connected hidden layer.A constant signal preserves linear terms while producing quadratic signals.
- Embedding Layer: The embedding layer maps each field’s one-hot vector into a distributed embedding representation.The embedding parameters connect the field-specific one-hot slice to its embedding vector.
- Training and Output: Supervised training minimizes log loss, with the sigmoid output representing predicted CTR and the ground truth indicating click or non-click.The hidden layers use ReLU activation in the described construction.
B. Inner Product-based Neural Network
IPNN defines pairwise feature interactions with inner products, combines linear and quadratic signals, and reduces the resulting computation through matrix factorization and decomposition.
- Inner-product interactions: IPNN defines each pairwise interaction as the inner product of two embedded field vectors.
- Product-layer signals: The product layer preserves linear information while generating quadratic pairwise interaction signals.
- Complexity reduction: Without factorization, the first hidden layer has space complexity O(D1N(M + N)) and time complexity O(N^2(D1 + M)).
- Complexity reduction: Matrix factorization reduces the quadratic interaction computation, while a general K-order decomposition increases expressiveness at K times the model complexity.
- Complexity reduction: After reduction, the first-layer space and time complexities become O(D1MN), linear rather than quadratic in the number of fields N.
C. Outer Product-based Neural Network
OPNN models pairwise interactions with outer products, which produce matrix-valued signals but initially incur quadratic complexity in both field count and embedding dimension. Element-wise superposition reduces this cost substantially.
- Outer-product interactions: OPNN defines feature interaction with the outer product of two embedded vectors, producing a square matrix rather than a scalar.
- Complexity reduction: Element-wise superposition is introduced to reduce the outer-product computation complexity.
- Complexity reduction: After superposition, the first-layer space and time complexities become O(D1M(M + N)).
D. Discussions
PNN’s product layer generalizes interaction modeling beyond its inner- and outer-product variants and connects these variants to established FNN and FM models. The paper motivates product operations as a way to model rules in categorical data.
- Relations to prior models: Removing PNN’s quadratic product component makes it identical to FNN, while removing hidden layers and using uniform output weights makes inner-product PNN identical to FM.
- Generalized product layers: PNN uses product layers to explore feature interactions, with inner and outer products representing two possible implementations.
- Generalized product layers: More general product layers could provide greater capability for exploring feature interactions.
- Motivation: The paper motivates multiplication as an “AND” operation and addition as an “OR” operation for modeling rules in multi-field categorical data.
IV. EXPERIMENTS
The experiments evaluate PNN models for CTR estimation and report that they outperform major state-of-the-art models on two real-world datasets.
- PNN models outperform major state-of-the-art models in CTR estimation on two real-world datasets.
A. Datasets
The Criteo 1TB click-log dataset uses seven consecutive days for training and the following day for evaluation, with negative down-sampling applied because of data volume and bias.
- Seven consecutive days of Criteo samples are used for training, followed by one evaluation day.
- Negative down-sampling is applied to the Criteo dataset because of its enormous data volume and high bias.
1) Criteo:
The experiments compare seven models and use regularization to control over-fitting, including L2 regularization for linear models and dropout for neural networks.
- Seven models are implemented with TensorFlow and trained using stochastic gradient descent.
- LR is fast and easy to train but cannot capture non-linear information, whereas FM explores feature interactions on sparse data.
- FNN captures high-order latent patterns, while CCPM learns local-global features but relies heavily on feature alignment and lacks interpretation.
- IPNN uses an inner-product layer, OPNN uses an outer-product layer, and PNN* concatenates both product types.
- L2 regularization is used for LR and FM, while dropout regularizes the neural networks during training.
C. Evaluation Metrics
The study evaluates response prediction with AUC, RIG, log loss, and RMSE, then compares model performance and convergence across datasets and training iterations.
- AUC and RIG are the two major evaluation metrics, while log loss and RMSE provide additional measures.
- Every PNN has one embedding layer, one product layer, and three hidden layers, while the compared architectures differ in depth and components.
- A dropout rate of 0.5 is used by default on neural-network hidden layers and is reported as effective in Figure 2.
- PNNs perform best on both Criteo and iPinYou for AUC, while log loss, RMSE, and RIG show similar results.
- T-tests under log loss show that PNNs significantly improve user-response prediction against the baseline models.
- PNN* has no obvious AUC advantage over IPNN and OPNN, suggesting the two individual product variants are sufficient for these interactions.
- On iPinYou, network models converge more quickly than LR and FM, while IPNN and OPNN converge better than other network models.
E. Ablation Study on Network Architecture
The ablation study examines embedding size, network depth, and activation function for IPNN and OPNN, with embedding order affecting memory use and over-fitting.
- The architecture study varies embedding-layer size, network depth, and activation function for IPNN and OPNN.
- The embedding layer converts sparse binary inputs into dense real-value vectors that represent information and relationships.
- Unlike FNN, which initializes embeddings from a pre-trained FM, this paper's PNN approach starts from an embedding layer without pre-training.
- Larger embedding orders are harder to fit in memory and more prone to over-fitting, so neural networks use order 10 in the experiments.
2) Network Depth:
Network depth and activation choices affect PNN performance: three hidden layers generally generalize best, while tanh and relu perform well across tested architectures.
- Network Depth: Three hidden layers generally provide better test-set generalization than networks with 1, 5, or 7 hidden layers.The comparison varies hidden-layer counts in FNN and PNNs.
- Network Depth: Representation layers, including product layers, capture complex feature patterns with fewer parameters and can improve training efficiency and test-set generalization.The paper groups convolution and product layers under this term.
- Network Depth: Figure 4 compares model performance as network depth increases.The supplied caption identifies network depth as the figure’s comparison dimension.
- Activation Function: Figure 5 compares AUC for sigmoid, tanh, and relu across FNN, IPNN, and OPNN.Tanh outperforms sigmoid, and relu also shows good performance.
- Activation Function: Relu’s reported advantages include sparse activation, efficient gradient propagation, and efficient computation.These properties are presented as possible reasons for its good performance on multi-field categorical data.
- Conclusion: PNN uses product layers to investigate feature interactions and reports outperforming state-of-the-art models across four metrics on two datasets.The paper presents inner- and outer-product PNN variants and discusses efficiency and scalability.
- Future Work: Future work will explore more general and complicated product layers and analyze the learned feature-vector representations.The authors also plan to apply these representations to other tasks.