Source-linked AI summary
Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction
Weinan Zhang, Tianming Du, Jun Wang
TL;DR
User-response prediction over multi-field categorical data is important, but conventional linear models miss interactions while manual combinations are costly. The paper proposes FNN and SNN deep networks with FM, RBM, and DAE-based transformations, and reports superior CTR estimation on real-world data. The authors conclude that these architectures can efficiently learn underlying data patterns and support future extensions.
Problem
User-response prediction uses large multi-field categorical feature spaces whose unknown interactions are difficult for linear models and expensive to engineer manually.
Method
The paper proposes FNN with supervised factorisation-machine embedding and SNN with sampling-based RBM or DAE unsupervised pre-training for deep prediction.
Results
The proposed models provide superior CTR estimation performance to compared models on multiple real-world advertiser datasets.
Takeaways & Limitations
The architectures and pre-training algorithms enable efficient DNN training while learning underlying categorical-data patterns.
Takeaways & Limitations
The study does not consider advanced Bayesian methods for hyperparameter tuning, which may be investigated in future work.
Abstract
from arXiv · showhide
Predicting user responses, such as click-through rate and conversion rate, are critical in many web applications including web search, personalised recommendation, and online advertising. Different from continuous raw features that we usually found in the image and audio domains, the input features in web space are always of multi-field and are mostly discrete and categorical while their dependencies are little known. Major user response prediction models have to either limit themselves to linear models or require manually building up high-order combination features. The former loses the ability of exploring feature interactions, while the latter results in a heavy computation in the large feature space. To tackle the issue, we propose two novel models using deep neural networks (DNNs) to automatically learn effective patterns from categorical feature interactions and make predictions of users' ad clicks. To get our DNNs efficiently work, we propose to leverage three feature transformation methods, i.e., factorisation machines (FMs), restricted Boltzmann machines (RBMs) and denoising auto-encoders (DAEs). This paper presents the structure of our models and their efficient training algorithms. The large-scale experiments with real-world data demonstrate that our methods work better than major state-of-the-art models.
1 Introduction
User-response prediction is central to web applications, but categorical, multi-field inputs make feature interactions difficult for conventional models. The paper introduces two DNN approaches using embedding and unsupervised or supervised transformations, reporting consistent improvement over state-of-the-art methods.
- User-response prediction supports web search, recommendation, sponsored search, and display advertising, including ad click-through-rate estimation.
- Linear models efficiently handle sparse one-hot features but have relatively low performance because they fail to learn feature interactions.
- Nonlinear models can explore feature combinations, but existing approaches cannot use all possible combinations and often require manual feature engineering.
- Multi-field categorical inputs make direct deep learning challenging because their dependencies are largely unknown and differ from continuous image or audio features.
- The proposed FNN and SNN models use factorisation machines, RBMs, and DAEs to transform sparse categorical features into representations for deep networks.
- Experiments on multiple real-world advertisers’ ad-click data demonstrate consistent CTR-estimation improvement over state-of-the-art models.
2 Related Work
Related work spans sparse linear CTR models, embedding methods such as factorisation machines, and deep learning with unsupervised pre-training. These approaches motivate learning compact representations and hierarchical patterns from categorical inputs.
- Most online advertising models use logistic regression over sparse binary features produced by one-hot encoding of categorical inputs.
- Embedding large feature vectors into low-dimensional spaces reduces data and model complexity while improving training and prediction efficiency.
- Factorisation machines estimate pairwise feature interactions by mapping features into vectors in a low-rank latent space.
- Deep neural networks extract hidden structures at multiple abstraction levels, while unsupervised pre-training can provide representations that support better generalisation.
- Typical deep-learning training initializes representations with unsupervised learning using RBMs or stacked denoising auto-encoders, then fine-tunes them with supervised back-propagation.
3 DNNs for CTR Estimation given Categorical Features
The paper presents FNN and SNN architectures that transform sparse, field-wise categorical inputs into dense representations for CTR estimation. FNN uses an FM bottom layer, while SNN uses a fully connected bottom layer initialized through sampled RBM or DAE pre-training.
- 3 DNNs for CTR Estimation given Categorical Features: The input categorical features are encoded field-wise with one positive unit per field, forming the sparse vector supplied to the models.For example, a city field has one active value such as city=London and inactive units for other values.
- 3.1 Factorisation-machine supported Neural Networks (FNN): The architectures use logistic output and hidden-layer activations, with tanh selected because it showed optimal empirical learning performance among the tested activation functions.The output is interpreted as predicted CTR, while the hidden-layer design includes parameters for successive transformations.
- 3.1 Factorisation-machine supported Neural Networks (FNN): FNN uses a factorisation machine as its bottom layer to transform sparse categorical inputs into representations for subsequent neural layers.FM models feature interactions in a low-rank latent space, reducing the computational burden of high-dimensional binary inputs.
- 3.2 Sampling-based Neural Networks (SNN): SNN differs from FNN through a fully connected sigmoid bottom layer whose weights are initialized with sampling-based RBM or DAE pre-training.The sampling procedure retains each field’s positive feature and samples negative units, reducing the cost of unsupervised pre-training.
- 3.2 Sampling-based Neural Networks (SNN): The sampled unsupervised models produce a real-valued dense vector that serves as input to SNN’s further layers, after which back-propagation fine-tunes the network.Sampling unsampled one-hot units reduces computational complexity while retaining high recovery performance.
- 3.3 Regularisation: Dropout is implemented alongside L2 regularisation to address overfitting in the deep networks.The experiments compare these regularisation approaches, with dropout applied by stochastically excluding hidden units during training.
4 Experiment
Experiments on the iPinYou display-ad dataset evaluate CTR estimation with AUC across proposed models, baselines, architectures, and regularisation settings. The proposed models consistently improve over LR and FM, while performance depends on architecture and dropout tuning.
- Experiment Setup: The iPinYou dataset contains 19.50M ad-display instances with 14.79K clicks and categorical features such as user, ad, and contextual attributes.Experiments use five advertisers and the whole dataset; AUC measures CTR estimation performance.
- Performance Comparison: FNN and SNN outperform LR and FM on all tested datasets, while SNN-RBM and SNN-DAE show generally consistent relative performance.FNN further learns patterns from FM latent features, producing consistent improvement over FM.
- Hyperparameter Tuning: Hyperparameter tuning selects m = 2 negative samples per field and tanh hidden-layer activations for SNN-RBM and SNN-DAE in most situations.The authors report that tanh is optimal in these comparisons and might converge faster than sigmoid.
- Architecture Selection: Three hidden layers, five layers in total, achieve the best AUC among the evaluated 3-, 4-, and 5-hidden-layer architectures.The comparison fixes all layer sizes while varying the number of hidden layers.
- Architecture Selection: Diamond-shaped networks outperform the other compared shapes in almost all layer-size settings, with performance peaking at a total hidden-unit size of 600.The authors suggest the shape may constrain network capacity and improve test-set generalisation.
- Regularisation Comparison: Dropout outperforms L2 regularisation for SNN-RBM across all compared settings, and dropout changes generalisation performance in both FNN and SNN.SNN is more sensitive to dropout than FNN; optimal dropout rates are around 0.8 for FNN and about 0.99 for SNN.
5 Conclusion
The paper concludes that two DNN architectures with field-wise embedding and unsupervised or supervised pre-training efficiently handle multi-field categorical ad-click prediction. Experiments report that these models learn underlying patterns and achieve superior CTR estimation performance.
- The paper proposes field-wise feature embedding with supervised factorisation-machine pre-training and fully connected DNNs with field-wise RBM or DAE pre-training.
- These architectures and pre-training algorithms make the DNNs train very efficiently.
- Experiments on a public real-world dataset show that the proposed models learn underlying data patterns and provide superior CTR estimation performance over compared models.
- The paper identifies momentum methods and extending partial connections to higher hidden layers as future directions.