Source-linked AI summary
DeepFM: A Factorization-Machine based Neural Network for CTR Prediction
Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, Xiuqiang He
TL;DR
CTR prediction must capture sophisticated low- and high-order feature interactions without relying on exhaustive expert feature engineering. DeepFM jointly combines factorization machines and deep neural networks using shared inputs and embeddings, outperforming state-of-the-art models in AUC and Logloss on two datasets with comparable efficiency to the most efficient deep model.
Problem
Existing CTR models are biased toward low- or high-order feature interactions or depend on feature engineering, while many interactions remain hidden and difficult to identify exhaustively.
Method
DeepFM jointly trains factorization-machine and deep components with shared inputs and embeddings to model low- and high-order feature interactions end to end.
Results
DeepFM outperforms state-of-the-art models in AUC and Logloss on both datasets, with efficiency comparable to the most efficient deep model.
Takeaways & Limitations
DeepFM provides a unified CTR prediction model that learns low- and high-order feature interactions without feature engineering beyond raw features.
Takeaways & Limitations
Future work includes strengthening learning of useful high-order interactions and training DeepFM on GPU clusters for large-scale problems.
Abstract
from arXiv · showhide
Learning sophisticated feature interactions behind user behaviors is critical in maximizing CTR for recommender systems. Despite great progress, existing methods seem to have a strong bias towards low- or high-order interactions, or require expertise feature engineering. In this paper, we show that it is possible to derive an end-to-end learning model that emphasizes both low- and high-order feature interactions. The proposed model, DeepFM, combines the power of factorization machines for recommendation and deep learning for feature learning in a new neural network architecture. Compared to the latest Wide \& Deep model from Google, DeepFM has a shared input to its "wide" and "deep" parts, with no need of feature engineering besides raw features. Comprehensive experiments are conducted to demonstrate the effectiveness and efficiency of DeepFM over the existing models for CTR prediction, on both benchmark data and commercial data.
1 Introduction
CTR prediction requires learning sophisticated low- and high-order feature interactions behind user behavior, but existing approaches are biased toward particular interaction orders or depend on feature engineering. DeepFM addresses this by combining FM and DNN architectures to learn all interaction orders end-to-end from raw features, with shared inputs and embeddings for efficient training.
- CTR prediction ranks recommended items by estimated click probability, while advertising may rank candidates by CTR×bid to improve revenue.
- Feature interactions behind clicks can be sophisticated, including order-2 category–time interactions and order-3 category–gender–age interactions.
- Hidden interactions are difficult to identify a priori, so machine learning must capture them automatically; manual interactions also generalize poorly to high-order or rare patterns.
- DeepFM integrates FM and DNN architectures to model low-order and high-order feature interactions, respectively, without feature engineering beyond raw features.
- DeepFM trains efficiently because its wide and deep parts share the same input and embedding vector, avoiding the huge manually expanded input used by Wide & Deep.
- Experiments on benchmark and commercial data show consistent improvement over existing CTR-prediction models.
2 Our Approach
DeepFM jointly learns low- and high-order feature interactions through FM and deep components that share feature embeddings and raw input features. This shared architecture avoids pre-training and expertise feature engineering while modeling representations through both interaction orders.
- DeepFM Architecture: DeepFM combines an FM component for low-order interactions with a deep component for high-order interactions, using the same input.FM models linear and pairwise interactions, while the feed-forward neural network learns high-order interactions.
- FM Component: FM represents order-1 feature importance with scalar weights and order-2 interactions through inner products of latent feature vectors.The latent vectors can be trained whenever their corresponding features appear, improving interaction learning for sparse datasets.
- Embedding and Deep Component: DeepFM maps each field to an embedding of shared size k, using FM latent vectors as learned network weights to compress field inputs.The resulting embedding representation is fed into the deep neural network, whose layers generate a dense feature vector for CTR prediction.
- Shared Feature Embeddings: Sharing feature embeddings lets DeepFM learn low- and high-order interactions directly from raw features without expertise feature engineering.Unlike FM-initialized FNN, DeepFM includes the FM model within the overall architecture rather than using FM only for initialization.
- Comparison with Existing Models: DeepFM’s embedding-sharing strategy jointly shapes feature representations through low- and high-order interactions, modeling them more precisely than replacing LR with FM alone.The comparison summary identifies DeepFM as the only model requiring neither pre-training nor feature engineering while capturing both interaction orders.
3 Experiments
Experiments compare DeepFM with state-of-the-art models, finding it more effective than the alternatives while maintaining efficiency comparable to the best-performing models.
- 3 Experiments: DeepFM is more effective than the other state-of-the-art models in empirical evaluation.
- 3 Experiments: DeepFM’s efficiency is comparable to that of the best competing models.
3.1 Experiment Setup
The experiments evaluate DeepFM on the Criteo and Company∗ datasets using AUC and Logloss, comparing it with nine CTR prediction models. The setup uses shared benchmark settings for Criteo and parameter studies for Company∗.
- Datasets: Experiments use the Criteo dataset with 45 million users’ click records, 13 continuous features, 26 categorical features, and a 90%/10% train-test split.The Company∗ dataset is used to assess DeepFM in real industrial CTR prediction, with 7 consecutive
- Evaluation Metrics: Model evaluation uses AUC (Area Under ROC) and Logloss (cross entropy).
- Compared Models: The comparison includes LR, FM, FNN, three PNN variants, LR & DNN, FM & DNN, Wide & Deep, and DeepFM.FM & DNN replaces LR with FM in the wide part to reduce feature engineering effort.
3.2 Performance Evaluation
DeepFM achieves nearly the highest efficiency in CPU and GPU tests while delivering the strongest CTR prediction performance. Its gains arise from learning low- and high-order interactions jointly with shared feature embeddings.
- Efficiency: DeepFM achieves almost the most efficient performance in both CPU and GPU efficiency tests.Pre-training makes FNN less efficient, while IPNN and PNN∗ remain computationally expensive because of inefficient inner product operations.
- CTR Prediction: DeepFM outperforms LR by 0.86% and 4.18% in AUC on Company∗ and Criteo, respectively, and by 1.15% and 5.60% in Logloss.LR is the only evaluated model that does not consider feature interactions.
- CTR Prediction: DeepFM beats models learning only low-order or only high-order interactions by more than 0.37% and 0.25% in AUC on Company∗ and Criteo, respectively.The corresponding Logloss gains are 0.42% and 0.29%.
- CTR Prediction: DeepFM outperforms separate-embedding LR & DNN and FM & DNN models by more than 0.48% and 0.33% in AUC on Company∗ and Criteo, respectively.The corresponding Logloss gains are 0.61% and 0.66%.
- CTR Prediction: DeepFM beats competitors by more than 0.37% in AUC and 0.42% in Logloss on Company∗ dataset.The paper notes that small offline AUC improvements can lead to significant online CTR increases.
3.3 Hyper-Parameter Study
The hyper-parameter study finds that suitable dropout improves robustness, while excessive model complexity through wider or deeper networks can cause overfitting. Among tested activation functions and network shapes, ReLU generally outperforms tanh and the constant shape performs best.
- Activation functions: ReLU is more appropriate than tanh for all tested deep models except IPNN, possibly because ReLU induces sparsity.
- Dropout rate: All models reach their best performance with dropout properly set between 0.6 and 0.9, indicating that reasonable randomness strengthens robustness.Dropout values tested were 1.0, 0.9, 0.8, 0.7, 0.6, and 0.5.
- Neurons per layer: Increasing neurons per layer does not always help: DeepFM remains stable from 400 to 800 neurons, while OPNN performs worse over that range because over-complexity promotes overfitting.
- Hidden layers: Increasing hidden layers initially improves model performance, but continued increases degrade it because of overfitting.
4 Related Work
The related work spans CTR prediction and deep learning in recommender systems. CTR methods include generalized linear models, factorization machines, and several alternative model families, while deep learning has also been applied to recommendation tasks beyond CTR prediction.
- Related domains: The paper situates its contribution at the intersection of CTR prediction and deep learning in recommender systems.These are identified as the two most related domains.
- CTR prediction: CTR prediction is important in recommender systems and has been addressed with generalized linear models, factorization machines, tree-based models, tensor-based models, support vector machines, and Bayesian models.The cited alternatives represent several established model families for CTR prediction.
- Deep learning in recommender systems: Deep learning has also been applied to recommendation tasks beyond CTR prediction, including several other recommendation settings.The paper notes that deep CTR models were discussed earlier and therefore omits them from this section.
5 Conclusions
DeepFM jointly trains factorization-machine and deep components to learn low- and high-order feature interactions without pre-training or feature engineering. Experiments show superior AUC and Logloss performance with efficiency comparable to the most efficient deep model, while future work targets stronger high-order interaction learning and GPU-cluster training.
- DeepFM jointly trains deep and FM components, learning both high- and low-order feature interactions without pre-training or feature engineering.It uses shared feature embeddings to avoid feature engineering.
- DeepFM outperforms state-of-the-art models in AUC and Logloss on both datasets.
- DeepFM’s efficiency is comparable to the most efficient deep model among the state-of-the-art models.
- Future work will strengthen learning of useful high-order feature interactions and train DeepFM on GPU clusters for large-scale problems.Suggested strategies include introducing pooling layers.