Source-linked AI summary
Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising
Junwei Pan, Jian Xu, Alfonso Lobos Ruiz, Wenliang Zhao, Shengjun Pan, Yu Sun, Quan Lu
TL;DR
CTR prediction uses sparse, multi-field categorical data in which field-dependent interactions matter, but FFMs impose an undesirable parameter burden. The paper proposes FwFMs, which use field-pair weights and an augmented linear representation to model these interactions more memory-efficiently. FwFMs remain competitive with FFMs using as few as 4% of their parameters and perform better at equal parameter counts.
Problem
CTR prediction involves sparse multi-field categorical data whose feature interactions vary across fields, while FFMs' O(mnK) parameter count is undesirable for production systems.
Method
FwFMs learn field-pair interaction weights, and an augmented variant replaces binary linear-term representations with embedding vectors.
Results
4% parameters of FFMs yield competitive prediction performance, while equal-parameter FwFMs achieve up to 0.9% AUC lift over FFMs.
Takeaways & Limitations
FwFMs provide a more memory-efficient alternative to FFMs while retaining competitive performance and modeling different interaction strengths across field pairs.
Takeaways & Limitations
The paper identifies exploring alternatives to one weight per field pair and combining FwFMs with deep learning as future research directions.
Abstract
from arXiv · showhide
Click-through rate (CTR) prediction is a critical task in online display advertising. The data involved in CTR prediction are typically multi-field categorical data, i.e., every feature is categorical and belongs to one and only one field. One of the interesting characteristics of such data is that features from one field often interact differently with features from different other fields. Recently, Field-aware Factorization Machines (FFMs) have been among the best performing models for CTR prediction by explicitly modeling such difference. However, the number of parameters in FFMs is in the order of feature number times field number, which is unacceptable in the real-world production systems. In this paper, we propose Field-weighted Factorization Machines (FwFMs) to model the different feature interactions between different fields in a much more memory-efficient way. Our experimental evaluations show that FwFMs can achieve competitive prediction performance with only as few as 4% parameters of FFMs. When using the same number of parameters, FwFMs can bring 0.92% and 0.47% AUC lift over FFMs on two real CTR prediction data sets.
1 INTRODUCTION
CTR prediction is crucial for display advertising but must handle sparse, multi-field categorical data, heterogeneous field interactions, and production memory constraints. The paper proposes FwFMs to capture field-pair differences more efficiently than FFMs.
- CTR prediction is crucial for delivering ads to the right people, context, and time.
- Multi-field CTR data are sparse categorical features, with millions or tens of millions of features distributed across tens to hundreds of fields.
- Feature conjunctions can associate with clicks differently from individual features, making feature interactions important to model.
- Features from the same field can interact differently with different fields, while production systems must store their parameters for real-time ad serving.
- FwFMs learn a field-pair weight matrix to capture heterogeneous field interactions with far fewer parameters than FFMs.
- 4% parameters of FFMs suffice for competitive performance, while equal-parameter FwFMs deliver up to 0.9% AUC lift over FFMs.
2 PRELIMINARIES
The preliminaries introduce linear, polynomial, and factorization-machine approaches for sparse CTR data, then motivate FFMs as a field-aware extension whose parameter cost is problematic.
- Logistic Regression models CTR using a linear combination of individual feature contributions.
- Poly2 adds feature conjunctions to address CTR tasks where feature interactions are crucial, using hashing to reduce parameter growth from O(m^2).
- FMs learn a K-dimensional embedding for each feature and model pairwise interactions with embedding-vector dot products.
- FMs can estimate interactions in sparse data but neglect that a feature may behave differently across partner fields.
- FFMs explicitly model field-dependent behavior by learning n−1 embedding vectors per feature, but require O(mnK) parameters.
3 INTERACTION STRENGTHS OF FIELD PAIRS
Field-pair interaction strengths vary substantially in multi-field categorical CTR data. The paper quantifies these strengths with mutual information and uses this heterogeneity to motivate a dedicated model.
- Field pairs can have different average interaction strengths across their feature pairs.The paper asks whether field-level interaction strength differs across field pairs.
- ADVERTISER–PUBLISHER interactions are typically strong, whereas HOUR_OF_DAY–DAY_OF_WEEK interactions tend to be weak.
- Mutual information between a field pair and the label quantifies that pair’s interaction strength.
- Oath CTR data show strong pairs such as AD_ID–SUBDOMAIN and CREATIVE_ID–PAGE_TLD, alongside weak pairs such as LAYOUT_ID–GENDER and DAY_OF_WEEK–AD_POSITION_ID.
- Existing models did not account for this field-level heterogeneity, motivating a model that captures different interaction strengths across field pairs.
4 FIELD-WEIGHTED FACTORIZATION MACHINES (FWFMS)
FwFMs extend factorization machines with field-pair weights that explicitly model heterogeneous interactions. This preserves a parameter count close to FMs and substantially below FFMs, with alternative linear-term parameterizations.
- FwFMs weight each feature-pair embedding interaction by the pair of fields containing those features.The weight r_F(i),F(j) represents the interaction strength between the two fields.
- FwFMs explicitly model field-pair interaction strengths, whereas FFMs model them through field-specific embedding vectors for each feature.
- 4.1 Model Complexity: FwFMs use m + mK + n(n − 1)/2 parameters, compared with m + m(n − 1)K for FFMs.Here m is the feature number, n the field number, and K the embedding dimension.
- 4.2 Linear Terms: FwFMs replace scalar linear weights with feature-wise or field-wise embedding-based linear representations as alternative parameterizations.
- 4.2 Linear Terms: Feature-wise and field-wise variants have total parameter counts 2mK + n(n − 1)/2 and nK + mK + n(n − 1)/2, respectively.The two variants are denoted FwFMs_FeLV and FwFMs_FiLV; original linear weights are denoted FwFMs_LW.
5 EXPERIMENTS
The experiments evaluate FwFMs on Criteo and Oath CTR data sets using comparisons with established CTR models and analyses of parameter-matched performance and linear-term variants.
- Experiments compare FwFMs_LW with LR, Poly2, FMs, and FFMs, then examine equal-parameter comparisons and linear-term enhancements.
- 5.1 Data sets: The study uses two CTR data sets: Criteo challenge data and two-week Oath display-advertising click logs.
- 5.1 Data sets: Criteo data are split randomly into training, validation, and test sets at 60%:20%:20%.
- 5.1 Data sets: Oath validation and test sets use the next day and following day, while negative samples are downsampled only for training.
- 5.1 Data sets: Criteo contains 26 anonymous categorical fields, while Oath contains 15 fields spanning user, publisher, advertiser, and context groups.
- 5.1 Data sets: Features appearing fewer than τ times are replaced by a NULL feature, using τ = 20 for Criteo and τ = 10 for Oath.
5.2 Implementations
The TensorFlow implementation represents each sample through active-feature embeddings, computes linear and pairwise interaction terms, and combines them at the output.
- Poly2 uses LibLinear with hashing tricks, while LR, FMs, FFMs, and FwFMs are implemented in TensorFlow.
- Each sparse binary input has one active feature per field and is projected into one embedding vector per active feature.
- The interaction layer computes dot products for every pair of active-feature embeddings.
- The output sums linear and interaction-layer inputs, using feature-wise or field-wise linear weights and field-pair weights where applicable.
5.3 Performance Comparisons
FwFMs outperform simpler CTR models and remain competitive with FFMs while using substantially fewer parameters. Under equal parameter budgets, FwFMs achieve higher test performance than FFMs on both evaluated datasets.
- FwFMs achieve better performance than LR, Poly2, and FMs on both evaluated CTR datasets.
- O(mnK) parameters make FFMs difficult to fit into production memory, motivating more compact alternatives.
- FwFMs use learned field-pair weights to model differing interaction strengths while reducing parameter requirements relative to FFMs.
- 0.70% and 0.45% AUC lift: FwFMs outperform FFMs on the Criteo and Oath test sets with equal parameter counts.
- 0.92% and 0.47% AUC lift: FwFMs_FiLV outperform FFMs on Oath and Criteo, respectively, under equal parameter counts.
5.4 Hyper-parameter Tuning
The tuning experiments examine regularization, learning rate, and embedding dimension for FwFMs. The reported settings favor moderate regularization, a small learning rate, and K = 10 as a performance–training-time trade-off.
- FwFMs_FiLV tuning on the Oath validation set varies regularization coefficient λ, embedding dimension K, and learning rate η.
- λ = 1e −5 gives the best validation performance among the tested regularization settings.
- Small η improves validation performance slowly during the first 20 epochs, whereas large η improves quickly and then over-fits.
- η = 1e −4 is selected for all experiments on the Oath dataset.
- K = 10 is chosen because it provides the best trade-off between performance and training time, despite only small AUC changes across dimensions.
6 STUDY OF LEARNED FIELD INTERACTION STRENGTHS
FwFMs learn field-pair interaction strengths that more closely reflect mutual information than those learned by FMs or FFMs. The field-pair weight matrix is central to this alignment.
- FwFMs model field-pair interaction strengths using a learned weight r_Fk,F_l, unlike models that do not explicitly encode this heterogeneity.
- The evaluation compares learned field interaction strengths with mutual information using heatmaps and Pearson correlation.
- −0.1210 Pearson correlation: FM interaction strengths poorly match mutual information because FMs ignore field information.
- 0.1544 Pearson correlation: FFMs align more closely with mutual information than FMs do.
- FwFM heatmaps assign higher strengths to field pairs with high mutual information and lower strengths to weakly associated pairs.
- 0.5554 Pearson correlation: the FwFM field-pair weight matrix aligns with mutual information, while learned strengths without it correlate poorly.
7 RELATED WORK
The related-work discussion situates CTR prediction among linear, polynomial, tree-based, tensor, Bayesian, factorization, and deep-learning approaches. It emphasizes FMs and FFMs as closely related models for sparse categorical data.
- CTR prediction research includes LR, Poly2, tree-based, tensor-based, Bayesian, FFM, and deep-learning models.
- FMs are closely related to recommendation methods and have been reported to outperform matrix factorization in many recommendation tasks.
8 CONCLUSION
The paper proposes FwFMs for CTR prediction and finds them competitive with FFMs while using significantly fewer parameters. It also reports improved performance at equal parameter budgets and verifies that FwFMs learn differing interaction strengths across field pairs.
- FwFMs are proposed for CTR prediction in online display advertising.
- FwFMs achieve competitive performance with significantly fewer parameters than FFMs.
- With the same number of parameters, FwFMs consistently outperform FFMs.
- A novel linear term representation further improves FwFM performance.
- Analysis on real-world data sets verifies that FwFMs learn different feature interaction strengths across field pairs.