AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction

Bin Liu; Chenxu Zhu; Guilin Li; Weinan Zhang; Jincai Lai; Ruiming Tang; Xiuqiang He; Zhenguo Li; Yong Yu

Source-linked AI summary

AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction

Bin Liu, Chenxu Zhu, Guilin Li, Weinan Zhang, Jincai Lai, Ruiming Tang, Xiuqiang He, Zhenguo Li, Yong Yu

arXiv:2003.11235v3Mar 25, 2020cs.LGcs.IRstat.ML

TL;DR

CTR models need useful feature interactions, yet enumerating all interactions is expensive and useless ones can add noise. AutoFIS addresses this with a two-stage architecture-parameter search and re-training procedure, improving factorization models offline and DeepFM in an online test.

Problem
Existing CTR models either enumerate feature interactions, incurring memory and computation costs and potentially adding noise, or require costly manual selection.
Method
AutoFIS learns interaction importance with continuous architecture parameters and a regularized optimizer, removes redundant interactions, then re-trains while retaining them as attention units.
Results
20.3% CTR and 20.1% CVR improvements over DeepFM were achieved in a ten-day Huawei App Store online A/B test.
Takeaways & Limitations
AutoFIS can select important low- and high-order interactions for factorization models and transfer selected interactions to other deep CTR models.
Takeaways & Limitations
Tree-based interaction methods are less suitable for high-dimensional categorical recommendation data because of low categorical-feature usage.

Abstract

from arXiv · show

Learning feature interactions is crucial for click-through rate (CTR) prediction in recommender systems. In most existing deep learning models, feature interactions are either manually designed or simply enumerated. However, enumerating all feature interactions brings large memory and computation cost. Even worse, useless interactions may introduce noise and complicate the training process. In this work, we propose a two-stage algorithm called Automatic Feature Interaction Selection (AutoFIS). AutoFIS can automatically identify important feature interactions for factorization models with computational cost just equivalent to training the target model to convergence. In the \emph{search stage}, instead of searching over a discrete set of candidate feature interactions, we relax the choices to be continuous by introducing the architecture parameters. By implementing a regularized optimizer over the architecture parameters, the model can automatically identify and remove the redundant feature interactions during the training process of the model. In the \emph{re-train stage}, we keep the architecture parameters serving as an attention unit to further boost the performance. Offline experiments on three large-scale datasets (two public benchmarks, one private) demonstrate that AutoFIS can significantly improve various FM based models. AutoFIS has been deployed onto the training platform of Huawei App Store recommendation service, where a 10-day online A/B test demonstrated that AutoFIS improved the DeepFM model by 20.3\% and 20.1\% in terms of CTR and CVR respectively.

1 INTRODUCTION

CTR prediction depends on identifying useful low- and high-order feature interactions, but existing approaches either enumerate too many interactions or rely on costly manual selection. AutoFIS automatically selects interactions through a two-stage factorization-model procedure and improves efficiency and prediction performance.

Enumerating feature interactions increases memory and computation costs, while useless interactions can add noise and complicate training.
AutoFIS: AutoFIS uses continuous architecture parameters and a regularized optimizer to learn each interaction's importance during one full training process.
Results: AutoFIS removes about 50%-80% of 2nd-order feature interactions, reducing the efficiency burden of the original models.
Results: 20.3% CTR and 20.1% CVR improvements over DeepFM were observed in a ten-day Huawei App Store online A/B test.
Results: Offline experiments on three large-scale datasets show improved factorization-model performance and useful high-order interactions transferable to other models.
AutoFIS: The two-stage method removes unimportant interactions, then re-trains the model while retaining architecture parameters as attention units.

2 RELATED WORK

Related work spans factorization, deep-learning, tree-based, and neural architecture-search approaches for modeling or discovering feature interactions. These methods still face restricted exploration, costly search, or difficulty identifying and removing useless interactions.

Factorization models: FM models second-order interactions with bilinear learning, while FFM gives features multiple representations for interactions across fields.
Deep models: Deep models such as DeepFM and PNN improve representational capacity, but their architectures differ in how interaction and MLP layers are arranged or combined.
Tree-based methods: GBDT transforms tree-discovered interactions for downstream logistic-regression or FFM models, while tree models are less suitable for high-dimensional categorical data.
AutoML methods: AutoCross reduces its enormous candidate space through beam search and field-aware logistic-regression evaluation, but these approximations may yield interactions unsuitable for deep models.
Neural architecture search: DARTS-style one-shot architecture search has been applied to interaction functions, whereas AutoFIS searches for important feature interactions themselves.

3 METHODOLOGY

AutoFIS selects useful low- and high-order feature interactions in factorization models through a two-stage architecture-parameter procedure. It replaces exhaustive interaction handling with learned selection, then retains selected interactions’ relative weights during retraining.

Factorization Models: Factorization models represent interactions among feature embeddings, with FM, DeepFM, and IPNN differing in how interaction and MLP layers are connected.DeepFM uses parallel interaction and MLP layers, whereas IPNN places the MLP layer after the interaction layer.
Feature Interaction Layer: Pairwise interaction layers pass all feature interactions forward equally, although useless interactions may degrade performance.The number of pairwise interactions is C(m,2), creating a large candidate set as the number of fields grows.
AutoFIS: AutoFIS divides interaction selection into search and re-train stages, detecting useful interactions first and retraining the reduced model afterward.The method is designed to apply to the feature interaction layer of factorization models.
Search Stage: In the search stage, continuous architecture parameters learn each interaction’s relative contribution by gradient descent instead of enumerating discrete gate configurations.Interactions with zero architecture parameters can then be assigned closed gates and removed.
Search Stage: Batch normalization stabilizes architecture-parameter estimation by decoupling the learned scale of an interaction from the interaction value itself.Without normalization, jointly learned interaction values and architecture parameters can have coupled scales that obscure relative importance.
Search Stage: AutoFIS jointly optimizes architecture parameters and model weights with one-level gradient optimization, avoiding DARTS-style weight approximation.The method argues that approximation error from one-step or bi-level optimization might downgrade performance.

4 EXPERIMENTS

The experiments evaluate AutoFIS through offline studies on Avazu, Criteo, and a private Huawei App Store dataset, alongside online testing. They examine performance, transferability, interaction usefulness, live-system impact, and component effects.

Research Questions: The experiments address whether AutoFIS improves factorization models, transfers selected interactions, identifies useful interactions, improves live recommendations, and benefits from its components.The listed research questions include inference-time reduction, prediction accuracy, interaction usefulness, online impact, and component analysis.
Datasets: Offline experiments use two public datasets, Avazu and Criteo, plus a private Huawei App Store recommendation dataset.The private data includes app, user, and context features.
Datasets: Avazu uses 80% of randomly shuffled data for training and validation and 20% for testing, with rare categories removed below 20 appearances.This preprocessing reduces dimensionality before evaluation.
Evaluation Metrics: Table 1 reports inference time for 2 million samples, retained interaction percentages, search and retraining GPU cost, and relative AUC improvement over FM.The caption notes that FFM’s lower time and cost reflect a smaller embedding size imposed by GPU memory constraints.
Datasets: Criteo uses data 6–12 for training and validation and day 13 for evaluation, applies negative down-sampling, and buckets 13 numerical fields.Features appearing fewer than 20 times in a field are assigned to an “other” dummy feature.

4.2 Experimental Settings

The experiments apply AutoFIS to factorization models across three datasets, using staged interaction selection and retraining. Results report accuracy, efficiency, and search-cost improvements for second- and third-order interactions.

Experimental setup: AutoFM and AutoDeepFM are evaluated against GBDT-based and factorization-machine baselines on Avazu, Criteo, and a private dataset.The evaluation uses AUC and Log loss as common CTR-prediction metrics.
Interaction-selection procedure: The search stage jointly trains architecture parameters and interaction embeddings before removing useless interactions and retraining the model.For third-order selection, the method reuses selected second-order interactions, searches third-order interactions, and retrains with both orders.
Efficiency: AutoDeepFM(3rd) searches important second- and third-order interactions in 24 minutes on Avazu and 128 minutes on Criteo using one GPU.Directly enumerating third-order interactions increases FM and DeepFM inference time about 7 to 12 times.

4.4 Transferability of the Selected Feature Interactions (RQ2)

The selected interactions transfer from the simple AutoFM model to IPNN. Using these selections preserves comparable second-order performance and significantly improves performance when third-order interactions are included.

Transferability: AutoIPNN(2nd) uses AutoFM-selected second-order interactions and achieves performance comparable to IPNN with around 30% and 50% of interactions on Avazu and Criteo.AutoIPNN(3rd) adds AutoFM-selected second- and third-order interactions.
Transferability: AutoIPNN(3rd) significantly improves performance over IPNN using selected second- and third-order interactions.The results support transferability of interactions selected by AutoFM to a state-of-the-art model.

4.5 The Effectiveness of Feature Interaction Selected by AutoFIS (RQ3)

AutoFIS-selected interactions are evaluated on real and synthetic data using statistical importance comparisons and controlled generation. The results indicate that selected interactions align with predictive importance and outperform a statistics_AUC-based selection under equal computational cost.

Real-data analysis: statistics_AUC measures an interaction’s importance by the AUC of a predictor using only that interaction’s training-set statistical CTR.Higher statistics_AUC indicates a more important role in prediction.
Real-data analysis: Most interactions with high absolute α have high statistics_AUC, but some high-statistics_AUC interactions are not selected because their information may exist elsewhere.The comparison links AutoFIS selection to predictive importance while allowing redundancy among interactions.
Real-data analysis: At equal computational cost, AutoFIS-selected interactions produce much better performance than selecting the same number of interactions by statistics_AUC.The comparison is conducted by retraining with the top-N statistics_AUC interactions.
Synthetic-data analysis: The synthetic dataset is generated from an incomplete poly-2 function whose selected bilinear terms represent important category interactions.The experiments compare FM and AutoFIS using test-set AUC and assess whether the known interactions are recovered.
Synthetic-data analysis: With m = 6 and N = 60, AutoFIS extracts the predefined interactions precisely, assigning them the highest α values while removing some unimportant interactions.The selected set is C = {(x0,x1), (x2,x5), (x3,x4)}.

4.6 Deployment & Online Experiments (RQ4)

AutoDeepFM was evaluated in a ten-day Huawei App Store A/B test against DeepFM, with the model launched on Day 8. The experiment measured experimental-over-control changes in CTR and CVR in a game recommendation scenario.

Online experiment: A ten-day A/B test compared AutoDeepFM with DeepFM in Huawei App Store game recommendations.Five percent of users received each system’s recommendations, with DeepFM serving as the control and AutoDeepFM as the experimental model.
Deployment constraints: The deployed ranking system was required to keep overall candidate-selection and ranking latency within a few milliseconds.Deployment used a three-node cluster with Intel Xeon CPUs, 400GB RAM per node, and two NVIDIA Tesla V100 GPUs per node.
Online experiment: Day 8 marked AutoDeepFM’s launch to the live system.Before launch, CTR and CVR fluctuated within 8% during A/A testing.

4.7 Ablation Study (RQ5)

The ablation study evaluates AutoFIS’s search and re-train components, showing that learned interaction selection outperforms random selection and that batch normalization and α improve re-training.

Stability: The learned α estimates had a Pearson correlation of about 0.86 across random seeds on Avazu.This indicates stable interaction-importance estimation across initializations.
Search stage: AutoFIS selection consistently outperformed random selection when retaining the same number of feature interactions.This supports the search stage’s ability to identify important interactions rather than merely reduce their number.
Search stage: Random selection sometimes outperformed FM retaining all interactions on Criteo, indicating that removing useless interactions can improve performance.The comparison used fewer interactions in the random-selection variant.
Re-train stage: Batch normalization improved AutoFM performance in the re-train stage.The AutoFM versus AutoFM-BN comparison isolates the contribution of BN.
Re-train stage: α improved re-training by differentiating the contributions of different feature interactions.This conclusion follows from the performance gap between AutoFM-BN and AutoFM-BN-α.
Optimization: One-level optimization outperformed bi-level optimization for both AutoFM and AutoDeepFM.The performance gaps between each one-level model and its bi-level counterpart support the one-level design.

5 CONCLUSION

AutoFIS automatically selects important second- and third-order feature interactions for factorization models and transfers the selected interactions to other deep CTR models. It is designed for marginal search cost and was deployed in Huawei App Store recommendation training.

Conclusion: AutoFIS automatically selects important 2nd- and 3rd-order feature interactions.The method is presented as generally applicable to factorization models.
Conclusion: Selected interactions can be transferred from factorization models to other deep learning models for CTR prediction.This allows a simple model to perform interaction selection for broader model families.
Conclusion: AutoFIS has marginal search costs and achieved significant improvement on two benchmark datasets and one private dataset.The method was also deployed onto Huawei App Store’s recommendation-service training platform.

A PARAMETER SETTINGS

The appendix lists baseline and AutoFIS-specific hyperparameters, optimizer settings, and notation for regularization and normalization components.

Parameter settings: AutoFM and AutoDeepFM use the corresponding FM and DeepFM hyperparameters, with additional settings for AutoFIS components.The reported baseline configurations for Avazu and Criteo follow prior work.
Parameter settings: The parameter table defines settings for batch size, optimizer, learning rate, embedding size, α initialization, regularization, temperature, and network structure.It also records normalization, GRDA optimizer, and attention-network parameters.