PaperlayerView exact revision on arXiv

Source-linked AI summary

FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction

Tongwen Huang, Zhiqi Zhang, Junlin Zhang

arXiv:1905.09433v1cs.LGcs.AIstat.ML

TL;DR

CTR prediction supports advertising and feed ranking, but many existing models use simple feature interactions and pay limited attention to feature importance. FiBiNET combines dynamic SENET-based feature weighting with bilinear interactions, and its shallow and deep variants outperform competing models across the Criteo and Avazu datasets.

  • Problem

    Many CTR models calculate feature interactions with simple Hadamard or inner products and pay less attention to feature importance.

  • Method

    FiBiNET uses SENET to dynamically weight features and three bilinear-interaction layer types to model feature interactions finely.

  • Results

    FiBiNET’s shallow model outperforms FM, FFM, and other shallow models, while deep FiBiNET consistently outperforms state-of-the-art deep models on Criteo and Avazu.

  • Takeaways & Limitations

    Combining SENET with bilinear feature interaction is an effective approach for CTR prediction across the reported real-world datasets.

Abstract

from arXiv · show

Advertising and feed ranking are essential to many Internet companies such as Facebook and Sina Weibo. Among many real-world advertising and feed ranking systems, click through rate (CTR) prediction plays a central role. There are many proposed models in this field such as logistic regression, tree based models, factorization machine based models and deep learning based CTR models. However, many current works calculate the feature interactions in a simple way such as Hadamard product and inner product and they care less about the importance of features. In this paper, a new model named FiBiNET as an abbreviation for Feature Importance and Bilinear feature Interaction NETwork is proposed to dynamically learn the feature importance and fine-grained feature interactions. On the one hand, the FiBiNET can dynamically learn the importance of features via the Squeeze-Excitation network (SENET) mechanism; on the other hand, it is able to effectively learn the feature interactions via bilinear function. We conduct extensive experiments on two real-world datasets and show that our shallow model outperforms other shallow models such as factorization machine(FM) and field-aware factorization machine(FFM). In order to improve performance further, we combine a classical deep neural network(DNN) component with the shallow model to be a deep model. The deep FiBiNET consistently outperforms the other state-of-the-art deep models such as DeepFM and extreme deep factorization machine(XdeepFM).

1 INTRODUCTION

CTR prediction is central to advertising and feed ranking, motivating many classical and deep learning models. FiBiNET addresses limited feature-importance modeling and simple interaction functions with SENET and bilinear interactions.

  • CTR prediction underpins advertising and feed-ranking systems used by Internet companies.
  • Existing CTR approaches include logistic regression, tree-based, tensor-based, Bayesian, factorization-machine, and deep-learning models.
  • FiBiNET dynamically learns feature importance and fine-grained feature interactions using SENET and bilinear functions.
  • The shallow model outperforms FM, FFM, and other shallow models on the Criteo and Avazu datasets.
  • Adding a DNN component produces a deep FiBiNET that consistently outperforms state-of-the-art deep models, including DeepFM and XDeepFM.

2 RELATED WORK

Related CTR work spans factorization machines, neural models, explicit interaction architectures, and attention mechanisms. SENET provides FiBiNET with a feature-reweighting mechanism originally developed for convolutional image features.

  • FM models feature interactions with factorized parameters, while FFM uses field-aware latent vectors and achieved success in Criteo and Avazu competitions.
  • FFM requires substantial memory and is therefore difficult to deploy in Internet companies.
  • Deep CTR models include FNN, WDL, DeepFM, DCN, and XDeepFM, which target memorization, generalization, or explicit feature interactions.
  • DeepFM replaces WDL’s manually engineered wide component with FM and shares feature embeddings between its FM and deep components.
  • AFM uses an attention network to learn weights for feature interactions because FM assigns all interactions the same weight.
  • SENET was introduced to model channel interdependencies in convolutional features and achieved first place in the ILSVRC 2017 classification task.

3 OUR PROPOSED MODEL

FiBiNET dynamically learns feature importance with a SENET mechanism and models fine-grained second-order feature interactions through bilinear functions. Its architecture combines sparse embeddings, SENET reweighting, interaction layers, feature combination, and optional deep neural layers.

  • Model Overview: FiBiNET targets dynamically learning feature importance and fine-grained feature interactions for CTR prediction.The model combines a SENET mechanism for feature weighting with bilinear interaction functions.
  • Model Architecture: The model processes sparse inputs through embeddings, SENET, Bilinear-Interaction, combination, hidden, and output layers.The sparse input and embedding layers convert raw features into dense vectors before subsequent modeling stages.
  • SENET Layer: SENET squeezes each field embedding into a statistic vector, uses two fully connected layers to learn field weights, and rescales the embeddings.The squeeze step can use max or mean pooling; the re-weight step multiplies each field embedding by its learned scalar weight.
  • SENET Layer: Mean pooling performs better than max pooling in the authors’ experiments, although max pooling is used in the original SENET.The comparison concerns the pooling operation used during SENET’s squeeze step.
  • Bilinear-Interaction Layer: The Bilinear-Interaction layer extends inner and Hadamard products with additional parameters to model second-order interactions more finely.It produces vector interactions from both the original embedding and the SENET-like embedding.
  • Bilinear-Interaction Layer: FiBiNET offers Field-All, Field-Each, and Field-Interaction bilinear functions, differing in whether matrices are shared globally, by field, or by field pair.Their parameter counts are k × k, f × k × k, and n × k × k, respectively, with n = f(f − 1)/2 for field interactions.
  • Combination and Deep Layers: The combination layer concatenates the original and SENET-like interaction vectors before prediction or further processing.A shallow model can directly produce a prediction, while adding a DNN creates the deep model.

4 EXPERIMENTS

The experiments evaluate FiBiNET against state-of-the-art CTR prediction methods and examine how interaction functions, field types, and network settings affect performance.

  • The experiments address four questions covering overall CTR performance, bilinear–Hadamard combinations, field types, and network settings.
  • The study presents fundamental experimental settings before answering these research questions.

4.1 Experimental Testbeds and Setup

Experiments use the Criteo and Avazu click-through datasets, evaluate models with AUC and log loss, and compare shallow and deep model groups under fixed training settings.

  • Datasets: Criteo contains 45 million instances with 26 categorical and 13 continuous fields, split randomly into 90% training and 10% testing.
  • Datasets: Avazu contains 40 million chronologically ordered ad click-through instances collected over several days.
  • Metrics: The evaluation uses AUC, where larger values are better, and log loss, where smaller values are better.
  • Baselines: Shallow baselines are LR, FM, FFM, and AFM, while deep baselines are FNN, DCN, DeepFM, and XDeepFM.
  • Evaluation context: A 1‰ AUC improvement is considered significant for CTR prediction because of its potential revenue impact at very large user bases.
  • Implementation: Models use TensorFlow, Adam optimization, dataset-specific embedding sizes, mini-batches, and a learning rate of 0.0001.

4.2 Performance Comparison(RQ1)

FiBiNET’s shallow and deep variants are compared with established CTR models on Criteo and Avazu. The shallow model consistently beats competing shallow models, while deep FiBiNET achieves the best reported performance.

  • Shallow models: Table 1 reports shallow-model performance on the Criteo and Avazu datasets, with SE-FM-ALL representing the Field-All variant.
  • Shallow models: SE-FM-All consistently outperforms FM, FFM, AFM, and other shallow models on Criteo and Avazu.
  • Shallow models: FiBiNET uses fewer parameters than FFM while achieving better performance, making it an alternative to FFM under its stated memory constraint.
  • Deep models: Table 2 presents deep-model performance, with DeepSE-FM-ALL denoting the Field-All deep variant.
  • Deep models: Combining the shallow component with a DNN improves the shallow model’s performance by adding implicit high-order feature interactions.
  • Deep models: Deep FiBiNET achieves the best performance among compared methods, outperforming FNN and DeepFM on AUC and log loss across Criteo and Avazu.Relative improvements over FNN are 0.571% and 0.386% in AUC, and 0.918% and 0.4% in log loss; relative improvements over DeepFM are 0.222% and 0.59% in AUC, and 0.494% and 0.6% in log loss.

4.3 Combinations of Bilinear-Interaction Layer(RQ2)

The study evaluates combinations of bilinear and Hadamard interactions in shallow and deep models. Results show different preferred combinations across model types, with no universally obvious conclusion.

  • Interaction combinations: Each combination is encoded by two digits indicating the interaction function for the original and SENET-Like embeddings, respectively.A 1 denotes bilinear interaction and 0 denotes Hadamard interaction; for example, 10 applies bilinear interaction to the original embedding and Hadamard interaction to the SENET-Like embedding.
  • Interaction combinations: Table 3 reports shallow and deep results for different bilinear–Hadamard combinations with the Field-Each interaction type.
  • Empirical observations: The experiments do not yield an obvious overall conclusion about the best interaction combination.
  • Empirical observations: On Criteo, combination 11 performs best among shallow models but worst among deep models.
  • Empirical observations: For deep models, combination 01 is preferred because bilinear interaction is applied only to the SENET-Like embedding layer.

4.4 Field Types of Bilinear-Interaction (RQ3)

Field type affects Bilinear-Interaction performance differently across datasets. Field-Interaction improves shallow FiBiNET on Criteo, while the best field type varies by dataset for deep models.

  • Shallow models: 0.382% AUC improvement is obtained by Field-Interaction over Field-All for shallow FiBiNET on Criteo.This corresponds to a relative improvement of 0.476%.
  • Deep models: Field-Interaction improves the deep model on Criteo, whereas Field-Each improves it on Avazu.
  • Dataset dependence: On Criteo, field types rank as Field-Interaction, Field-Each, then Field-All.
  • Dataset dependence: Field-type performance depends on the dataset, with no obvious ranking reported for Avazu.

4.5 Hyper-parameter Investigation(RQ4)

The hyper-parameter study examines embedding size and DNN structure. Embedding size has opposite effects across datasets, while excessive DNN depth can degrade performance through overfitting.

  • Study design: The investigation varies embedding dimension, neurons per DNN layer, and DNN depth.The study also notes that activation functions and dropout rates can be changed, though it focuses on neural units and depth.
  • Embedding part: Embedding expansion from 10 to 50 substantially improves performance on Avazu but degrades performance on Criteo.The authors suggest that more features in Criteo may create optimization difficulties as parameter counts increase.
  • DNN part: Increasing DNN depth initially improves performance, but continued increases degrade it because over-complicated models can overfit.Three hidden layers are reported as a good choice for both Avazu and Criteo.
  • DNN part: The reported DNN analyses include performance comparisons across different numbers of layers and neurons per layer.

4.6 Ablation Study (RQ5)

Ablation experiments show that both SENET and Bilinear-Interaction are important components of FiBiNET. Removing either component causes an apparent performance drop.

  • Ablation setup: Removing SENET and Bilinear-Interaction reduces shallow and deep FiBiNET to FM and FNN, respectively.
  • Ablation results: Removing either the Bilinear-Interaction layer or SENET layer causes an apparent performance drop.
  • Ablation results: Both the Bilinear-Interaction and SENET layers are necessary for FiBiNET’s performance.
  • Component importance: The Bilinear-Interaction layer is reported to be as important as the SENET layer.

5 CONCLUSIONS

FiBiNET dynamically models feature importance with SENET and fine-grained interactions with bilinear layers. Its shallow version outperforms FM and FFM, while deep FiBiNET outperforms DeepFM and XDeepFM.

  • Contributions: FiBiNET dynamically learns feature importance and fine-grained feature interactions for CTR prediction.
  • Contributions: SENET dynamically increases weights for important features and decreases weights for uninformative features.
  • Contributions: Three Bilinear-Interaction layer types model feature interactions instead of relying only on Hadamard or inner products.
  • Empirical findings: The shallow FiBiNET outperforms shallow models including FM and FFM, while deep FiBiNET consistently outperforms DeepFM and XDeepFM.
Loading 1905.09433v1…