PaperlayerView exact revision on arXiv

Source-linked AI summary

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, Guangzhong Sun

arXiv:1803.05170v3cs.LGcs.IR

TL;DR

Manual cross-feature engineering is costly, difficult to scale, and unable to generalize to unseen interactions, while DNNs represent interactions implicitly at the bit-wise level. The paper introduces xDeepFM, combining an explicit vector-wise CIN with a classical DNN, and reports that it outperforms state-of-the-art models on three real-world datasets.

  • Problem

    Manual cross-feature engineering is costly, infeasible for large-scale systems, and unable to generalize to unseen interactions, while DNN interaction degree and representation remain theoretically unspecified.

  • Method

    xDeepFM combines a Compressed Interaction Network for bounded-degree explicit vector-wise interactions with a DNN for implicit low- and high-order interactions.

  • Results

    xDeepFM outperforms several state-of-the-art models in experiments on three real-world datasets.

  • Takeaways & Limitations

    The unified model automatically learns explicit and implicit feature interactions, reducing the need for manual feature engineering.

  • Takeaways & Limitations

    The CIN module has high time complexity, and the current model uses simple sum pooling for multivalent fields.

Abstract

from arXiv · show

Combinatorial features are essential for the success of many commercial models. Manually crafting these features usually comes with high cost due to the variety, volume and velocity of raw data in web-scale systems. Factorization based models, which measure interactions in terms of vector product, can learn patterns of combinatorial features automatically and generalize to unseen features as well. With the great success of deep neural networks (DNNs) in various fields, recently researchers have proposed several DNN-based factorization model to learn both low- and high-order feature interactions. Despite the powerful ability of learning an arbitrary function from data, plain DNNs generate feature interactions implicitly and at the bit-wise level. In this paper, we propose a novel Compressed Interaction Network (CIN), which aims to generate feature interactions in an explicit fashion and at the vector-wise level. We show that the CIN share some functionalities with convolutional neural networks (CNNs) and recurrent neural networks (RNNs). We further combine a CIN and a classical DNN into one unified model, and named this new model eXtreme Deep Factorization Machine (xDeepFM). On one hand, the xDeepFM is able to learn certain bounded-degree feature interactions explicitly; on the other hand, it can learn arbitrary low- and high-order feature interactions implicitly. We conduct comprehensive experiments on three real-world datasets. Our results demonstrate that xDeepFM outperforms state-of-the-art models. We have released the source code of xDeepFM at \url{https://github.com/Leavingseason/xDeepFM}.

1 INTRODUCTION

The paper addresses the cost, scalability, and generalization limits of manually engineered cross features, while distinguishing implicit bit-wise DNN interactions from explicit vector-wise interactions. It proposes xDeepFM, which combines a CIN and DNN to learn both interaction types without manual feature engineering, and reports superior performance to several state-of-the-art models.

  • Motivation: Manual cross-feature engineering is costly, infeasible at web scale, and unable to generalize to unseen training interactions.These limitations motivate learning feature interactions automatically.
  • Existing approaches: Factorization Machines represent pairwise feature interactions as inner products of latent feature vectors multiplied by their input values.The interaction is modeled as f^(2)(i,j) = ⟨v_i,v_j⟩x_ix_j.
  • Existing approaches: DNNs learn high-order interactions implicitly and at the bit-wise level, leaving the maximum interaction degree theoretically unspecified.This differs from the vector-wise interactions used in classical Factorization Machines.
  • Proposed approach: xDeepFM jointly learns explicit and implicit high-order feature interactions without manual feature engineering.Its CIN learns explicit vector-wise interactions with increasing interaction degree across layers.
  • Results: Experiments on three real-world datasets show that xDeepFM significantly outperforms several state-of-the-art models.The paper presents this as evidence for the model’s effectiveness.

2 PRELIMINARIES

The section introduces field embeddings for sparse categorical inputs and reviews implicit and explicit approaches to modeling high-order feature interactions. It highlights that DNN-based models capture bit-wise interactions, while CrossNet explicitly models interactions but has a constrained output form.

  • 2.1 Embedding Layer: Sparse categorical inputs are converted into dense field embeddings, producing a fixed-length concatenated vector with m fields and embedding dimension D.Univalent fields use one feature embedding, while multivalent fields sum their feature embeddings.
  • 2.2 Implicit High-order Interactions: Feed-forward neural networks learn high-order interactions implicitly and at the bit-wise level over the field embedding vector.Elements within the same field embedding can influence one another.
  • 2.2 Implicit High-order Interactions: PNN and DeepFM augment DNNs with a two-way interaction layer, combining bit-wise and vector-wise interactions.PNN connects product-layer outputs to the DNN, whereas DeepFM connects its FM layer directly to the output unit.
  • 2.3 Explicit High-order Interactions: CrossNet computes hidden layers through a cross operation designed to explicitly model high-order feature interactions.Its layer outputs are described as scalar multiples of the original input vector.
  • 2.3 Explicit High-order Interactions: CrossNet’s output is restricted to a scalar multiple of x0, and its interactions are formed bit-wise despite its efficient computation.The scalar coefficient remains sensitive to x0, so the restriction does not make the network linear in x0.

3 OUR PROPOSED MODEL

CIN explicitly models increasing-degree feature interactions at the vector-wise level, then combines these interactions with implicit DNN interactions in xDeepFM. Its compressed architecture uses pooled hidden representations and avoids exponential growth in interaction degree, though its main downside is computational time complexity.

  • 3.1 Compressed Interaction Network: CIN applies interactions at the vector-wise level, explicitly measuring higher-order feature interactions as network depth increases.Each layer interacts the preceding hidden representation with the original feature matrix while preserving embedding-vector structure.
  • 3.1 Compressed Interaction Network: CIN is structured like a recurrent network and has CNN-like operations over an intermediate tensor formed from hidden and original feature representations.The tensor is constructed through outer products, and learned filters slide along the embedding dimension.
  • 3.1 Compressed Interaction Network: Each CIN layer compresses Hk−1 × m potential vectors into Hk feature maps, with sum pooling connecting every hidden layer to the output.The pooled vectors are concatenated before the output layer, and binary classification uses a sigmoid node.
  • 3.2 Complexity Analysis: CIN uses O(mTHL + TH^2L) space after L-order decomposition, compared with O(mTH^2) without decomposition.Here L is much smaller than both H and m.
  • 3.2.2 Time Complexity: A T-layer CIN requires O(mH^2DT) computation time, making time complexity its major downside.The stated cost comes from computing the intermediate tensor at each layer.
  • 3.3 Combination with Implicit Networks: xDeepFM combines CIN and a plain DNN to learn explicit and implicit feature interactions in one end-to-end model.With restricted settings, xDeepFM generalizes DeepFM and can reduce to traditional FM.

4 EXPERIMENTS

The experiments examine CIN’s ability to learn high-order interactions, the value of combining explicit and implicit interactions, and the influence of network settings.

  • Experimental Questions: The experiments ask how CIN performs in high-order interaction learning, whether explicit and implicit interactions should be combined, and how network settings affect xDeepFM.These questions are labeled Q1, Q2, and Q3.

4.1 Experiment Setup

The study evaluates xDeepFM and related models on three recommendation datasets using AUC and Logloss, with validation-based hyperparameter tuning and reported network depths.

  • Datasets: Experiments use Criteo for ad click-through prediction, Dianping for restaurant recommendation, and Bing News for news-reading prediction.The datasets cover industry benchmarking, consumer reviews, and commercial news impression logs.
  • Datasets: Criteo and Dianping instances are randomly split 8:1:1, while Bing News uses three days for training and validation and two days for testing.The dataset statistics are summarized in Table 1.
  • Evaluation Metrics: AUC measures ranking quality, whereas Logloss measures the distance between predicted scores and true labels.Logloss is emphasized when predicted probabilities are used to estimate ranking-strategy benefit.
  • Compared Models: xDeepFM is compared with LR, FM, DNN, PNN, Wide & Deep, DCN, and DeepFM without including handcrafted cross features.The comparison targets automatic feature-interaction learning.
  • Implementation: Hyperparameters are tuned by grid search on validation data, with Adam optimization, learning rate 0.001, and mini-batches of 4096.The implementation uses TensorFlow and model-specific regularization or dropout settings.

4.2 Performance Comparison among Individual Neural Components (Q1)

Across the three practical datasets, CIN consistently outperforms the other individual interaction models, supporting explicit high-order interaction learning for sparse features.

  • Individual Model Results: CIN consistently outperforms the other individual models on the Criteo, Dianping, and Bing News datasets.The comparison includes FM, DNN, and CrossNet, whose interaction mechanisms differ from CIN’s explicit high-order modeling.
  • Individual Model Results: DNN, CrossNet, and CIN significantly outperform FM on all three datasets, indicating the value of higher-order interactions over sparse features.The paper presents this pattern as evidence that higher-order interactions are necessary for these practical datasets.
  • Network Depth: A k-layer CIN models k-degree feature interactions, and five layers produce the best result on the Bing News dataset.The optimal depth therefore varies with the dataset.

4.3 Performance of Integrated Models (Q2)

xDeepFM combines CIN and DNN to learn explicit and implicit high-order feature interactions. It achieves the best performance across all evaluated datasets, with its best depth indicating interactions up to fourth order.

  • xDeepFM integrates CIN and DNN in an end-to-end model for jointly learning explicit and implicit high-order feature interactions.
  • xDeepFM achieves the best performance on all datasets among the compared models.
  • The best depth setting for xDeepFM is 3, indicating that the learned interactions are at most 4-order.

4.4 Hyper-Parameter Study (Q3)

The hyper-parameter study varies CIN depth, neurons per layer, and activation functions while holding the DNN settings fixed. Performance generally favors depth 3, dataset-dependent neuron counts, and identity activation in CIN.

  • The study varies hidden-layer count, neurons per layer, and activation functions in CIN while holding the DNN settings fixed.
  • Depth of Network: Performance improves as network depth initially increases but degrades beyond 3 layers because of overfitting.Training loss continues decreasing when additional hidden layers are added.
  • Number of Neurons per Layer: On Bing News, performance increases steadily from 20 to 200 neurons per layer, whereas 100 is more suitable for Dianping.The depth is fixed at 3 in this experiment.
  • Activation Function: Identity is the most suitable activation function for CIN neurons among the compared alternatives.The DNN activation function remains relu during this comparison.

5 RELATED WORK

Related work addresses sparse recommender-system features through linear models, factorization methods, and deep networks. These approaches differ in feature-interaction capacity, generalization, and whether interactions are modeled explicitly or implicitly.

  • Web-scale recommender systems use sparse, high-dimensional, categorical-continuous mixed features, making feature handling central to model design.
  • Linear models are easy to manage and deploy but require engineered cross features because they cannot learn feature interactions.
  • Factorization Machines embed features into low-dimensional latent vectors to generalize to unseen feature interactions.
  • Deep recommender models learn categorical interaction patterns automatically and include FNN, PNN, DeepCross, NFM, DCN, Wide&Deep, and DeepFM.
  • Other neural recommender systems model auxiliary information, collaborative filtering, multiple domains, or users’ diverse interests rather than focusing primarily on feature interactions.

6 CONCLUSIONS

The paper presents xDeepFM as a framework for explicit and implicit high-order feature interactions and reports consistent gains on three real-world datasets. It identifies multivalent-field pooling and CIN efficiency as directions for future work.

  • xDeepFM combines CIN and DNN to learn high-order feature interactions explicitly and implicitly, reducing manual feature-engineering work.CIN targets bounded-degree, vector-wise interactions, while the combined model also learns implicit interactions.
  • The authors report that xDeepFM consistently outperforms state-of-the-art models on three real-world datasets.
  • Future work includes replacing sum pooling for multivalent fields and developing a distributed xDeepFM implementation because CIN has high time complexity.
Loading 1803.05170v3…