PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Factorization Machines for Sparse Predictive Analytics

Xiangnan He, Tat-Seng Chua

arXiv:1708.05027v1cs.LG

TL;DR

Sparse categorical features make interaction modelling important, but FM is limited by linear pairwise modelling and deep neural approaches can be difficult to optimize. NFM combines FM-style second-order interactions with nonlinear neural layers, and with one hidden layer outperforms FM and deeper baselines on two benchmarks.

  • Problem

    Sparse categorical features require effective interaction modelling, while FM may be insufficient for complex nonlinear structure and deep neural models can be difficult to optimize.

  • Method

    NFM uses Bi-Interaction pooling to model second-order interactions and nonlinear neural layers to learn higher-order interactions.

  • Results

    7.3% improvement: one-hidden-layer NFM significantly outperforms FM and consistently improves on Wide&Deep and DeepCross across two public benchmarks.

  • Takeaways & Limitations

    NFM bridges linear factorization machines and nonlinear neural networks with a shallower structure and fewer parameters than the compared deep approaches.

  • Takeaways & Limitations

    FM may suffer insufficient representation ability for complex real-world structure, while deep concatenation-based architectures can be difficult to optimize.

Abstract

from arXiv · show

Many predictive tasks of web applications need to model categorical variables, such as user IDs and demographics like genders and occupations. To apply standard machine learning techniques, these categorical predictors are always converted to a set of binary features via one-hot encoding, making the resultant feature vector highly sparse. To learn from such sparse data effectively, it is crucial to account for the interactions between features. Factorization Machines (FMs) are a popular solution for efficiently using the second-order feature interactions. However, FM models feature interactions in a linear way, which can be insufficient for capturing the non-linear and complex inherent structure of real-world data. While deep neural networks have recently been applied to learn non-linear feature interactions in industry, such as the Wide&Deep by Google and DeepCross by Microsoft, the deep structure meanwhile makes them difficult to train. In this paper, we propose a novel model Neural Factorization Machine (NFM) for prediction under sparse settings. NFM seamlessly combines the linearity of FM in modelling second-order feature interactions and the non-linearity of neural network in modelling higher-order feature interactions. Conceptually, NFM is more expressive than FM since FM can be seen as a special case of NFM without hidden layers. Empirical results on two regression tasks show that with one hidden layer only, NFM significantly outperforms FM with a 7.3% relative improvement. Compared to the recent deep learning methods Wide&Deep and DeepCross, our NFM uses a shallower structure but offers better performance, being much easier to train and tune in practice.

1 INTRODUCTION

Sparse predictive tasks require models that learn feature interactions without relying on costly manual feature engineering. NFM extends FM with neural nonlinearity, achieving stronger results than FM and deeper neural approaches on two benchmarks.

  • Motivation: Categorical variables produce high-dimensional sparse vectors, making feature interactions crucial for effective prediction.Manual combinatorial features require substantial engineering effort and domain knowledge.
  • Limitations of Existing Models: FM learns pairwise feature interactions efficiently but may lack expressiveness for complex, nonlinear real-world structure.Higher-order FM variants remain linear models and provide only marginal improvements over FM.
  • Proposed Approach: NFM introduces Bi-Interaction pooling and stacks nonlinear layers above it to model higher-order and nonlinear feature interactions.The approach places a more informative interaction representation beneath the neural layers.
  • Results: 7.3% improvement: with one hidden layer, NFM significantly outperforms the official LibFM implementation on two public benchmarks.The benchmarks cover context-aware prediction and personalized tag recommendation.
  • Results: NFM also consistently improves on 3-layer Wide&Deep and 10-layer DeepCross while using a simpler structure and fewer parameters.The comparison is reported across the two public benchmark tasks.

2 MODELLING FEATURE INTERACTIONS

The paper contrasts factorization-based linear interaction models with deep neural approaches for sparse data. NFM uses Bi-Interaction pooling to provide informative second-order representations before nonlinear layers learn higher-order interactions.

  • Interaction Modelling: Embedding-based methods map high-dimensional sparse features into a low-dimensional latent space to learn interactions from raw data.They avoid relying exclusively on manually constructed combinatorial features.
  • Factorization Machines: FM models pairwise feature interactions through factorized embedding inner products and generalizes linear or logistic regression.Its generality allows it to represent several specific factorization models.
  • Factorization Machines: FM remains linear with respect to each model parameter, limiting representation ability when real-world data has complex nonlinear structure.This is identified as an expressiveness limitation of FM.
  • Deep Neural Networks: Standard deep sparse-feature models concatenate embeddings and rely on multiple layers to learn interactions implicitly.Examples include FNN, Wide&Deep, and DeepCross.
  • Deep Neural Networks: Deep architectures can be difficult to optimize because concatenated embeddings provide little low-level interaction information and subsequent layers must learn meaningful interaction functions.The paper connects this weakness to poor results from simple embedding concatenation and optimization problems in deep networks.
  • NFM: Bi-Interaction pooling models second-order interactions before nonlinear layers, creating a more informative representation for learning higher-order interactions.Nonlinear activations allow higher-order interactions to be learned nonlinearly rather than only linearly.

3 NEURAL FACTORIZATION MACHINES

NFM combines FM-style second-order interaction modeling with neural layers that learn higher-order, non-linear interactions. Its Bi-Interaction pooling is efficient, NFM generalizes FM, and dropout can regularize both FM-like and deeper components.

  • NFM Model: NFM represents feature interactions with a multi-layered feedforward network built above a Bi-Interaction pooling layer.The embedding layer produces dense feature representations, while Bi-Interaction pooling encodes second-order interactions before hidden layers model more complex interactions.
  • NFM Model: Bi-Interaction pooling converts embedded features into a k-dimensional vector encoding second-order interactions through element-wise products.The operation introduces no additional model parameters and can be evaluated efficiently using the sparse input representation.
  • NFM Model: NFM’s hidden layers use non-linear activation functions to learn higher-order feature interactions beyond the linear modeling of higher-order FM methods.The hidden-layer computation adds parameters for learning these higher-order interactions.
  • NFM Generalizes FM: FM is a special case of NFM obtained by setting the number of hidden layers to zero and projecting Bi-Interaction pooling directly to the prediction score.Fixing the prediction vector h to (1, ..., 1) exactly recovers the FM model; a trainable h does not increase NFM-0’s expressiveness.
  • Relation to Wide&Deep and DeepCross: NFM differs from Wide&Deep and DeepCross through its Bi-Interaction pooling component, while its overall evaluation complexity matches theirs.The overall complexity is O(kN_x + Σ_{l=1}^L d_{l-1}d_l), with Bi-Interaction pooling computed in O(kN_x) time.
  • Learning: Dropout is applied to the Bi-Interaction layer and hidden layers to reduce embedding co-adaptation and overfitting.For NFM without hidden layers, dropout on Bi-Interaction pooling provides a way to regularize the corresponding FM model.

4 EXPERIMENTS

The experiments evaluate NFM on two public datasets and study its regularization, normalization, nonlinear hidden layers, depth, pre-training, and comparative performance against FM and deep baselines.

  • Experimental Settings: Negative sampling pairs each positive instance with two negatives, assigning negative instances a target value of −1.This constructs evaluation data from the positive-only original datasets.
  • Experimental Settings: Experiments use Frappe and MovieLens, with model properties studied on validation data and final comparisons conducted using held-out test data.The datasets are randomly split into 70% training, 20% validation, and 10% test sets, and RMSE is used for evaluation.
  • Study of Bi-Interaction Pooling: Dropout and L2 regularization both improve NFM-0 generalization, while Frappe achieves validation error 0.3562 with dropout ratio 0.3 versus 0.3799 with L2 regularization.The authors report that dropout performs better and may act like an ensemble of sub-models.
  • Study of Bi-Interaction Pooling: Batch normalization accelerates convergence and slightly improves generalization, but combining it with dropout makes learning less stable.On Frappe, BN reaches lower training error at epoch 20 than the no-BN model reaches at epoch 60.
  • Study of Bi-Interaction Pooling: Nonlinear hidden layers improve NFM over NFM-0 by 11.3% on Frappe and 5.2% on MovieLens, whereas a linear activation performs poorly.The result supports modeling higher-order feature interactions nonlinearly.
  • Impact of Hidden Layers: One hidden layer gives the best NFM performance; deeper NFM models do not improve further, while NFM consistently performs best on both datasets with few parameters besides FM.The informative Bi-Interaction layer reduces the need for deeper structures, and NFM outperforms Wide&Deep and DeepCross in the reported comparison.
  • Impact of Hidden Layers: FM-embedding pre-training makes NFM converge rapidly but does not improve final performance over random initialization.With five epochs of pre-training, performance is reported as comparable to 40 epochs of NFM trained from scratch.
  • Performance Comparison: HOFM improves over FM by 1.45% on Frappe and 1.04% on MovieLens, but NFM’s nonlinear higher-order modeling yields a larger performance gap.DeepCross, despite using 10 layers, performs relatively weakly and can underperform FM.

5 CONCLUSION AND FUTURE WORK

NFM combines factorization machines with nonlinear neural networks to improve sparse predictive analytics. Its Bi-Interaction operation supports informative feature interactions, and experiments show strong performance with only one hidden layer while future work targets efficiency and broader applications.

  • NFM combines linear factorization machines with nonlinear neural networks for sparse predictive analytics.
  • The Bi-Interaction operation lets the neural network learn more informative lower-level feature interactions.
  • NFM significantly outperforms FM, higher-order FM, Wide&Deep, and DeepCross with one hidden layer.
  • NFM bridges linear models and deep learning for sparse prediction, where effective deep-learning solutions remain limited.
  • Future work will improve NFM efficiency with hashing and evaluate it on search ranking and targeted advertising.
Loading 1708.05027v1…