Source-linked AI summary
Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks
Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, Tat-Seng Chua
TL;DR
Factorization machines model pairwise feature interactions but cannot distinguish their importance, which may make irrelevant interactions noisy. AFM adds an attention network that learns interaction importance from data, achieving the best performance among the compared methods while retaining a shallow structure and fewer parameters than Wide&Deep.
Problem
Factorization machines assign feature interactions the same weight, despite some interactions being irrelevant and potentially introducing noise.
Method
AFM uses a neural attention network to learn feature-interaction importance automatically from data.
Results
AFM achieves the best performance among the compared methods, improving over LibFM by 8.6% and over Wide&Deep by 4.3%.
Takeaways & Limitations
Attention improves FM’s representation ability and interpretability by indicating which feature interactions contribute more to prediction.
Takeaways & Limitations
AFM has complexity quadratic in the number of non-zero features, motivating future work on improving its learning efficiency.
Abstract
from arXiv · showhide
Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Despite effectiveness, FM can be hindered by its modelling of all feature interactions with the same weight, as not all feature interactions are equally useful and predictive. For example, the interactions with useless features may even introduce noises and adversely degrade the performance. In this work, we improve FM by discriminating the importance of different feature interactions. We propose a novel model named Attentional Factorization Machine (AFM), which learns the importance of each feature interaction from data via a neural attention network. Extensive experiments on two real-world datasets demonstrate the effectiveness of AFM. Empirically, it is shown on regression task AFM betters FM with a $8.6\%$ relative improvement, and consistently outperforms the state-of-the-art deep learning methods Wide&Deep and DeepCross with a much simpler structure and fewer model parameters. Our implementation of AFM is publicly available at: https://github.com/hexiangnan/attentional_factorization_machine
1 Introduction
The introduction motivates modeling feature interactions in supervised learning and identifies a limitation of factorization machines: uniform interaction weighting. AFM addresses this by learning interaction importance from data with an attention network.
- Motivation: Categorical feature interactions can determine predictions, while independence assumptions can make linear regression inaccurate.The paper illustrates this with income prediction, where the relationship between occupation and income changes by seniority level.
- Motivation: Polynomial regression and related cross-feature methods cannot estimate parameters for unobserved cross features in sparse datasets.These methods explicitly augment inputs with products of features and learn a weight for each cross feature.
- Motivation: Factorization machines generalize to unobserved cross features by factorizing each cross-feature weight into the inner product of constituent feature embeddings.FM learns an embedding vector for each feature, allowing weights for arbitrary feature pairs to be estimated.
- AFM: AFM uses a neural attention network to learn each feature interaction’s importance automatically from data.Experiments on two public benchmark datasets also aim to improve performance and reveal which interactions contribute more to prediction.
2 Factorization Machines
Factorization machines extend linear regression by modeling pairwise feature interactions through factorized weights. However, standard FM assigns all estimated interactions the same weight, limiting its ability to distinguish useful interactions from noise.
- FM formulation: Factorization machines model the target using a global bias, individual feature weights, and pairwise feature interactions.The paper presents FM as a supervised-learning model originally proposed for collaborative recommendation.
- FM formulation: Each cross-feature weight is factorized as the inner product of the embedding vectors of its two constituent features.For feature i, vi is its embedding vector and k is the embedding size.
- FM formulation: Only interactions between non-zero features are considered because each interaction is multiplied by x_i x_j.This follows directly from the feature coefficient in the FM interaction term.
- FM limitation: Standard FM models all feature interactions uniformly, using shared feature embeddings and an interaction weight of 1.Interactions involving irrelevant features can therefore behave as noise with no contribution to prediction.
3 Attentional Factorization Machines
AFM extends FM with pair-wise interaction modeling and an attention-based pooling layer that learns different interaction weights from data. The model supports multiple prediction tasks and uses regularization to control overfitting.
- Model architecture: AFM retains FM’s sparse input and embedding layers while adding pair-wise interaction and attention-based pooling as its main components.Each non-zero feature is embedded into a dense vector before interactions are modeled.
- Pair-wise Interaction Layer: The pair-wise interaction layer expands m embedded feature vectors into m(m −1)/2 vectors using element-wise products of distinct feature pairs.This expresses FM within a neural network architecture.
- Model architecture: Setting the prediction-layer weights p to 1 and bias b to 0 exactly recovers FM from the neural architecture.AFM therefore contains FM as a special case of its pooling and prediction design.
- Attention-based Pooling Layer: AFM compresses interaction vectors with an attention-weighted sum, allowing different feature interactions to contribute differently to the prediction.Attention scores represent interaction importance and are learned through an attention network rather than assigned uniformly.
- Attention-based Pooling Layer: The attention network uses an MLP with ReLU activation and softmax-normalized scores, with hidden-layer size t called the attention factor.Its parameters include W, b, and h.
- Learning: AFM can support regression, classification, and ranking, but this paper focuses on regression optimized with squared loss.The objective is tailored to the prediction task; binary classification or recommendation can instead use log loss.
- Overfitting Prevention: Dropout on the interaction layer and L2 regularization on the attention network are used to reduce overfitting.The authors do not apply dropout to the attention network because joint dropout was unstable and degraded performance.
4 Related Work
Related work addresses sparse feature interactions through factorization machines, feature selection, and deep neural networks. AFM differs by learning the importance of each interaction while retaining explicit interaction modeling.
- Factorization Machines: Factorization machines estimate weights for unobserved cross features by representing them as inner products of constituent feature embeddings.This generalization makes FMs useful in sparse applications such as recommendation and natural language processing.
- Feature Selection: GBFM selects good features with gradient boosting but assigns the selected interactions the same summed weight as FM.The paper characterizes GBFM as feature selection rather than interaction-specific weighting.
- Deep Learning Methods: Wide&Deep and DeepCross capture feature interactions implicitly with deep neural networks rather than explicitly modeling each interaction as an inner product.The paper states that AFM is more interpretable because its attention mechanism learns each interaction’s importance.
5 Experiments
Experiments evaluate AFM’s hyperparameters, interpretability, convergence, and performance against FM, LibFM, HOFM, Wide&Deep, and DeepCross on Frappe and MovieLens. AFM consistently benefits from attention-weighted interactions and achieves the strongest reported test performance with fewer parameters than deep alternatives.
- Experimental Design: Experiments use the public Frappe and MovieLens datasets to study AFM’s hyperparameters, learned interaction importance, and sparse-data prediction performance.The comparisons include LibFM, HOFM, Wide&Deep, and DeepCross under squared-loss training with embedding size 256.
- Hyperparameter Analysis: Dropout improves both AFM and FM when tuned, with AFM’s optimal ratios being 0.2 on Frappe and 0.5 on MovieLens.The result supports dropout on the pair-wise interaction layer as a generalization aid.
- Hyperparameter Analysis: Positive L2 regularization on the attention network further improves AFM, showing that interaction-layer dropout alone is insufficient to prevent overfitting.The authors report that tuning the attention network can further improve AFM’s generalization.
- Attention Analysis: AFM remains strong across attention factors, including factor 1, where its attention component reduces to linear regression over interacted vectors.This supports estimating interaction importance from the interacted vector itself.
- Convergence and Generalization: AFM converges faster than FM and achieves lower test error; on MovieLens, its higher training error is accompanied by better generalization to unseen data.On Frappe, both AFM’s training and test errors are much lower than FM’s.
- Performance Comparison: AFM beats LibFM by 8.6% relative improvement and Wide&Deep by 4.3%, while using much fewer parameters than the deep-learning comparison methods.The authors also report that AFM outperforms FM and LibFM by a large margin, whereas DeepCross performs worst because of severe overfitting.
6 Conclusion and Future Work
AFM enhances FM by learning feature-interaction importance with an attention network, improving representation ability and interpretability. Future work targets deeper AFM variants, more efficient learning, and broader applications.
- Conclusion: AFM enhances FM by learning feature-interaction importance with an attention network.
- Conclusion: The attention-based design improves both FM’s representation ability and interpretability.
- Future Work: Future work includes deeper AFM architectures, improved efficiency, semi-supervised and multi-view variants, and applications to other data types.