Source-linked AI summary
Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions
Weiyu Cheng, Yanyan Shen, Linpeng Huang
TL;DR
Existing factorization methods trade off higher-order expressiveness against computational cost and may introduce noisy interactions by enumerating all candidate cross features. AFN learns arbitrary-order cross features and their weights adaptively through a logarithmic transformation layer, and experiments on four datasets report superior predictive performance.
Problem
Factorization methods must choose a maximum interaction order and enumerate candidate cross features, limiting expressiveness, increasing cost, and potentially retaining noisy combinations.
Method
AFN transforms feature embeddings into logarithmic space and learns feature powers as coefficients through a logarithmic transformation layer, then applies feed-forward networks to learned cross features.
Results
AFN achieves the best or competing performance across four datasets, while AFN+ achieves the best performance on all four datasets.
Takeaways & Limitations
AFN adaptively generates discriminative cross features and corresponding weights without explicitly modeling every feature combination within a fixed maximum order.
Abstract
from arXiv · showhide
Various factorization-based methods have been proposed to leverage second-order, or higher-order cross features for boosting the performance of predictive models. They generally enumerate all the cross features under a predefined maximum order, and then identify useful feature interactions through model training, which suffer from two drawbacks. First, they have to make a trade-off between the expressiveness of higher-order cross features and the computational cost, resulting in suboptimal predictions. Second, enumerating all the cross features, including irrelevant ones, may introduce noisy feature combinations that degrade model performance. In this work, we propose the Adaptive Factorization Network (AFN), a new model that learns arbitrary-order cross features adaptively from data. The core of AFN is a logarithmic transformation layer to convert the power of each feature in a feature combination into the coefficient to be learned. The experimental results on four real datasets demonstrate the superior predictive performance of AFN against the start-of-the-arts.
1 Introduction
Existing factorization methods must choose a maximum interaction order and enumerate candidate cross features, creating a trade-off between expressiveness, cost, and noise. AFN addresses both issues by learning arbitrary-order cross features and their weights adaptively through a logarithmic transformation layer.
- Limitations of Existing Methods: Higher maximum orders can model more complex interactions but increase the number of cross features and computational complexity.Large model size also raises training and prediction costs, limiting practical use of higher-order interactions.
- Limitations of Existing Methods: Useful cross features vary in predictive power, while irrelevant interactions can add noise or degrade model performance.Attention-based filtering reduces the influence of useless interactions but substantially increases computational cost and is limited to second-order interactions in AFM.
- Limitations of Existing Methods: Existing methods predefine a maximum order, enumerate all cross features within it, and filter irrelevant combinations during training.This enumerating-and-filtering procedure restricts the discovery of discriminative higher-order features and cannot always remove useless combinations successfully.
- Proposed Approach: AFN learns arbitrary-order cross features and their weights adaptively from data instead of relying on a fixed maximum order.Its logarithmic neural transformation layer converts feature powers into learnable coefficients, allowing each neuron to learn feature combinations of possibly different orders.
2 Background
The background introduces feature embeddings, factorization machines, logarithmic neural networks, and AFN's use of logarithmic neurons for adaptive feature orders. It also contrasts the computational and modeling limitations of fixed-order factorization methods with AFN's design.
- Feature Embeddings: Sparse categorical features are mapped into dense low-dimensional embeddings, while numerical features can also be embedded for interactions with categorical features.The resulting collection of feature embeddings is used by factorization machines or neural networks for prediction.
- Factorization Machines: Factorization Machines explicitly model second-order interactions, whereas Higher-Order Factorization Machines extend this idea to higher-order feature combinations.HOFM prediction with maximum order n has direct complexity O(kmn), or O(kmn^2) with dynamic programming.
- Limitations of Factorization Methods: FMs and HOFMs assign the same weight to all interactions, so irrelevant cross features can introduce noise and degrade predictions.Attention or retained-feature weighting adds cost, while these approaches still require a preset maximum order n, usually kept small to control model size.
- Logarithmic Neural Networks: Logarithmic neural networks transform inputs into logarithmic space, converting multiplication and powers into addition and multiplication by a constant.AFN uses logarithmic neurons to learn feature powers from data, with vector-wise powers shared across embeddings from the same field.
- Adaptive Factorization Network: AFN's logarithmic transformation layer learns feature-field powers through multiple vector-wise logarithmic neurons.Appropriate coefficients can produce cross features of arbitrary orders, after which feed-forward layers model element-wise interactions.
3 Adaptive Factorization Network
AFN adaptively learns cross features by transforming positive feature embeddings logarithmically, then combines the resulting interactions for prediction. Its architecture also supports DNN ensembling, arbitrary learned feature orders, and computational complexity based on the number of logarithmic neurons.
- Input Layer and Embedding Layer: AFN transforms raw inputs into positive feature embeddings before applying logarithmic transformations.A small positive value is added to zero embeddings to avoid numerical overflow.
- Logarithmic Transformation Layer: Each vector-wise logarithmic neuron learns feature powers as coefficients, allowing it to represent cross features of arbitrary orders.For example, coefficients of 1 for two fields and 0 for the others produce a second-order cross feature.
- Feed-forward Hidden Layers and Prediction: AFN concatenates the learned cross features and feeds them through fully connected hidden layers before producing the final prediction.ReLU activations are used in the hidden layers to capture nonlinear element-wise feature interactions.
- Optimization: For binary classification, AFN optimizes logarithmic loss, while other prediction tasks require corresponding objective functions.The paper focuses on binary classification and uses Adam for optimization.
- Ensemble AFN with DNNs: AFN+ combines separately trained AFN and DNN predictions, using separate embedding layers because AFN embeddings must remain positive.The separation slightly increases model complexity but leads to better performance according to the experiments.
- Discussions: AFN can learn decimal or negative feature orders, and its transformation layer has complexity O(kmN).Here k is embedding rank, m is the number of feature fields, and N is the number of logarithmic neurons.
4 Experiments
Experiments evaluate AFN and AFN+ across four datasets, compare hyperparameter settings, and analyze learned feature orders and selected cross features. AFN performs best or competitively across datasets, while analyses show adaptive-order interactions and logarithmic transformations contribute to performance.
- Experiments answer research questions about predictive performance, hyperparameter sensitivity, and whether AFN learns useful feature orders from data.
- Experimental Settings: AFN and AFN+ are evaluated on Criteo, Avazu, Movielens, and Frappe using AUC and Logloss, with 8:1:1 training, validation, and test splits.
- Performance Comparison (RQ1): AFN achieves the best or competing performance across all datasets; on Criteo and Frappe, it improves AUC by 0.0037 and decreases log loss by 0.0286 on average versus CIN.
- Performance Comparison (RQ1): AFN+ achieves the best performance on all four datasets and improves AUC by 0.003 and log loss by 0.012 on average versus xDeepFM.
- Hyperparameter Investigation (RQ2): AFN performance rises and then declines as logarithmic neurons or hidden-layer neurons increase, while hidden-layer depth is beneficial but not highly influential.
- Quantitative Analysis and Case Study (RQ3): Learned individual feature orders typically lie in [−1, 1], while final cross-feature orders span 4 to 10 rather than using a fixed order such as 2.
- Quantitative Analysis and Case Study (RQ3): With three logarithmic neurons on Frappe, AFN approximately learns the cross features (item id, is free, country), (user id, item id), and (item id, is free).
5 Conclusion
AFN learns arbitrary-order feature interactions adaptively from data rather than explicitly modeling all cross features under a fixed maximum order. It uses logarithmic transformation and feedforward networks, and experiments on four real-world datasets show superior predictive performance.
- AFN learns arbitrary-order feature interactions adaptively from data.It generates discriminative cross features and automatically learns their corresponding weights.
- A logarithmic transformation converts each feature’s power in a combination into a learnable coefficient.Feedforward neural networks then combine the learned cross features for prediction.
- AFN generalizes factorization machines and higher-order factorization machines with computational efficiency.
- Experiments on four real-world datasets demonstrate AFN’s superior predictive performance compared with state-of-the-art methods.