AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks

Weiping Song; Chence Shi; Zhiping Xiao; Zhijian Duan; Yewen Xu; Ming Zhang; Jian Tang

Source-linked AI summary

AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks

Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, Jian Tang

arXiv:1810.11921v2Oct 29, 2018cs.IRcs.AIcs.LG

TL;DR

CTR prediction must handle sparse, high-dimensional features and difficult-to-engineer high-order interactions. AutoInt embeds numerical and categorical features into a shared low-dimensional space and explicitly models interactions with multi-head self-attention and residual connections. On four real-world datasets, it reports strong predictive performance and explainability, while future work targets contextual information and broader tasks.

Problem
CTR prediction is challenging because inputs are sparse and high-dimensional, while useful high-order feature combinations are costly to hand-craft and impossible to enumerate fully.
Method
AutoInt maps numerical and categorical features into a shared low-dimensional space and uses multi-head self-attention with residual connections to explicitly learn high-order interactions.
Results
AutoInt achieves the best performance among compared models, is more efficient during online inference than CIN, and provides explainability through learned combinatorial features.
Takeaways & Limitations
The method offers an end-to-end approach for explicit, explainable high-order interaction learning in CTR prediction, with improved offline AUC and Logloss when combined with implicit interactions.
Takeaways & Limitations
Future work is needed to incorporate contextual information, improve online recommender-system performance, and extend AutoInt to regression, classification, and ranking.

Abstract

from arXiv · show

Click-through rate (CTR) prediction, which aims to predict the probability of a user clicking on an ad or an item, is critical to many online applications such as online advertising and recommender systems. The problem is very challenging since (1) the input features (e.g., the user id, user age, item id, item category) are usually sparse and high-dimensional, and (2) an effective prediction relies on high-order combinatorial features (\textit{a.k.a.} cross features), which are very time-consuming to hand-craft by domain experts and are impossible to be enumerated. Therefore, there have been efforts in finding low-dimensional representations of the sparse and high-dimensional raw features and their meaningful combinations. In this paper, we propose an effective and efficient method called the \emph{AutoInt} to automatically learn the high-order feature interactions of input features. Our proposed algorithm is very general, which can be applied to both numerical and categorical input features. Specifically, we map both the numerical and categorical features into the same low-dimensional space. Afterwards, a multi-head self-attentive neural network with residual connections is proposed to explicitly model the feature interactions in the low-dimensional space. With different layers of the multi-head self-attentive neural networks, different orders of feature combinations of input features can be modeled. The whole model can be efficiently fit on large-scale raw data in an end-to-end fashion. Experimental results on four real-world datasets show that our proposed approach not only outperforms existing state-of-the-art approaches for prediction but also offers good explainability. Code is available at: \url{https://github.com/DeepGraphLearning/RecommenderSystems}.

1 INTRODUCTION

CTR prediction is important but difficult because sparse, high-dimensional inputs require meaningful high-order feature interactions that are costly to engineer or enumerate. AutoInt addresses this by embedding features and explicitly learning interactions with multi-head self-attention, while retaining explainability.

CTR prediction directly affects business revenue in online advertising and recommender systems.
Sparse, high-dimensional, and categorical input features make CTR prediction challenging for standard supervised learning.One-hot encoding can produce extremely large feature vectors.
Meaningful high-order combinations are difficult to hand-craft, while enumerating them exponentially increases dimensionality and sparsity and can worsen overfitting.The paper illustrates this with a third-order user-profile and product-category combination.
AutoInt embeds categorical and numerical features into a shared low-dimensional space and uses multi-head self-attention to model their interactions.The shared space enables different feature types to interact through vector arithmetic.
Attention measures feature correlations, supporting explicit high-order interaction learning and model explainability.
Experiments on real-world datasets report that AutoInt outperforms existing state-of-the-art approaches for CTR prediction while offering explainability.

2 RELATED WORK

Prior CTR and feature-interaction methods range from low-order factorization models to implicit deep networks and attention-based architectures. The paper positions AutoInt within this progression by targeting explicit, interpretable high-order interactions.

CTR prediction is widely studied in online advertising and recommender systems, including context-aware and hierarchical importance-aware models.
Factorization-machine variants model mainly first- and second-order interactions, including field-aware and importance-aware relationships.
Deep CTR models such as NFM, PNN, FNN, DeepCrossing, Wide&Deep, and DeepFM learn high-order interactions implicitly and therefore lack good explainability.
Attention mechanisms model dependencies across inputs, while multi-head self-attention captures complicated dependencies in separate attention heads.
Residual connections are used to support gradient flow through intermediate layers and facilitate training of very deep neural networks.

3 PROBLEM DEFINITION

The paper defines CTR prediction over sparse user and item features and formulates the goal as learning low-dimensional representations that model high-order combinatorial features. AutoInt’s overview connects this representation to click-probability estimation.

CTR prediction estimates the probability that a user clicks an item from a concatenated feature vector containing one-hot categorical features.
Directly applying standard classifiers to the sparse, high-dimensional raw vector can lead to overfitting, motivating low-dimensional continuous representations.
AutoInt’s model overview comprises an embedding layer and an interacting layer, whose details are presented separately in Figures 2 and 3.
A p-order combinatorial feature combines features from distinct fields with a non-additive function such as multiplication or outer product.
The problem objective is to learn a low-dimensional representation of the input that models high-order combinatorial features.

4 AUTOINT: AUTOMATIC FEATURE INTERACTION LEARNING

AutoInt embeds categorical and numerical fields into a shared low-dimensional space, then uses stacked multi-head self-attention with residual connections to learn explicit feature interactions of increasing order.

4.1 Overview: AutoInt maps sparse categorical and numerical inputs into a shared low-dimensional embedding space for cross-type interactions.Categorical fields use embedding vectors, including averaged embeddings for multi-valued fields; numerical fields are represented in the same space.
4.4 Interacting Layer: Multi-head key-value attention weights feature correlations and combines relevant fields into learned combinatorial features.Inner products are used as the attention function, while separate heads learn distinct interaction subspaces.
4.4 Interacting Layer: Residual connections preserve raw and previously learned lower-order features while combining the outputs of multiple attention heads.A projection matrix handles dimension mismatches, followed by a ReLU activation.
4.4 Interacting Layer: Multiple interacting layers model increasingly high-order combinations by feeding each layer’s output into the next.The paper states that stacking layers can model arbitrary-order combinatorial features.
Complexity Analysis: The interacting layer has space complexity O(Ldd′H), independent of the number of feature fields M, and runs in O(MHd′(M + d)) time.The method is described as memory-efficient because H, d, and d′ are usually small; the experiments use H = 2 and d′ = 32.

5 EXPERIMENT

The experiment evaluates AutoInt on CTR prediction, efficiency, configuration effects, feature dependencies, explainability, and the value of integrating implicit feature interactions.

Research Questions: RQ1 asks whether AutoInt performs well on CTR prediction and remains efficient for large-scale sparse, high-dimensional data.
Research Questions: RQ2 examines how different model configurations influence AutoInt’s behavior and performance.
Research Questions: RQ3 investigates feature dependency structures and whether AutoInt provides explainable predictions.
Research Questions: RQ4 tests whether integrating implicit feature interactions further improves performance.

5.1 Experiment Setup

The evaluation uses four public datasets, AUC and Logloss, and baselines spanning linear, second-order, and high-order interaction models, with specified implementation settings and comparisons.

Evaluation Metrics: AUC measures positive-versus-negative ranking probability, while Logloss is used because all models minimize it.Higher AUC indicates better performance, and AUC or Logloss differences at the 0.001 level are considered significant for CTR prediction.
Compared Methods: The baselines cover linear models, factorization-machine methods for second-order interactions, and techniques designed to capture high-order interactions.Compared methods include LR, FM, AFM, DeepCrossing, NFM, CrossNet, CIN, and HOFM.
Implementation Details: AutoInt uses embedding dimension d = 16, three interacting layers, hidden size d′ = 32, and two attention heads per layer by default.Dropout is selected for MovieLens-1M but is unnecessary for the other three large datasets.
Effectiveness Comparison: AutoInt almost outperforms all baselines across four datasets and both evaluation metrics.The table caption summarizes the effectiveness comparison; significance is reported against the strongest baseline for Criteo, KDD12, and MovieLens-1M.

5.2 Quantitative Results (RQ1)

AutoInt achieves the strongest overall prediction performance while remaining efficient in runtime and model size. Results also indicate that explicit attention-based feature interactions distinguish it from implicit high-order interaction models.

Prediction performance: One attention head performs worse than two heads, while adding more heads yields no significant further improvement.
Prediction performance: AutoInt achieves the best performance overall among the compared models on three of four real-world datasets.CIN performs slightly better on Avazu in AUC, while AutoInt obtains lower Logloss.
Prediction performance: Deep neural models that learn high-order interactions implicitly do not consistently improve over FM and AFM, whereas explicit interaction modeling with CIN performs consistently better.
Efficiency: AutoInt is comparable in runtime to efficient methods DeepCrossing and NFM, unlike the more time-consuming CIN.CIN’s complicated crossing layer may make it impractical in industrial scenarios.
Efficiency: AutoInt uses much fewer parameters than CIN, the strongest baseline model, while achieving the best overall performance.The counted parameters in Table 3 exclude the embedding layer.

5.3 Analysis (RQ2)

Ablation and sensitivity analyses examine residual connections, interacting-layer depth, and embedding dimension. Higher-order interactions improve performance, while embedding-size effects depend on dataset scale and can reflect overfitting.

Residual connections: Removing residual connections decreases performance on all datasets, with large declines on KDD12 and MovieLens-1M.Residual connections carry previously learned combinatorial features and support very high-order combinations.
Interacting layers: One interacting layer dramatically improves performance over using no interacting layer, and additional layers further improve performance by modeling higher-order combinations.Figure 5 reports this trend on both evaluated datasets, with results for Criteo and Avazu omitted because they are similar.
Embedding dimensions: On KDD12, performance continuously increases with embedding dimension, whereas on MovieLens-1M it decreases after dimension 24 because the model overfits.The contrasting behavior is attributed to the smaller size of MovieLens-1M.

5.4 Explainable Recommendations (RQ3)

AutoInt uses attention scores to expose meaningful individual and global feature relationships in MovieLens-1M recommendations. These visualized interactions provide interpretable combinatorial features and field-level rules.

Visualization: The explainability analysis uses heat maps whose axes are the seven feature fields Gender, Age, Occupation, Zipcode, RequestTime, RealeaseTime, and Genre.
Case-level interactions: Attention scores identify the meaningful feature combination <Gender=Male, Age=[18-24), MovieGenre=Action&Triller> for a recommendation.The paper describes this combination as reasonable because young men are likely to prefer action and thriller movies.
Global-level interactions: Average attention scores across the dataset reveal strong correlations among Gender–Genre, Age–Genre, RequestTime–ReleaseTime, and Gender–Age–Genre.The paper presents these correlations as explainable recommendation rules.

5.5 Integrating Implicit Interactions (RQ4)

AutoInt is combined with feed-forward networks to capture implicit feature interactions, improving performance across all datasets. The combined AutoInt+ model achieves state-of-the-art CTR prediction results.

The comparison includes Wide&Deep, DeepFM, Deep&Cross, and xDeepFM as competing feed-forward-based CTR models.These methods combine feed-forward neural networks with logistic regression, factorization machines, or explicit interaction modules.
Joint training with feed-forward neural networks improves AutoInt’s performance on all datasets.The authors attribute this improvement to integrating implicit feature interactions.
AutoInt+ outperforms all competitive methods after integrating implicit feature interactions.The joint model achieves new state-of-the-art performance on the evaluated CTR prediction datasets.
AutoInt’s individual model remains powerful because its improvement from joint training is fairly small compared with other models.

6 CONCLUSION AND FUTURE WORK

The paper concludes that self-attention explicitly learns high-order feature interactions while supporting efficient CTR prediction and explainability. Future work targets contextual information, online recommender systems, and broader machine learning tasks.

AutoInt uses self-attention to automatically learn high-order feature interactions explicitly.Its interacting layer lets each feature interact with others and learn their relevance.
Experiments on four real-world datasets demonstrate the effectiveness and efficiency of the proposed model.
Visualizing learned combinatorial features provides model explainability.
Future work will incorporate contextual information and improve performance for online recommender systems.The authors also plan to extend AutoInt to regression, classification, and ranking tasks.