Source-linked AI summary
Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction
Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, Liang Wang
TL;DR
CTR models need to capture sophisticated interactions among multi-field features, but concatenating field embeddings treats those fields as an unstructured combination. Fi-GNN represents fields as graph nodes and interactions as edges, then models node interactions with a graph neural network; the paper reports superiority over state-of-the-art methods on two real-world datasets.
Problem
Existing CTR models combine field embeddings in an unstructured way, limiting flexible and explicit modeling of sophisticated feature interactions.
Method
Fi-GNN represents each feature field as a graph node, models field interactions through edges, and uses graph neural networks to model node interactions.
Results
Experiments on two real-world datasets show that Fi-GNN outperforms state-of-the-art methods and provides model explanations.
Takeaways & Limitations
Graph-structured representations provide a new paradigm for modeling multi-field feature interactions in CTR prediction.
Abstract
from arXiv · showhide
Click-through rate (CTR) prediction is an essential task in web applications such as online advertising and recommender systems, whose features are usually in multi-field form. The key of this task is to model feature interactions among different feature fields. Recently proposed deep learning based models follow a general paradigm: raw sparse input multi-filed features are first mapped into dense field embedding vectors, and then simply concatenated together to feed into deep neural networks (DNN) or other specifically designed networks to learn high-order feature interactions. However, the simple \emph{unstructured combination} of feature fields will inevitably limit the capability to model sophisticated interactions among different fields in a sufficiently flexible and explicit fashion. In this work, we propose to represent the multi-field features in a graph structure intuitively, where each node corresponds to a feature field and different fields can interact through edges. The task of modeling feature interactions can be thus converted to modeling node interactions on the corresponding graph. To this end, we design a novel model Feature Interaction Graph Neural Networks (Fi-GNN). Taking advantage of the strong representative power of graphs, our proposed model can not only model sophisticated feature interactions in a flexible and explicit fashion, but also provide good model explanations for CTR prediction. Experimental results on two real-world datasets show its superiority over the state-of-the-arts.
1 INTRODUCTION
CTR prediction depends on modeling interactions among multi-field categorical features. Existing approaches either capture only limited second-order interactions or learn higher-order interactions through unstructured, implicit combinations, motivating Fi-GNN’s graph-based formulation.
- CTR prediction estimates users’ probabilities of clicking ads or items, making it important for online advertising and recommender systems.
- FM and FFM model second-order feature interactions, but their linearity and interaction-order limits constrain representation power.
- DNN-based CTR models concatenate field embeddings to learn high-order interactions, but these interactions are learned implicitly and bit-wise with limited explanations.
- Fi-GNN represents feature fields as graph nodes whose edges model interactions, converting feature-interaction modeling into node-interaction modeling.The model updates node states recurrently through neighbor communication, with interaction steps corresponding to interaction order.
- Fi-GNN learns edge and node importance weights, supporting flexible interaction modeling and explanations for CTR prediction.
- The paper’s contributions are a graph representation of multi-field features, the Fi-GNN model, and experiments on two real-world datasets.The reported experiments claim state-of-the-art performance and model explanations.
2 RELATED WORK
Prior CTR work progresses from linear and factorization-based interaction models to deep and graph-based methods. Fi-GNN applies graph neural networks to feature interactions because graphs naturally represent nodes and their relationships.
- CTR research extensively studies feature interactions because they are central to CTR prediction.
- LR models first-order interactions, while FM, FFM, and AFM address second-order interactions with increasingly field-aware or weighted representations.
- DNN-based models concatenate field embeddings to learn high-order interactions using deeper structures and nonlinear activations.
- Graphs model objects as nodes and their relationships as edges, providing a structure for representing relational information.
- GNNs operate on graph domains by aggregating neighborhood information and updating node hidden states.GGNNs use GRU-based state updates, while GCNs are another GNN variant.
- Fi-GNN is based on GGNN and uses GNNs to model feature interactions on graph-structured features for CTR prediction.
3 OUR PROPOSED METHOD
The proposed method converts sparse multi-field categorical inputs into dense field embeddings, organizes fields as a feature graph, and applies Fi-GNN before attention-based CTR estimation.
- 3.1 Problem Formulation: The task uses m-field categorical features with binary click labels and predicts a click probability for each input.
- 3.2 Overview: The overview maps sparse inputs to dense field embeddings, represents fields as graph nodes, models node interactions with Fi-GNN, and estimates CTR with attention scoring.
- 3.3 Embedding Layer: Each categorical field is represented by a sparse one-hot vector and then embedded into a low-dimensional dense real-value field embedding.
- 3.3 Embedding Layer: The field embedding e_i belongs to R^d, where d denotes the embedding-vector dimension.
3.4 Multi-head Self-attention Layer
Fi-GNN uses multi-head self-attention to capture pairwise dependencies between feature fields across semantic subspaces and uses the resulting representations as initial graph-node states.
- Multi-head self-attention captures complex dependencies between feature-field pairs in different semantic subspaces.
- Scaled dot-product attention produces feature representations covering pairwise interactions for each attention head.
- The representations from all attention heads are concatenated to preserve pairwise feature interactions across semantic subspaces.
- The combined representation H1 is used as the initial node states of the graph neural network.
3.5 Feature Graph
Fi-GNN represents each multi-field input as a feature graph, turning feature-field interaction modeling into node interaction modeling.
- Each graph node corresponds to one feature field, with m nodes for m fields.
- The feature graph is weighted and fully connected because every pair of fields can interact.Edge weights represent the importance of different feature interactions.
3.6 Feature Interaction Graph Neural Network
Fi-GNN models feature-field interactions through recurrent message passing on the feature graph, combining attention-based aggregation, edge-wise transformations, GRU updates, and residual connections.
- Fi-GNN repeatedly aggregates transformed state information from neighboring nodes and updates each node’s state using its history.The initial node states come from multi-head self-attention representations.
- Attention-derived edge weights encode the importance of interactions between feature fields.The adjacency matrix stores these weights, which determine node interactions during aggregation.
- Edge-wise Transformation: Node-specific input and output matrices implement edge-wise transformations without assigning a separate transformation matrix to every edge.This makes the parameter count proportional to the number of nodes rather than the number of edges.
- State Update: After aggregation, GRU and residual connections update node states while supporting low-order feature reuse and gradient back-propagation.Residual connections are combined with GRU updates to retain low-order interactions alongside higher-order ones.
3.7 Attentional Scoring Layer
The attentional scoring layer converts neighborhood-aware final node states into a graph-level CTR prediction by weighting each field’s contribution.
- Final node states capture global information after the nodes have interacted through the graph.
- Two multilayer perceptrons estimate each field’s prediction score and its attentional weight.The weight represents the field’s influence on the overall prediction.
- The overall prediction is computed as the sum of the attention-weighted contributions from all field nodes.
3.8 Training
Fi-GNN trains its parameters by minimizing log loss with RMSProp, while balancing positive and negative samples within each training batch.
- The training objective is log loss over the training samples.N denotes the total number of training samples, and i indexes the samples.
- Parameters are updated by minimizing log loss using RMSProp.
- Each training batch contains equal numbers of randomly selected positive and negative samples.This addresses the unbalanced class proportions common in CTR datasets.
- The main learned parameters include two node-associated transformation matrices per feature field, attention-layer matrices, and GRU parameters.The node-associated matrices total 2m and scale with the number of nodes m.
3.9 Model Analysis
Fi-GNN represents multi-field CTR features as graph nodes and models their interactions through flexible, edge-specific mechanisms. It adds learned attention, edge-wise transformations, and residual updates to address limitations of prior interaction modeling.
- Prior CTR models concatenate dense field embeddings, imposing a fixed and unstructured interaction pattern.
- Fi-GNN converts feature-field interactions into node interactions on a graph.
- A residual connection alongside the GRU update helps recover low-order information by adding initial node states.
- Learned attention edge weights and edge-wise transformations let Fi-GNN model the importance and form of each interaction explicitly.
4 EXPERIMENTS
Experiments evaluate Fi-GNN on the Criteo and Avazu CTR datasets against representative first-, second-, and high-order interaction models, then examine its components, interaction depth, and learned feature relations. Fi-GNN achieves the best reported performance on both datasets, while ablations support the value of edge-wise interactions and residual connections.
- Experimental setup: Experiments evaluate Fi-GNN on the Criteo and Avazu CTR datasets using AUC and Logloss against representative baseline models.The baselines span logistic regression, factorization-machine methods, and deep models for high-order interactions.
- Model comparison: Second-order interaction models outperform logistic regression, while high-order interaction methods generally outperform second-order methods.The comparisons are presented as evidence that pairwise interactions help and that second-order interactions alone are insufficient.
- Model comparison: Fi-GNN achieves the best performance among all compared methods on both datasets, with especially strong superiority on Criteo.The paper notes that AUC improvements at the 0.001 level are regarded as significant in CTR prediction.
- Ablation study: Removing edge-wise interaction causes a large performance drop, and removing residual connections causes a further decline relative to the corresponding variants.The full Fi-GNN model outperforms the three ablation variants, indicating that the two improvements jointly boost performance.
- Ablation study: Edge-wise transformation produces greater improvement than attentional edge weights alone, while combining both measures gives the best ablation performance.The paper attributes the stronger effect to transformation matrices having more influence on interactions than scalar attentional edge weights.
5 CONCLUSIONS
The paper proposes graph-structured representations of multi-field features to address limitations of unstructured feature combinations in CTR models. Fi-GNN converts feature interaction modeling into node interaction modeling and supports global- and case-level explanations through attentional node weights.
- Previous CTR models treat multi-field features as an unstructured combination, limiting flexible and explicit modeling of sophisticated interactions.
- Figure 6 uses global- and case-level attentional node weights to reflect feature-field importance for final predictions.
- Fi-GNN represents each feature field as a graph node, with edges enabling interactions among fields.
- The model converts feature interaction modeling into node interaction modeling on graphs, establishing a new CTR prediction paradigm.