Source-linked AI summary
Sequential Recommendation with Graph Neural Networks
Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, Yong Li
TL;DR
Sequential recommendation must predict next interactions from noisy histories while users’ preferences change over time. SURGE reconstructs histories as metric-learned interest graphs, then dynamically fuses and pools them to extract activated core interests. Across two real-world datasets, it significantly outperforms state-of-the-art methods and remains effective and efficient on long sequences.
Problem
Long sequential histories contain implicit, noisy feedback, while rapidly changing preferences make current activated interests difficult to identify.
Method
SURGE reconstructs loose item sequences into tight item-item interest graphs, then uses graph neural propagation and dynamic pooling to extract activated core interests.
Results
SURGE consistently achieves the best performance, improving AUC by around 0.03 on Taobao and 0.04 on Kuaishou over all baselines.
Takeaways & Limitations
The method effectively and efficiently models long behavioral sequences, with performance remaining 0.8919 on Taobao and 0.8502 on Kuaishou for the longest histories.
Takeaways & Limitations
The paper leaves online A/B testing and integration of multiple behavior types, such as clicks and favorites, for future work.
Abstract
from arXiv · showhide
Sequential recommendation aims to leverage users' historical behaviors to predict their next interaction. Existing works have not yet addressed two main challenges in sequential recommendation. First, user behaviors in their rich historical sequences are often implicit and noisy preference signals, they cannot sufficiently reflect users' actual preferences. In addition, users' dynamic preferences often change rapidly over time, and hence it is difficult to capture user patterns in their historical sequences. In this work, we propose a graph neural network model called SURGE (short for SeqUential Recommendation with Graph neural nEtworks) to address these two issues. Specifically, SURGE integrates different types of preferences in long-term user behaviors into clusters in the graph by re-constructing loose item sequences into tight item-item interest graphs based on metric learning. This helps explicitly distinguish users' core interests, by forming dense clusters in the interest graph. Then, we perform cluster-aware and query-aware graph convolutional propagation and graph pooling on the constructed graph. It dynamically fuses and extracts users' current activated core interests from noisy user behavior sequences. We conduct extensive experiments on both public and proprietary industrial datasets. Experimental results demonstrate significant performance gains of our proposed method compared to state-of-the-art methods. Further studies on sequence length confirm that our method can model long behavioral sequences effectively and efficiently.
1 INTRODUCTION
Sequential recommendation must infer current preferences from long, noisy behavior histories despite rapidly changing interests. SURGE addresses these challenges with graph-based preference extraction and dynamic pooling, achieving significant gains and efficient long-sequence modeling.
- Challenges: Implicit feedback in long behavior sequences often provides noisy signals that do not reliably reflect users’ actual preferences.Clicks and watches may reflect items users do not genuinely prefer, adding noise to historical records.
- Challenges: Rapidly drifting preferences make it difficult to identify which extracted interests remain activated at the current time.Some preferences may be active while others have deactivated, even within the same history.
- Approach: SURGE reconstructs loose item sequences into item-item interest graphs and applies graph convolutional networks to extract implicit preference signals.The graph representation supports aggregation of weak signals into stronger preference representations.
- Approach: Dynamic graph pooling adaptively reserves users’ activated core preferences for predicting their next behavior.This pooling stage filters historical noise while retaining currently relevant interests.
- Results: SURGE shows significant performance improvements over state-of-the-art sequential recommenders and models long behavioral sequences effectively and efficiently.The paper reports experiments on two large-scale datasets collected from real-world applications.
2 PROBLEM FORMULATION
Sequential recommendation predicts a user’s next item from chronological implicit feedback in the user’s interaction history. The target item is the next interaction that matches the user’s preferences.
- Task Definition: Sequential recommendation predicts the next item x_n+1 from a chronological sequence of user-item interactions.The sequence contains interacted items x_1 through x_n, where n is the number of interactions.
- Task Definition: User preferences are inferred from chronological implicit feedback in the interaction sequence.The formulation treats historical behavior as the information source for next-item prediction.
3 METHODOLOGY
SURGE consists of graph construction, interest-fusion graph convolution, interest-extraction graph pooling, and subsequent modeling of the activated core-interest sequence.
- Interest Graph Construction: Interest graph construction converts loose item sequences into tight item-item graphs using metric learning.This explicitly integrates and distinguishes different preference types in long-term behavior histories.
- Interest Fusion: The interest-fusion graph convolutional layer dynamically combines interests while strengthening important behaviors and weakening noise.Graph convolution propagates information across the constructed interest graph.
- Interest Extraction: The interest-extraction graph pooling layer extracts activated core interests from the fused graph for downstream recommendation modeling.The pooled graph is flattened into a currently activated core-interest sequence.
3.1 Interest Graph Construction
SURGE learns sparse item-item interest graphs from interaction sequences so dense subgraphs can represent core interests. Metric learning and relative ranking address connectivity, sparsity, and noise concerns.
- Interest Graph Construction: Metric learning converts loose interaction sequences into tight item-item interest graphs to distinguish different preference types.The learned graph explores the distribution of interests for each interaction sequence.
- Graph Representation: Each graph contains item vertices and learned edges represented by an adjacency matrix indicating item relatedness.The adjacency matrix A has dimensions n×n, with one vertex for each interacted item.
- Interest Structure: Core-interest nodes tend to have higher degree, while denser and larger subgraphs represent recurring similar interests.The construction assumes neighboring nodes are similar and dense subgraphs correspond to core interests.
- Metric Learning: The graph-learning problem is transformed into node-similarity metric learning jointly trained with the downstream recommendation task.The method is described as general, easy to implement, and compatible with inductive learning for new test-time items.
- Metric Learning: SURGE uses multi-head metric learning to compute independent similarity matrices from multiple semantic perspectives and average them.Each head uses a separate trainable weight vector to capture a different perspective.
- SURGE Pipeline: The resulting graph is dynamically fused and pooled so its activated core-interest sequence can support further modeling and prediction.Figure 1 presents construction, fusion, extraction, and flattening as the model pipeline.
- Graph Sparsification: Sparsification avoids fully connected graphs that are computationally expensive and may introduce unimportant edges that obscure relevant graph structure.Uniform per-node sparsity is also contrasted with preserving the graph’s natural dense or sparse distribution.
- Graph Sparsification: Relative ranking sparsifies the metric matrix into a symmetric, non-negative adjacency matrix by retaining the strongest node-pair connections.The threshold is determined by ranking metric values across the graph, with ε controlling overall sparsity.
3.2 Interest-fusion Graph Convolutional Layer
The layer strengthens important interest signals while weakening noise by aggregating item representations with attention based on both cluster centrality and query relevance.
- Cluster- and query-aware aggregation: The cluster- and query-aware convolution separates diverse interests and emphasizes signals associated with core and query-related interests.It redistributes edge weights during message passing to strengthen important signals and weaken noise.
- Output representation: A residual connection, nonlinearity, and multi-head attention produce refined node embeddings from weighted neighbor combinations.The independent attention heads concatenate their outputs, yielding embeddings with dimension 𝜙d′.
- Cluster-aware attention: Cluster attention compares each target node with the average embedding of its k-hop neighborhood to assess cluster centrality.The target node is treated as a cluster medoid, while its neighborhood defines the cluster receptive field.
- Query-aware attention: Query attention gives greater aggregation weight to source nodes more correlated with the target item and discards irrelevant information.This makes the aggregation focus on behaviors relevant to the current query item.
- Attention coefficients: The layer combines cluster and query scores additively, normalizes them with softmax, and uses the resulting coefficients for propagation.With self-loops, separate coefficients control information received by the target and information sent by the source.
3.3 Interest-extraction Graph Pooling Layer
The pooling layer coarsens the interest graph into clusters, preserving important signals and temporal order while addressing assignment and ordering difficulties through regularization.
- Graph coarsening: Graph pooling transforms loose interest structure into a tightly coarsened graph representing stronger user-interest signals.It downsizes the graph by aggregating node information into clusters.
- Soft clustering: A soft assignment matrix S pools n node embeddings and scores into m cluster embeddings and scores, where m < n.The importance score γ_i is obtained from β_i through softmax, and S assigns nodes probabilistically to clusters.
- Assignment learning: A GNN learns S through message passing and softmax, while S^TAS constructs connectivity among pooled clusters and supports hierarchical pooling.Repeating the pooling equations enables multi-layer compression of the interest graph.
- Optimization challenge: The assignment matrix is difficult to train from recommendation gradients alone because its non-convex optimization can reach early local optima.This motivates adding regularization to the pooling objective.
- Assignment regularization: Three regularizers encourage graph-connected nodes to share clusters, make assignments nearly one-hot, and preserve temporal order across pooling.The same-mapping term aligns assignments with adjacency strength, entropy reduces affiliation uncertainty, and position regularization favors order-preserving assignments.
- Graph readout: A weighted raw-graph readout aggregates post-propagation node embeddings using node importance scores and feeds the graph representation to prediction.The implementation uses sum readout for permutation invariance while retaining cluster information from pooling.
3.4 Prediction Layer
The prediction layer models the evolution of pooled interests, combines current-interest representations with the target item, and estimates the probability of the next interaction.
- Interest evolution: Core interests evolve over time, so the model supplements graph readout with a sequence model over temporally ordered pooled cluster embeddings.Relative position regularization lets the pooled graph function as a reduced sequence with enhanced interest signals.
- Interest evolution: AUGRU uses fused-interest importance scores to scale update gates, reducing the effect of less-related interests on the hidden state.This is intended to reduce disturbance from interest drifting and support smoother interest evolution.
- Prediction function: The model concatenates graph-level and evolution representations with the target-item embedding, then applies fully connected layers to predict interaction probability.The prediction function is a two-layer feedforward neural network.
- Training objective: Training minimizes negative log-likelihood with L2 regularization, using positive and negative instances and predicted next-item click probabilities.The three assignment regularizers are also added to the final recommendation objective.
4 EXPERIMENT
The experiments evaluate SURGE against sequential and non-sequential recommenders on public and industrial datasets using classification and ranking metrics. They also examine sequence-length handling, component effects, and implementation settings.
- The evaluation addresses comparative performance, sequence-length effectiveness, and component contributions through three research questions.
- Experiments use Taobao e-commerce and Kuaishou short-video interaction data, with dataset-specific temporal splits for training, validation, and testing.
- Performance is assessed with AUC, GAUC, MRR, and NDCG@2, covering classification ability and ranking quality.
- SURGE is compared with static-interest models including NCF, DIN, and LightGCN, alongside sequential models such as Caser, GRU4REC, and DIEN.
- The experiments implement models in TensorFlow with Adam, batch size 500, embedding size 40, and two-layer interaction estimators.
- Sequence lengths are capped at 50 for Taobao and 250 for Kuaishou, with pooling lengths searched across dataset-specific ranges.
4.2 Overall Performance (RQ1)
SURGE consistently outperforms the baselines on classification and ranking metrics, with larger gains on Kuaishou’s longer histories. It also maintains strong long-sequence performance and improves training efficiency through sequence compression and noise filtering.
- Overall performance: 0.03 AUC gain on Taobao and 0.04 AUC gain on Kuaishou distinguish SURGE from all baselines, with both improvements statistically significant.Both comparisons report p-value < 0.001.
- Sequence length: Most models improve with moderately longer sequences but decline at greater lengths as noise signals accumulate.DIN and DIEN show the most significant declines, while short sequences remain difficult because of data sparsity.
- Sequence length: SURGE’s performance advantage over other methods increases as interaction-sequence length grows.The longest-history groups still achieve GAUC values of 0.8919 on Taobao and 0.8502 on Kuaishou.
- Sequence length: 0.8919 and 0.8502 are SURGE’s reported performances for the longest-history groups on Taobao and Kuaishou, respectively.The method retains these values despite the additional noise in long behavioral histories.
- Efficiency comparison: More than 20% efficiency improvement over all baselines except DIN on Kuaishou results from pooling sequences before recurrent processing.Pooling reduces recurrent steps, while noise filtering leaves a compressed sequence containing core interests.
4.4 Ablation and Hyper-parameter Study (RQ3)
The ablations show that graph convolutional fusion, interest extraction, and compressed-sequence modeling each contribute to SURGE's recommendation performance.
- Interest fusion: Graph convolution improves performance by fusing weak signals into stronger signals, while core-interest enhancement and target-interest activation add further gains.The comparison includes no propagation, cluster-aware propagation, query-aware propagation, and their combination.
- Interest extraction: Interest extraction filters irrelevant noise and focuses modeling on critical information.Graph pooling with assignment regularization and weighted readout better compresses user interests and improves recommendation performance.
- Interest evolution: AUGRU performs slightly better than the other sequential models because it can use the cluster score from interest extraction.The results compare different prediction layers applied after pooling.
- Interest evolution: Compressed sequences benefit all tested prediction layers, including DIN, GRU4Rec, DIEN, and SLi-Rec.The compared layers are Attention (DIN), GRU (GRU4Rec), AUGRU (DIEN), and TIME4LSTM (SLi-Rec).
- Overall findings: Experiments on two real-world datasets verify that SURGE outperforms existing recommendation methods and models long sequences effectively and efficiently.The conclusion reports both comparative effectiveness and sequence-length findings.
5 RELATED WORK
Related sequential-recommendation methods model evolving preferences through recency weighting, recurrent networks, or attention, while SURGE instead represents weak signals and activated interests with graphs.
- Sequential Recommendation: Sequential recommendation predicts users' next behaviors from historical sequences while modeling preferences that evolve over time.The task differs from static recommendation because preferences can change, such as during a World Cup period.
- Sequential Recommendation: Earlier methods use hand-designed rules or attention to assign time-decaying weights, whereas recurrent models struggle with long-range dependencies.The related-work discussion identifies these as approaches to fast-changing short-term preferences.
- Sequential Recommendation: SURGE argues that sequential behaviors provide weak preference signals and that some preferences may be deactivated at a given time.This perspective differs from simply dividing interests into long-term and short-term parts.
- Graph Neural Networks for Recommendation: Unlike prior graph-recommendation applications, SURGE uses graph convolution to fuse weak signals and graph pooling to extract dynamically activated core preferences from long sequences.The model constructs item-item interest graphs for this purpose.
6 CONCLUSIONS AND FUTURE WORK
SURGE reconstructs behavioral sequences as interest graphs and uses graph neural networks to fuse and extract activated core interests. Experiments support its effectiveness and long-sequence modeling, while future work includes online testing and richer behavior types.
- Conclusions: SURGE reconstructs loose item sequences into tight item-item interest graphs to model sequential recommendation.The graph-based framework targets noisy user behavior sequences.
- Conclusions: Graph neural networks dynamically fuse and extract users' activated core interests from noisy behavior sequences.This summarizes the model's central use of graph representation and propagation.
- Conclusions: Experiments on public and proprietary industrial datasets demonstrate SURGE's effectiveness, and sequence-length studies support effective and efficient modeling of long behavioral sequences.The conclusion reports both dataset-based evaluation and long-sequence analysis.
- Future Work: Future work will evaluate SURGE through online A/B tests and incorporate multiple behavior types such as clicks and favorites.These directions aim to assess online recommendation performance and explore finer-grained interactions.