Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Ziwei Fan; Zhiwei Liu; Jiawei Zhang; Yun Xiong; Lei Zheng; Philip S. Yu

Source-linked AI summary

Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer

Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, Philip S. Yu

arXiv:2108.06625v2Aug 14, 2021cs.IRcs.AIcs.LG

TL;DR

Existing sequential recommendation methods often omit temporal collaborative signals, despite their coexistence with sequential patterns in evolving interactions. TGSRec combines a continuous-time bipartite graph, Temporal Collaborative Transformer, and graph propagation; on five datasets, it significantly outperforms existing models.

Problem
Existing sequential recommendation methods often ignore temporal collaborative signals latent in evolving user-item interactions alongside sequential patterns.
Method
TGSRec uses a continuous-time bipartite graph, temporal embeddings, a Temporal Collaborative Transformer, and graph propagation to jointly model sequential and collaborative signals.
Results
TGSRec consistently and significantly outperforms all baselines across five datasets, with 22.51% absolute gain in Recall@10 over the second best.
Takeaways & Limitations
The results support unifying sequential patterns with temporal collaborative signals for continuous-time sequential recommendation.
Takeaways & Limitations
The studied setting requires future timestamps for prediction and permits any history interactions before testing, while parameters are learned only from training data.

Abstract

from arXiv · show

In order to model the evolution of user preference, we should learn user/item embeddings based on time-ordered item purchasing sequences, which is defined as Sequential Recommendation (SR) problem. Existing methods leverage sequential patterns to model item transitions. However, most of them ignore crucial temporal collaborative signals, which are latent in evolving user-item interactions and coexist with sequential patterns. Therefore, we propose to unify sequential patterns and temporal collaborative signals to improve the quality of recommendation, which is rather challenging. Firstly, it is hard to simultaneously encode sequential patterns and collaborative signals. Secondly, it is non-trivial to express the temporal effects of collaborative signals. Hence, we design a new framework Temporal Graph Sequential Recommender (TGSRec) upon our defined continuous-time bi-partite graph. We propose a novel Temporal Collaborative Trans-former (TCT) layer in TGSRec, which advances the self-attention mechanism by adopting a novel collaborative attention. TCT layer can simultaneously capture collaborative signals from both users and items, as well as considering temporal dynamics inside sequential patterns. We propagate the information learned fromTCTlayerover the temporal graph to unify sequential patterns and temporal collaborative signals. Empirical results on five datasets show that TGSRec significantly outperforms other baselines, in average up to 22.5% and 22.1%absolute improvements in Recall@10and MRR, respectively.

1 INTRODUCTION

Sequential recommendation methods model item transitions but often omit temporal collaborative signals in evolving user-item interactions. TGSRec addresses this gap by combining continuous-time graph modeling, temporal collaborative attention, and graph propagation.

Existing sequential recommendation methods use time-ordered purchasing sequences to predict future items but often ignore temporal collaborative signals.
A toy example shows collaborative context changing the recommendation from i3, favored by sequence counts, to i4, favored by similar user histories.Users u2 and u4 share interactions with i1 and i2, making u2’s sequence more relevant to u4.
Two challenges are simultaneously encoding sequential and collaborative signals and expressing how temporal gaps affect collaborative importance.Existing methods may model item order without time differences or fail to generalize to unseen timestamps.
TGSRec uses a Temporal Collaborative Transformer with target-node queries and connected-neighbor keys and values to learn interaction importance.The TCT layer is designed to model collaborative signals and temporal correlations within sequences.
The proposed continuous-time bipartite graph propagates temporal collaborative information around nodes, unifying it with sequential patterns.
TGSRec is presented as using dynamic temporal node embeddings inferred at specified timestamps, with experiments conducted on five real-world datasets.The reported experiments demonstrate state-of-the-art performance and effectiveness in modeling temporal collaborative signals.

2 RELATED WORK

Related work spans recurrent, attention-based, temporal, and graph-based recommendation models. The paper positions TGSRec at the intersection of sequential modeling, continuous temporal information, and collaborative graph structure.

Sequential Recommendation: Early sequential recommendation models use recurrent architectures such as GRU4Rec, hierarchical RNNs, and LSTMs to model shopping sequences.
Sequential Recommendation: Attention-based models including SASRec and BERT4Rec apply transformer mechanisms to weight or encode items in recommendation sequences.
Temporal Information: Temporal recommendation research incorporates time through biases, tensor dimensions, temporal graphs, random walks, attention, recurrent updates, or time vectors.
Temporal Information: Existing temporal sequential models consider absolute or relative time, but they do not unify temporal collaborative signals with sequential patterns.
Graph-based Recommender Systems: Graph-based recommender systems capture structural information and collaborative signals that are not directly represented by sequence-only learning.

3 DEFINITIONS AND PRELIMINARIES

The paper formulates recommendation over a continuous-time user-item bipartite graph whose timestamped edges preserve interaction order. Recommendations are generated for users at specified future timestamps.

Connected neighbors preserve users’ and items’ sequential order through timestamps attached to graph edges.
A Continuous-Time Bipartite Graph represents users and items as disjoint node sets connected by timestamped interaction edges.Each edge records a user-item interaction and its timestamp.
At timestamp t, continuous-time recommendation ranks items outside the user’s previously interacted set, placing interested items higher.
Continuous-time sequential recommendation makes a recommendation for each timestamp in a user’s set of future timestamps.
Unlike position-only sequential recommendation, this formulation explicitly includes timestamps, which may be any real value.Using position numbers reduces the setting to order-based recommendation.

4 PROPOSED MODEL

TGSRec is organized around node and timestamp embeddings, a Temporal Collaborative Transformer, and graph-based information propagation. Together, these components target unified modeling of sequential and temporal collaborative signals.

The embedding layer encodes nodes and timestamps consistently to connect sequential recommendation with graph embedding.
The Temporal Collaborative Transformer uses temporal collaborative attention to discriminate the temporal impacts of neighbors.
Graph information propagation completes the framework for unifying sequential patterns with temporal collaborative signals.

4.1 Embedding Layer

TGSRec uses long-term node embeddings together with continuous-time edge embeddings to represent structural information and temporal effects in the interaction graph.

Embedding Types: The embedding layer encodes both long-term node states and continuous-time timestamp representations.Node embeddings initialize inference, while timestamp embeddings represent temporal information on edges.
Long-Term Embeddings: Long-term user and item embeddings represent holistic structural information in the continuous-time bipartite graph.They are parameterized as vectors and serve as node features optimized from the graph structure.
Continuous-Time Encoding: Continuous-time encoding maps scalar timestamps into vectors in R^d_T.The mapping is defined as Φ: T → R^d_T, with T in the positive real numbers.
Temporal Effects: Temporal effects are represented as the dot product of encoded timestamps through a temporal kernel.The kernel directly models time representations and supports temporal correlation between timestamps.

4.2 Temporal Collaborative Transformer

The TCT layer combines long-term node embeddings, temporal information, and neighbor interactions to infer temporal user or item embeddings. Its temporal collaborative attention weights historical interactions using both neighboring information and edge timestamps, then aggregates the result through stacked layers.

4.2.1 Information Construction: TCT constructs layer inputs by concatenating long-term or prior-layer node embeddings with a timestamp vector.Concatenation unifies temporal information with collaborative signals; the first layer uses long-term embeddings, while later layers use outputs from the previous TCT layer.
4.2.2 Information Propagation: For a user query, TCT samples historical user-item interactions before the target time and propagates their information to infer the user’s temporal embedding.Neighbor information includes sampled items and their interaction timestamps; the same construction generalizes analogously to item queries.
4.2.3 Temporal Collaborative Attention: Temporal collaborative attention assigns each sampled interaction an importance weight based on the query, neighbor information, and temporal information.The weights represent the impact of historical interactions on temporal inference and are normalized across sampled interactions with softmax.
4.2.3 Temporal Collaborative Attention: Unlike self-attention, the mechanism jointly models user-item interactions and temporal information rather than only item-item correlations.Its dot-product form supports multi-head attention, whose outputs are concatenated for aggregation.
4.2.4 TCT Output: The layer concatenates query and aggregated neighbor information, passes them through an FFN, and outputs a temporal embedding for the next layer or prediction.Stacked layers entangle collaborative signals with temporal effects, and the resulting item-query formulation is analogous to the user-query formulation.

4.3 Model Prediction

TGSRec produces temporal embeddings for users and items at a specified timestamp, scores candidate items by their embedding dot product, and ranks them for recommendation.

4.3 Model Prediction: For each test triplet, the model obtains final-layer temporal embeddings for both the user and item at the specified timestamp.The model contains L TCT layers and uses the final-layer embeddings for prediction.
4.3 Model Prediction: The recommendation score r(u,i,t) is the dot product of the user and item temporal embeddings at time t.Scores are calculated for all candidate items and sorted to produce a ranking list.
4.3 Model Prediction: Continuous-time embeddings allow TGSRec to infer user and item representations at arbitrary timestamps and support multiple-step recommendation.The model ranks candidate items at the given timestamp rather than only predicting the next item.

4.4 Model Optimization

TGSRec is trained with pairwise Bayesian Personalized Ranking loss using observed interactions as positives and unobserved items as sampled negatives, with Adam optimization; BCE is also evaluated.

4.4 Model Optimization: The primary objective is pairwise BPR loss, which assumes observed implicit-feedback items should receive higher scores than unobserved items.BPR is used for ranking-based top-N recommendation.
4.4 Model Optimization: Training samples contain a positive user-item interaction from the CTBG edge set and a negative item sampled from items unobserved by that user at the timestamp.The learnable parameters include long-term embeddings, time-embedding parameters, and linear transformation matrices.
4.4 Model Optimization: The loss is optimized with mini-batch Adam using an adaptive learning rate.The model can alternatively be optimized with binary cross-entropy loss, which is compared with BPR in experiments.

5 EXPERIMENTS

Experiments evaluate TGSRec on five real-world datasets against static, temporal, and sequential recommendation baselines using continuous-time evaluation and ranking metrics. TGSRec consistently outperforms the baselines, with transformer structure, temporal information, and collaborative modeling contributing to performance.

Experimental Setup: Experiments use four Amazon review datasets and MovieLens ML-100K, chronologically split into 80% training, 10% validation, and 10% test interactions.All datasets use Unix timestamps.
Baselines: The comparison includes static models, temporal models, transformer-based sequential recommenders, and other sequential recommendation models.Baselines include BPRMF, LightGCN, CTDNE, TiSASRec, SASRec, BERT4Rec, SSE-PT, FPMC, GRU4Rec, Caser, and SR-GNN.
Experimental Setup: Continuous-time evaluation allows models to use any history before each test timestamp, while parameters are learned only from training data.Historical interactions may come from training, validation, or testing portions during prediction.
Evaluation: Evaluation reports Recall@N, NDCG@N, and MRR for N equal to 10 or 20, using sampled negatives with an unbiased rank estimator.One thousand negative items are sampled for evaluation instead of the full negative-item set.
Results: 22.51%, 16.90%, and 22.15% are TGSRec’s absolute gains over the second-best method for Recall@10, Recall@20, and MRR, respectively.TGSRec also significantly outperforms other methods in NDCG across datasets.
Results: Transformer-based sequential methods outperform other baseline types, while TGSRec remains substantially better than them by modeling temporal collaborative effects.The reported explanation highlights temporal collaborative signals, explicit temporal effects, and propagation across stacked TCT layers.

5.4 Parameter Sensitivity (RQ2)

Parameter and component analyses examine TGSRec’s sensitivity to layers, embedding size, neighbors, temporal construction, and attention mechanisms. The results support temporal inference, temporal kernels, learned temporal effects, and appropriate model capacity.

Number of Layers: With L=0, TGSRec performs worst across datasets because it has no TCT layer and cannot infer temporal embeddings.Adding one layer enables temporal inference, while additional propagation is examined through deeper configurations.
Embedding Size: Performance increases with embedding size but drops at d=64, which the authors attribute to over-fitting from too many parameters.The embedding-size analysis therefore indicates a capacity trade-off.
Number of Neighbors: Increasing the neighbor count S produces gains on most datasets because more neighbors provide information for encoding sequences and temporal collaborative signals.The tested values are S in {5, 10, 20}.
Temporal Construction: Time embeddings significantly outperform position embeddings, indicating the necessity of a temporal kernel for capturing temporal effects in sequences.A trainable ω achieves the best performance in the reported comparison.
Temporal Information: Removing temporal vectors shows that item-without-time variants outperform user-without-time variants, emphasizing the importance of temporal user-embedding inference.The authors relate this difference to dynamic user preferences and relatively static items.
Temporal Correlation: Attention weights vary across timestamps for the same items, and arbitrary time increments demonstrate TGSRec’s continuous temporal inference.The visualization uses rows for time increments from the last interaction and columns for items.
Temporal Correlation: Only TGSRec places the ground-truth item in the top four at the times of interest, including when the prediction time approaches the test timestamp.SASRec and TiSASRec are described as using only sequential patterns in this comparison.

6 CONCLUSION

The paper concludes that TGSRec unifies sequential patterns and temporal collaborative signals through a continuous-time graph and temporal collaborative attention. Experiments, ablations, and temporal analyses support the effectiveness of this framework for sequential recommendation with temporal information.

Conclusion: TGSRec is defined on a continuous-time bipartite graph and uses a temporal kernel to map continuous edge timestamps into vectors.The framework is designed to represent temporal information in user-item interactions.
Conclusion: Its TCT layer samples neighbors and aggregates node embeddings with time vectors using attention to encode sequences, collaborative signals, and temporal effects.The learned information is propagated through multiple TCT layers.
Conclusion: Experiments on five real-world datasets show that TGSRec significantly outperforms existing transformer-based sequential recommendation models.Ablation studies and detailed analyses verify the efficacy of its components.