Source-linked AI summary
Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks
Srijan Kumar, Xikun Zhang, Jure Leskovec
TL;DR
Existing dynamic embedding methods often update representations only when users act, rather than modeling future embedding trajectories. JODIE uses coupled recurrent networks and a projection operator to forecast those trajectories, while t-Batch makes training scalable; across four datasets and two tasks, it outperforms six state-of-the-art algorithms.
Problem
Existing dynamic embedding methods generate embeddings when users act but do not explicitly model how user or item embeddings evolve into the future.
Method
JODIE couples recurrent networks that update user and item embeddings with a projection operator that predicts future user embeddings from temporal interactions.
Results
JODIE outperforms six state-of-the-art algorithms by at least 20% on future interaction prediction and 12% on average on user state-change prediction.
Takeaways & Limitations
t-Batch creates temporally consistent parallel training batches and makes JODIE 9.2× faster than similar dynamic embedding baselines.
Takeaways & Limitations
Learning individual user and item trajectories is expensive, motivating future work on group-level trajectories to reduce parameters.
Abstract
from arXiv · showhide
Modeling sequential interactions between users and items/products is crucial in domains such as e-commerce, social networking, and education. Representation learning presents an attractive opportunity to model the dynamic evolution of users and items, where each user/item can be embedded in a Euclidean space and its evolution can be modeled by an embedding trajectory in this space. However, existing dynamic embedding methods generate embeddings only when users take actions and do not explicitly model the future trajectory of the user/item in the embedding space. Here we propose JODIE, a coupled recurrent neural network model that learns the embedding trajectories of users and items. JODIE employs two recurrent neural networks to update the embedding of a user and an item at every interaction. Crucially, JODIE also models the future embedding trajectory of a user/item. To this end, it introduces a novel projection operator that learns to estimate the embedding of the user at any time in the future. These estimated embeddings are then used to predict future user-item interactions. To make the method scalable, we develop a t-Batch algorithm that creates time-consistent batches and leads to 9x faster training. We conduct six experiments to validate JODIE on two prediction tasks---future interaction prediction and state change prediction---using four real-world datasets. We show that JODIE outperforms six state-of-the-art algorithms in these tasks by at least 20% in predicting future interactions and 12% in state change prediction.
1 INTRODUCTION
JODIE addresses dynamic interaction modeling by learning user and item embedding trajectories, including future user embeddings between interactions. Its t-Batch training procedure preserves temporal consistency while enabling parallelism, and experiments show strong gains over existing methods.
- Existing dynamic embedding methods update users mainly when they act, leaving representations unchanged between actions despite changing user intent.
- JODIE uses static and dynamic embeddings to represent long-term stationary and time-varying properties of users and items.
- Two coupled RNNs update user and item embeddings after each interaction while incorporating the interacting entity’s current embedding and interaction features.
- A projection operation estimates a user’s future embedding after elapsed time ∆, which is then used to predict the likely next item.
- 9.2× faster training is achieved by t-Batch, which groups independent interactions into temporally consistent batches processed in parallel.
- At least 20% improvement is reported for future interaction prediction and 12% average improvement for user state-change prediction over six algorithms.
2 RELATED WORK
Related methods include recurrent recommenders, dynamic co-evolution models, and temporal network embedding approaches. JODIE differs by mutually updating user and item embeddings, forecasting user trajectories, and outperforming key baselines.
- Table 1 compares desired properties across existing algorithms and JODIE, which satisfies all listed desirable properties.
- JODIE improves over recurrent recommender methods by at least 20% on next-interaction prediction and 12% on average on user state-change prediction.Its running time is reported as comparable to these methods.
- Dynamic co-evolution models also let user and item embeddings influence each other, but JODIE additionally projects user embeddings to arbitrary future times.
- CTDNE generates node embeddings with temporally increasing random walks, whereas JODIE learns and predicts continuous user and item embedding trajectories.
- 4.4× better next-interaction prediction than CTDNE is reported with comparable running time.
3 JODIE: JOINT DYNAMIC USER-ITEM EMBEDDING MODEL
JODIE jointly models static and dynamic user/item properties with mutually recursive RNN updates and a projection operator for future user embeddings. It also uses t-Batch to scale training and directly predicts future item embeddings.
- JODIE assigns each user and item static embeddings for stationary properties and dynamic embeddings whose sequences form time-varying trajectories.
- Embedding update operation: Two mutually recursive RNNs update user and item embeddings using the other entity’s current dynamic embedding, interaction features, and elapsed time.
- Embedding projection operation: The projection operator uses a user’s current embedding and elapsed time to estimate its future embedding for downstream interaction prediction.
- Embedding projection operation: As elapsed time increases, projected user embeddings drift farther from the previously observed embedding until the next interaction triggers an update.
- Training to predict next item embedding: JODIE directly outputs the next item’s embedding, reducing inference computation from linear in the number of items to near-constant time.
- Training with t-Batch: t-Batch selects temporally consistent batches of interactions that share no users or items, enabling parallel processing while preserving interaction order.
- Differences between JODIE and DeepCoevolve: 9.2× faster than the closest baseline, JODIE outperforms DeepCoevolve by 45% in future-interaction prediction and 13.9% in user-state-change prediction on average.
4 EXPERIMENTS
JODIE is evaluated on future interaction and user state change prediction across multiple datasets against six state-of-the-art baselines. It consistently achieves stronger predictive performance, remains robust across training-data and embedding-size settings, and trains efficiently with t-Batch.
- Experimental setting: JODIE is compared with six state-of-the-art algorithms on future interaction and user state change prediction tasks.The experiments use datasets from Reddit, Wikipedia, LastFM, and MOOC activity, with time-based training, validation, and testing.
- Case study: JODIE can predict student dropout five interactions in advance in a MOOC case study.This result is reported as part of the user state change experiments.
- Future interaction prediction: JODIE outperforms all baselines by at least 20% in MRR and 14% in recall@10 for future interaction prediction.The comparison covers three datasets and both ranking metrics.
- User state change prediction: JODIE outperforms baselines by at least 12.63% on average in AUC for user state change prediction.It exceeds RRN by at least 2.2% for ban prediction and by 28% for student dropout prediction.
- Efficiency: JODIE is 9.2× faster than DeepCoevolve and has comparable running time to other baselines.With t-Batch, one Reddit epoch takes 5.13 minutes instead of 43.53 minutes, an 8.4× speed-up.
- Robustness: JODIE remains stable as training-data availability changes and consistently outperforms baselines across the evaluated training-data percentages.This stability is reported for both future interaction and user state change prediction.
- Robustness: Embedding dimension has little effect on JODIE’s performance across the tested dynamic-embedding sizes.The reported evaluation varies the dynamic embedding dimension from 32 to 256 on LastFM, with similar effects on other datasets.
5 CONCLUSIONS
JODIE learns dynamic user and item embeddings from temporal interactions, predicts their future embeddings, and improves prediction performance while accelerating training. The authors identify expensive individual trajectories as a direction for future work.
- JODIE is a coupled recurrent neural network model that learns dynamic user and item embeddings from sequences of temporal interactions.
- JODIE predicts future user and item embeddings, improving future interaction and user-state-change prediction performance.
- JODIE’s training data batching method is an order of magnitude faster than similar baselines.
- Learning trajectories for individual users and items is expensive, motivating future work on group-level trajectories to reduce the number of parameters.
A APPENDIX
The appendix introduces the model’s technical details, identifies the project website for code and datasets, and points to tables containing dataset and parameter information.
- The appendix provides technical details of the model.
- The project website provides the code and datasets.
- Table 6 contains dataset details, while Table 5 contains model parameters.