Source-linked AI summary
Contextual Sequence Modeling for Recommendation with Recurrent Neural Networks
Elena Smirnova, Flavian Vasile
TL;DR
Existing RNN recommenders model item sequences but omit contextual signals such as interaction type and timing, despite their relevance to next-event prediction. The paper proposes CRNNs that incorporate context into item representations and hidden-state dynamics, and reports significantly better results than sequential and non-sequential baselines. The approach performs especially well on difficult cases such as sale events and non-historical items.
Problem
Existing sequence models summarize user state from interacted items without incorporating interaction types, time gaps, or time of day.
Method
CRNNs incorporate context through context-dependent input/output modeling and context-parameterized hidden-state transitions.
Results
CRNNs significantly outperform sequential and non-sequential baselines on two e-commerce datasets, with all models improving over the best baseline in Recall@10.
Takeaways & Limitations
Improvements are especially large for difficult next-item cases involving sale events and non-historical items.
Takeaways & Limitations
The ItemId x EventType baseline could not be trained because Cartesian-product sparsity caused numerical instability in the loss function.
Abstract
from arXiv · showhide
Recommendations can greatly benefit from good representations of the user state at recommendation time. Recent approaches that leverage Recurrent Neural Networks (RNNs) for session-based recommendations have shown that Deep Learning models can provide useful user representations for recommendation. However, current RNN modeling approaches summarize the user state by only taking into account the sequence of items that the user has interacted with in the past, without taking into account other essential types of context information such as the associated types of user-item interactions, the time gaps between events and the time of day for each interaction. To address this, we propose a new class of Contextual Recurrent Neural Networks for Recommendation (CRNNs) that can take into account the contextual information both in the input and output layers and modifying the behavior of the RNN by combining the context embedding with the item embedding and more explicitly, in the model dynamics, by parametrizing the hidden unit transitions as a function of context information. We compare our CRNNs approach with RNNs and non-sequential baselines and show good improvements on the next event prediction task.
1 INTRODUCTION
Next-item recommendation methods model user activity sequences, but interaction context such as event types and time gaps can substantially change likely continuations. The paper introduces CRNNs to incorporate this context and reports improvements over sequential and non-sequential baselines.
- Next-item prediction methods estimate the most likely continuation of a user's activity, whereas missing-link methods predict unseen items.
- RNNs model user-action sequences with nonlinear hidden dynamics that can discover patterns predictive of the next item.
- Interaction types, time gaps, and time of day provide contextual information beyond item order.
- Event types and time-gap patterns can substantially alter the likely next product, such as favoring a purchase after an add-to-basket event or a complementary item after a sale.
- CRNNs incorporate context through context-dependent input/output representations or context-parameterized hidden-state dynamics.
- CRNNs significantly outperform sequential and non-sequential baselines on the YooChoose and internal e-commerce datasets.
2 RELATED WORK
Prior recommendation research combines sequence modeling, contextual representations, and context-dependent recurrent dynamics in different ways. CRNNs bring these directions together by conditioning item representations or hidden transitions on recommendation context.
- Session-based RNNs have been applied to next-item recommendation, while multimodal extensions add product text and image information.
- Context-aware recommendation research shows that context can change user behavior, item similarities, and recommendation accuracy.
- Factorization methods model user-item-context interactions in latent space through multiplication and addition.
- Conditional RNNs assign probabilities to sequences given a representation of conditioning context.
- Complementary information has been incorporated by concatenating feature vectors with input and output representations or by using multiplicative interactions.
- Prior recurrent architectures adapt hidden transitions using input-dependent matrices, gating structures, auxiliary RNNs, or recurrent depth.
- CRNNs condition either item representations on context or the hidden dynamics of the RNN.
3 PROPOSED APPROACH
The proposed CRNNs incorporate context into item representations, outputs, and recurrent hidden-state dynamics to model context-dependent user-item sequences.
- CRNNs model both context-dependent item representations and context-dependent RNN dynamics for recommendation sequences.The approach targets improved modeling when conditioning on interaction type and time information.
- Model and optimization: The model represents each event as a paired item and context input and predicts the next item from current context and historical items and contexts.The sequence objective is to model p(x_t | c_t, x_<t, c_<t).
- Recurrent architecture: At each step, input, recurrent, and output modules create a context-integrated embedding, update the hidden state, and return an item probability distribution.The recurrent cell may be a GRU or LSTM, while the output applies softmax to produce item probabilities.
- Conditioning on context: Context can enter input and output modules through concatenation, multiplicative interaction, or both.Multiplicative interaction binds context more tightly to the item representation than concatenation.
- Conditioning on context: Context Wrapper conditions hidden transitions by using context to parameterize the recurrent update without a separate transition matrix for every context.The implementation applies this structure to a GRU cell.
4 EXPERIMENTS
The experiments evaluate recommendation methods on next-event prediction and report results after describing tasks, metrics, baselines, contextual configurations, and datasets.
- The experimental section covers evaluation tasks, success metrics, baselines, contextual-model configurations, datasets, and experimental results.
4.1 Setup
The setup evaluates top-K recommendation quality using Recall@K averaged over test events.
- Recall@K is the proportion of events whose test item appears in the top-K predicted items, averaged over all events.The paper notes that this metric was reported to correlate with click-through rate.
4.2 Baselines
The baselines range from last-item and orderless history methods to context-free and event-type-aware recurrent models, all using GRU cells.
- CoVisit uses only the last visited product, while BagOfItems sums historical item embeddings without modeling order.
- The context-free RNN models historical item order but excludes associated context.
- RNN on ItemId × EventType represents each item-event-type combination with a separate item vector.All experiments use the GRU cell as the recurrent function.
4.3 Contextual RNNs
The experiments compare four GRU-based contextual RNN configurations that introduce context through input/output representations, hidden dynamics, or both.
- CRNN configurations: Four configurations are evaluated: Mult-GRU-RNN, Concat-GRU-RNN, Concat-Mult-GRU-RNN, and Concat-Mult-Context-RNN.They vary multiplication, concatenation, and contextual hidden-state dynamics in the input/output modules and GRU cell.
- Model architecture: All models use GRU cells, with the Context-RNN variant adding a Context Wrapper to the GRU cell.
- Training setup: Item embeddings and RNN hidden states are fixed at size 100 across models.
- Training setup: Optimization uses Adam with learning-rate decay from 0.01 to 0.001, batch size 256, and 10,000 training iterations.
4.4 Datasets
The evaluation uses two real-world e-commerce datasets and contextual features describing time, event timing, and interaction type.
- Datasets: Experiments use the public YooChoose dataset and a proprietary Internal dataset.YooChoose contains sessions of clicks and purchases; the Internal dataset contains browsing and purchasing activity across multiple e-commerce verticals.
- YooChoose: YooChoose contains user sessions from multiple e-commerce websites collected over six months.
- Internal dataset: The Internal dataset covers three months of browsing and purchasing activity across multiple e-commerce websites and verticals.
- Dataset characteristics: The Internal dataset has fewer sessions and a heavier long tail of distinct items than YooChoose.
- Data splits: Sequences are filtered to remove length-one cases, retain the 20 latest events, and use dataset-specific validation and test splits.YooChoose uses consecutive temporal periods, while the Internal dataset holds out randomly selected users.
- Context features: Context features include month, hour, day of week, log2-discretized time differences, and event type, represented as concatenated one-hot vectors.
4.5 Results
CRNNs outperform sequential and non-sequential baselines on Recall@10, with especially strong gains for rare sale events and non-historical items.
- Overall performance: All CRNN models significantly outperform the best baseline on Recall@10 across the evaluated datasets.The results are statistically significant at the 95% confidence level.
- Overall performance: The Concat-Mult-Context-RNN achieves the largest overall improvement by combining contextual input/output representations with contextual hidden dynamics.
- Baselines: The GRU RNN without context is the strongest baseline, outperforming CoVisit and BagOfItems.CoVisit uses only the last product, whereas BagOfItems ignores item order.
- Event-type analysis: CRNNs improve Recall@10 on sale events more than on view events in both datasets.The largest gains are +12% on YooChoose and +6% on the Internal dataset, from Concat-Mult-Context-RNN; sale events comprise less than 5% of events.
- Historical-item analysis: CRNNs recommend up to +10% more relevant non-historical items than models without contextual information.The largest improvement again comes from Concat-Mult-Context-RNN.
- Sequence and timing analysis: Performance improvement increases with sequence length and is largest for time gaps below 2 seconds.
5 CONCLUSIONS
The paper introduces CRNN architectures for contextual next-item recommendation and reports better results than sequential and non-sequential baselines, especially on difficult cases.
- Contribution: The paper introduces Contextual Sequential Modeling for Next Item Recommendation and the CRNN architecture family.
- Findings: CRNNs achieve significantly better results than sequential and non-sequential state-of-the-art baselines.
- Findings: Improvements are especially large for sale events and non-historical items.
- Future work: Future work includes predicting the next K items and predicting future context together with the future item identifier.
Uplift in % of Recall@10. Dataset: YooChoose
Figure 4 shows Recall@10 uplift of the best CRNN over the best baseline across event type, item history, time since the last event, and sequence length.
- Recall@10 uplift is measured for the best CRNN relative to the best baseline across four projection dimensions.These dimensions are next-event type, new versus historical item, time since the last event, and sequence length.
- Next-event types are encoded as view, sale, and basket.The legend maps 1 to view, 2 to sale, and 3 to basket.
- Error bars represent 95% confidence intervals obtained from 30 bootstraps.
- Gray bars indicate the relative volume of test examples for each projection value.