PaperlayerView exact revision on arXiv

Source-linked AI summary

Evaluation of Session-based Recommendation Algorithms

Malte Ludewig, Dietmar Jannach

arXiv:1803.09587v2cs.IRcs.AIcs.LGcs.NE

TL;DR

Session-based recommendation addresses next-action prediction when long-term user profiles are unavailable. The paper conducts a broad comparison of simple and sophisticated algorithms across datasets and measures, finding that simpler methods often match or outperform deep-learning approaches. The results support stronger baselines and indicate room for more sophisticated session-based models.

  • Problem

    Long-term user profiles are often unavailable, making it important to recommend items from behavior observed during an ongoing session; comparisons also lack standard datasets and protocols.

  • Method

    The paper performs an in-depth comparison across multiple domains and datasets, covering simple methods, nearest neighbors, recurrent neural networks, and factorization-based approaches.

  • Results

    Simple methods often produce similarly accurate or better predictions than recent deep-learning and recurrent-neural-network approaches.

  • Takeaways & Limitations

    Researchers should include simpler methods as alternative baselines, while further improving sophisticated session-based recommendation algorithms.

  • Takeaways & Limitations

    The study’s experiments focus on past item-view events and next-view prediction, leaving the weighting and interpretation of other event types for future research.

Abstract

from arXiv · show

Recommender systems help users find relevant items of interest, for example on e-commerce or media streaming sites. Most academic research is concerned with approaches that personalize the recommendations according to long-term user profiles. In many real-world applications, however, such long-term profiles often do not exist and recommendations therefore have to be made solely based on the observed behavior of a user during an ongoing session. Given the high practical relevance of the problem, an increased interest in this problem can be observed in recent years, leading to a number of proposals for session-based recommendation algorithms that typically aim to predict the user's immediate next actions. In this work, we present the results of an in-depth performance comparison of a number of such algorithms, using a variety of datasets and evaluation measures. Our comparison includes the most recent approaches based on recurrent neural networks like GRU4REC, factorized Markov model approaches such as FISM or FOSSIL, as well as simpler methods based, e.g., on nearest neighbor schemes. Our experiments reveal that algorithms of this latter class, despite their sometimes almost trivial nature, often perform equally well or significantly better than today's more complex approaches based on deep neural networks. Our results therefore suggest that there is substantial room for improvement regarding the development of more sophisticated session-based recommendation algorithms.

1 INTRODUCTION

Session-based recommendation predicts a user’s next actions from ongoing-session behavior when long-term profiles are unavailable. This paper compares diverse algorithms and finds that simple methods often match or outperform deep-learning approaches.

  • Long-term user models are often unavailable for first-time or unlogged-in users, requiring recommendations based on recent session interactions.
  • Session-based recommendation research typically predicts the next user action from time-ordered interaction logs.
  • No standard datasets or evaluation protocols make comparisons difficult, especially when studies use different baselines.
  • The study compares simple and sophisticated algorithms across multiple domains and datasets to establish a common research base.
  • Simple methods often achieve accuracy similar to or better than recent deep-learning techniques, motivating stronger simple baselines and further model development.
  • The evaluation extends prior work with more datasets, richer measures, newer algorithms, sequence-aware similarity measures, and a new Session-based Matrix Factorization method.

2 REVIEW OF SESSION-BASED RECOMMENDATION APPROACHES

Session-based recommendation includes pattern-based, Markov, reinforcement-learning, recurrent-neural-network, embedding, and hybrid factorized-Markov approaches. The paper also treats simple sequential and co-occurrence methods as baselines and restricts its scope to session-based scenarios.

  • Early approaches mine frequent sequential patterns, but their computational demands and parameter-selection challenges limit their simplicity in practice.
  • Simple item co-occurrence patterns can perform no worse than frequent item sequences in some application domains, motivating both as baselines.
  • More sophisticated sequence-modeling approaches include Markov chains, reinforcement learning, Markov decision processes, and recurrent neural networks.
  • Markov-chain methods face data sparsity and must choose how many preceding interactions to consider when predicting the next action.
  • GRU-based recurrent networks were introduced for next-action prediction, while session-based nearest neighbors remained competitive in the same problem setting.
  • Hybrid methods combine matrix factorization with sequence modeling through factorized Markov chains.
  • The paper focuses exclusively on session-based recommendation, leaving combinations of long-term and short-term models for future research.

3 DETAILS OF THE INVESTIGATED METHODS

The evaluation compares four method families using past sequential sessions to predict the next event. It includes simple baselines, nearest neighbors, recurrent networks, and factorization-based approaches, with session-aware scoring and implicit-feedback inputs.

  • The comparison covers simple heuristics, nearest-neighbor techniques, recurrent neural networks, and factorization-based methods.
  • Models train on sequential past sessions and predict the next event; inactivity times define session boundaries when datasets do not provide them.
  • The study uses implicit feedback such as item views or listening events and excludes domain-specific content features.
  • The baseline set includes Simple Association Rules, first-order Markov Chains, Sequential Rules, and Bayesian Personalized Ranking matrix factorization.
  • Simple Association Rules count how often item pairs co-occur within sessions, using rules of maximum size two.
  • The Markov-chain baseline counts immediate transitions from the current session’s last item and normalizes counts into relative transition probabilities.
  • Sequential Rules allow intervening events and weight rules by distance, using w_sr(x) = 1/(x).
  • For session-based BPR-MF, each training session acts as a distinct user, and the current user vector is the average of its session’s latent item vectors.

3.2 Nearest Neighbors

The paper compares item- and session-based nearest-neighbor methods, including sequence-aware variants that emphasize event order or restrict recommendations to observed transitions. These methods use similarity, recency weighting, and efficient indexing or sampling to support session-based prediction.

  • Nearest-neighbor variants: Nearest-neighbor methods are included because they often perform surprisingly well despite their simplicity.The comparison covers a traditional item-based variant and three session-based variations.
  • Item-based kNN: The iknn method recommends items similar to the last session item using co-occurrence across past sessions.Items are represented as binary vectors over sessions, and cosine similarity can measure item relatedness.
  • Session-based kNN: The sknn method compares the entire current session with past sessions and recommends items from the most similar neighbors.Session similarity can use measures such as the Jaccard index or cosine similarity on binary representations.
  • Scalability: Nearest-neighbor recommendation uses pre-computed in-memory indexes and neighborhood sampling to provide fast online responses.The sample size parameter m controls how many recent sessions are considered, and experiments found 1,000 sessions sufficient in one setting.
  • Sequence-aware extensions: Sequence-aware variants emphasize event order by weighting more recent session elements or by filtering recommendations to observed next-item transitions.The weighting approach can assign higher scores when a neighbor contains a more recent matching item, while sf-sknn retains items that followed the current last item in training data.
  • Additional experiments: Alternative Word2Vec and Doc2Vec encodings of sequential information did not improve accuracy over sknn and were omitted from later discussion.This result is reported as an additional experiment rather than as part of the main comparison.

3.3 Neural Networks – gru4rec

gru4rec models session behavior with a recurrent neural network using Gated Recurrent Units to predict subsequent events. Its design combines ordered session processing, session-parallel mini-batch training, and specialized ranking losses.

  • Model architecture: gru4rec uses an RNN with Gated Recurrent Units to predict the probability of subsequent events from a session beginning.The network architecture may include embedding, feedforward, and additional GRU layers, although a single GRU layer performed best in the cited experiments.
  • Session representation: The network receives one-hot item vectors and outputs a ranking distribution for the subsequent item while its hidden state records earlier session items.Items must be processed in order, and the GRU hidden state is reset when a session ends.
  • Design choices: The approach’s key design elements are its network architecture, loss functions, and session-parallel mini-batches for faster training.These elements are identified as the method’s innovative components in the paper’s discussion.
  • Training: Session-parallel mini-batches preserve within-session item order while allowing multiple sessions to be processed during training.Each batch segment belongs to a specific session, and the network maintains separate session states.
  • Loss functions: The MAX loss improves on earlier losses by using the highest-rated negative sample, with softmax providing a differentiable approximation.The resulting BPRmax and TOP1max functions showed superior performance to BPR and TOP1 in the cited work.
  • Implementation: The experiments use the maintained gru4rec version 2.0 implementation shared online, including the method, baselines, and evaluation procedure.The implementation is described as regularly maintained by the authors.

3.4 Factorization-based Methods

The comparison covers factorization-based sequential methods, including adapted existing models and the novel session-based matrix factorization approach. These methods combine latent-factor representations with item-transition information or session embeddings to address anonymous session recommendation.

  • The evaluation includes fpmc, fism, fossil, and the novel session-based matrix factorization method smf.The existing methods were adapted to the session-based setting, while smf was designed specifically for it.
  • Factorized Personalized Markov Chains (fpmc): Fpmc factorizes a user-item-transition tensor whose third dimension captures transition probabilities between items.The method combines Markov-chain modeling with traditional user-item matrix factorization.
  • In anonymous sessions, each training session represents a user, and session latent vectors are estimated by averaging the latent factors of its items.This adaptation treats each new session as a user cold-start situation after training.
  • Factored Item Similarity Models (fism): Fism uses item-item latent-factor products over previously inspected session items, but does not model sequential item-to-item transitions.Bias terms and a normalization parameter are included, and BPR is used to optimize top-n recommendations from implicit feedback.
  • FOSSIL: Fossil extends fism with factorized Markov-chain terms, combining long-term preference factors with weighted sequential dynamics.In the session setting, sessions represent users, the current session supplies the positive items, and BPR ranks items over negative examples.
  • Session-based Matrix Factorization (smf): Smf replaces a latent user vector with an embedded session-preference vector and combines session preferences with sequential dynamics.Its training incorporates BPRmax and TOP1max losses, session dropout, and skip-rate controls to reduce overfitting.

4 EXPERIMENT SETUP

The experiments compare session-based recommenders across e-commerce, music, and news datasets using iterative next-item evaluation and additional quality measures. The protocol combines multiple data-splitting strategies with measurements of accuracy, coverage, popularity bias, scalability, and robustness to reduced training data.

  • Evaluation Task: The general task is to generate a ranked list matching a session beginning, with the target depending on the application scenario.Examples include alternative shopping items and continuations of music-listening sessions.
  • Accuracy Measures: The experiments iteratively reveal session entries and measure immediate-next-item Hit Rate and Mean Reciprocal Rank for different recommendation-list lengths.This scheme is used to compare predictions after progressively longer session prefixes.
  • Sessionization: Sessions come from dataset-provided identifiers, idle-time heuristics, day-level grouping, or playlist boundaries, depending on the dataset.For some datasets, a session ends after defined inactivity; for TMALL, all events from one day form a session.
  • Training and Test Splits: The main analysis uses a sliding-window protocol with five equal-size time slices, while a single split is retained for comparability with prior work.The sliding-window design enables multiple measurements with different test sets.
  • Evaluation Conditions: For playlist datasets without timestamps, the study uses random cross-validation, whereas e-commerce and news data retain time-aware splits because of temporal effects.The evaluation also tests reduced-training-data conditions by artificially removing older training data.
  • Additional Quality Factors: Coverage, popularity bias, training and prediction time, and memory requirements supplement accuracy measurements.These measures assess catalog diversity, popularity tendencies, computational demands, and resource use.
  • Datasets: The study evaluates algorithms on datasets from three domains: e-commerce, music, and news.The e-commerce collection includes four datasets, while the media datasets cover music and news.
  • Dataset Reporting: The reported dataset characteristics are averaged over five non-overlapping splits except for RSC15-S, which uses one train-test split.This exception preserves comparability with previous evaluations.

5 RESULTS

Across e-commerce, music, and news datasets, neighborhood-based methods generally achieved the strongest accuracy, while results depended on the metric, list length, and domain. Simpler methods also had practical advantages, whereas some model-based methods showed coverage, popularity, or stability trade-offs.

  • Accuracy: The lowest accuracy values were almost consistently produced by factorized Markov approaches and the session-aware bpr-mf variant.bpr-mf sometimes performed competitively at list length 1, but with high popularity bias and low coverage.
  • Accuracy: GRU4REC was consistently among the top five for hit rate and competitive for MRR, but was often significantly worse than the best methods on other datasets.It outperformed all other methods on RSC15(-S) in hit rate, where MRR differences among top methods were tiny.
  • Accuracy: Neighborhood-based methods usually achieved the best hit rate and MRR across datasets, with v-sknn the most consistently strong variant.Exceptions included RSC15(-S) and MRR on ZALANDO.
  • Alternative accuracy measures: Evaluated with precision and recall over the remaining session, neighborhood-based methods again performed best, while GRU4REC was less effective than under immediate-next-action evaluation.The difference is attributed to GRU4REC being optimized for predicting the immediate next action.
  • List length: Shorter recommendation lists changed algorithm rankings: differences narrowed between nearest-neighbor methods and GRU4REC or sr, while bpr-mf and sometimes fpmc performed best at length 1.The effect may be overemphasized because the evaluation protocol focuses only on the next item; bpr-mf also had high popularity tendency and low coverage.

6 CONCLUSION AND FUTURE DIRECTIONS

The paper finds that lightweight session-based methods often match or outperform recent deep-learning approaches, while identifying several directions for more flexible and context-aware models.

  • Main findings: Across multiple datasets, simpler session-neighborhood methods often outperform recurrent neural-network approaches in prediction accuracy.The paper reports that their computational demands can remain low through in-memory cache structures and data sampling.
  • Model development: The results indicate a need for sophisticated models that flexibly control how much sequential information from a session enters training.The authors also identify further possible improvements to nearest-neighbor methods, including alternative similarity measures.
  • Additional comparison: An initial evaluation found that STAMP did not outperform the trivial sr method in MRR on Diginetica, although it appeared advantageous in hit rate.This comparison concerns the particular Diginetica dataset used in the STAMP evaluation.
  • Interpretation: The study could not identify one best baseline across all datasets and settings because relative algorithm performance depends on factors not yet fully understood.At least one strong simpler method was found for each dataset, but its identity varied.
  • Practical implications: Simple baselines can serve as indicators of the characteristics that more complex models should aim to implement.The paper gives different best-performing baselines as examples for guiding choices between sequential and neighborhood-based models.
  • Future directions: Future work should examine previous sessions, contextual factors, short-term community trends, and dataset characteristics when designing session-aware or hybrid recommenders.The authors connect these factors with preference drift, popularity, recency, sequential patterns, and algorithm suitability.

AUTHOR BIOGRAPHIES

The biographies describe the authors’ academic positions and research backgrounds in recommender systems and artificial intelligence.

  • Malte Ludewig is a TU Dortmund computer-science PhD candidate whose research focuses on session-based recommendation and e-commerce personalization.
  • Dietmar Jannach is an AAU Klagenfurt computer-science professor leading an information-systems research group.His work spans recommender systems, model-based diagnosis, knowledge-based systems, and practical artificial-intelligence applications.
  • Jannach has authored more than one hundred research papers and leads a textbook on recommender systems.

A PARAMETER CONFIGURATIONS

Appendix A lists the parameter configurations used for GRU4REC, SMF, V-SKNN, SKNN, S-SKNN, and SF-SKNN across all datasets.

  • Table 9 provides the parameters used for the gru4rec algorithm across all datasets.
  • Table 10 provides the parameters used for the smf algorithm across all datasets.
  • Table 11 provides the parameters used for the v-sknn algorithm across all datasets.
  • Table 12 provides the parameters used for sknn, s-sknn, and sf-sknn across all datasets.

B FULL RESULT TABLES

Appendix B presents full evaluation tables for ten datasets, reporting hit rate, mean reciprocal rank, item coverage, and average popularity at several list lengths.

  • The tables report hit rate, mean reciprocal rank, item coverage, and average popularity for list lengths of 20, 10, 5, 3, and 1.
  • Full results are provided for the RSC15 and RSC15-S datasets.
  • Each dataset table is sorted by MRR@20.
  • Full results are provided for the TMALL, RETAILROCKET, and ZALANDO datasets.
  • Full results are provided for the 8TRACKS, AOTM, 30MUSIC, NOWPLAYING, and CLEF datasets.

C ADDITIONAL RESULTS FOR PRECISION AND RECALL

This section reports precision and recall results across eight datasets, using list lengths of 20, 10, 5, and 3. The tables are sorted by P@20.

  • Precision and recall are reported for list lengths 20, 10, 5, and 3 on TMALL and RETAILROCKET.
  • Precision and recall are reported for list lengths 20, 10, 5, and 3 on ZALANDO and 8TRACKS.
  • Precision and recall are reported for list lengths 20, 10, 5, and 3 on AOTM and 30MUSIC.
  • Precision and recall are reported for list lengths 20, 10, 5, and 3 on NOWPLAYING and CLEF.

D ADDITIONAL SINGLE SPLIT RESULTS

This section reports single-split results across eight datasets. The tables include hit rate, mean reciprocal rank, item coverage, and average popularity for list lengths from 20 to 1, sorted by MRR@20.

  • Single-split results for TMALL and RETAILROCKET report HR, MRR, COV, and POP at list lengths 20, 10, 5, 3, and 1.
  • Single-split results for ZALANDO and 8TRACKS report HR, MRR, COV, and POP at list lengths 20, 10, 5, 3, and 1.
  • Single-split results for AOTM and 30MUSIC report HR, MRR, COV, and POP at list lengths 20, 10, 5, 3, and 1.
  • Single-split results for NOWPLAYING and CLEF report HR, MRR, COV, and POP at list lengths 20, 10, 5, 3, and 1.
Loading 1803.09587v2…