PaperlayerView exact revision on arXiv

Source-linked AI summary

Empirical Analysis of Session-Based Recommendation Algorithms

Malte Ludewig, Noemi Mauro, Sara Latifi, Dietmar Jannach

arXiv:1910.12781v2cs.IRcs.LGcs.NE

TL;DR

Session-based recommendation relies on ongoing-session interactions, but the progress of neural methods over simpler approaches remained unclear. The authors benchmarked twelve methods under identical conditions and found limited neural accuracy progress, while nearest-neighbor heuristics often performed better.

  • Problem

    Session-based recommendation research lacked mature, consistent evaluation practices, while simpler nearest-neighbor methods could be competitive with neural techniques.

    Source: §1 · p. 2

  • Method

    The authors compared twelve session-based methods, including six recent neural approaches, under identical conditions across several datasets.

    Source: §1 · p. 2

  • Results

    Neural accuracy progress was limited, and in most experiment configurations a simple technique outperformed all neural approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The shared session-rec framework includes algorithms, evaluation procedures, metrics, and dataset pointers for benchmarking and reproducibility.

    Source: §7 · pp. 21–22

    Offline evaluations of historical logs may reflect prior recommender choices or item promotions, limiting what strong log prediction establishes about user preferences.

    Source: §6 · p. 20

Abstract

from arXiv · show

Recommender systems are tools that support online users by pointing them to potential items of interest in situations of information overload. In recent years, the class of session-based recommendation algorithms received more attention in the research literature. These algorithms base their recommendations solely on the observed interactions with the user in an ongoing session and do not require the existence of long-term preference profiles. Most recently, a number of deep learning based ("neural") approaches to session-based recommendations were proposed. However, previous research indicates that today's complex neural recommendation methods are not always better than comparably simple algorithms in terms of prediction accuracy. With this work, our goal is to shed light on the state-of-the-art in the area of session-based recommendation and on the progress that is made with neural approaches. For this purpose, we compare twelve algorithmic approaches, among them six recent neural methods, under identical conditions on various datasets. We find that the progress in terms of prediction accuracy that is achieved with neural methods is still limited. In most cases, our experiments show that simple heuristic methods based on nearest-neighbors schemes are preferable over conceptually and computationally more complex methods. Observations from a user study furthermore indicate that recommendations based on heuristic methods were also well accepted by the study participants. To support future progress and reproducibility in this area, we publicly share the session-rec evaluation framework that was used in our research.

A Comparison of Neural and Non-Neural Approaches

The paper compares neural and non-neural session-based recommendation methods under identical conditions. It finds limited neural accuracy progress and generally favors simple nearest-neighbor heuristics, which were also well accepted in a user study.

A Comparison of Neural and Non-Neural Approaches

1 INTRODUCTION

Session-based recommendation uses interactions from an ongoing session, and interest in the area grew alongside neural methods. This paper benchmarks twelve approaches, reports limited neural progress, and supplements offline results with a user study and a shared evaluation framework.

1 INTRODUCTION

  • Session-based recommendation adapts from interactions in the ongoing session, a setting relevant when long-term user information is unavailable.The topic saw increased academic interest in recent years alongside the availability of public datasets and the rise of neural methods.Source: §1 · pp. 1–2
  • The authors compared twelve session-based recommendation techniques under identical conditions, including six recent neural approaches selected from major conferences.The selection excluded methods using item side information and required available source code that could be integrated with reasonable effort.Source: §1 · p. 2
  • In most experiment configurations, a simple technique outperformed all neural approaches, and newer neural methods did not consistently outperform gru4rec.The authors conclude that neural progress remains limited.Source: §1 · p. 2
  • The authors publicly shared session-rec, a framework with the compared algorithms, evaluation procedures, metrics, and pointers to the datasets.Source: §1 · p. 2
  • The user study compared heuristic methods, a neural approach, and Spotify in an online radio setting; heuristic playlists were well accepted by users.The study also examined users’ interest in discovering new items.Source: §1 · p. 2
  • The authors describe this work as extending earlier studies with newer methods, scalability and stability measurements, and comparison to user-study findings.Source: §1 · p. 2

2 ALGORITHMS

The paper surveys the development of session-based recommendation and selects a mix of simple and neural approaches for comparison. Its roster reflects practical application domains and restricts neural-method inclusion to support reproducible evaluation.

2.1 Historical Development and Algorithm Selection

  • Session-based recommendation spans related-item suggestions and automated playlists, with web, news, e-commerce, and music as key application domains.Source: §2.1 · p. 3
  • The algorithm selection reflects a history from early pattern-mining, Markov, and nearest-neighbor methods to gru4rec and a continuing wave of neural proposals.The first deep-learning session-based method, gru4rec, appeared in late 2015; earlier playlist work also used nearest neighbors.Source: §2.1 · pp. 3–4

2 ALGORITHMS

  • The algorithm roster includes co-occurrence, sequential-pattern, Markov-process, and session-based nearest-neighbor methods, alongside gru4rec and five later neural approaches.Source: §2.1 · p. 4

2.1 Historical Development and Algorithm Selection

  • Neural methods were selected only if they used no item side information, appeared at a major CORE A or A* conference, and had source code integrable with reasonable effort.Source: §2.1 · p. 4

2.2 Considered Algorithms

  • The study considered twelve algorithms, with Table 1 overviewing non-neural methods and Table 2 overviewing neural methods.The authors note that only one recent neural-method paper used session-based nearest neighbors as a baseline, despite prior documentation of their competitiveness.Source: §2.2 · pp. 4–5, Table 1 · p. 5, Table 2 · p. 6

3 EVALUATION METHODOLOGY

The evaluation compares recommendation methods under shared conditions using a reproducible framework and datasets from e-commerce and music. Its protocol combines temporal train-test splits, tuned parameters, accuracy measures, and additional checks of item coverage, popularity, stability, and running time.

3 EVALUATION METHODOLOGY

  • All methods were benchmarked under the same conditions using the shared evaluation framework to support reproducibility.The framework was also used in the reported experiments.Source: §3 · p. 6

3.1 Datasets

  • The evaluation used eight datasets from e-commerce and music, removing one-interaction sessions and items appearing fewer than five times while retaining repeated items.All datasets except ZALANDO and 8TRACKS were shared online.Source: §3.1 · p. 7

3.2 Evaluation Procedure and Metrics

  • Five non-overlapping contiguous splits preserve temporal order; playlist datasets received randomly generated timestamps, and test items had to occur in training data.The number of test days varied by dataset; revealed session interactions could inform subsequent predictions but were not added to training.Source: §3.2 · pp. 7–8
  • Hyperparameters were systematically optimized for each method and dataset using MRR@20, with 100 random-search iterations generally and fewer for specified methods or datasets.NARM and CSRM used 50 iterations; tuning was also limited to 50 for SR-GNN on ZALANDO and NextItNet on RSC15.Source: §3.2 · p. 8
  • Accuracy was measured both for the immediate next item and for the remaining session, with reported results using recommendation lists of length 20.Measures included Hit Rate, Mean Reciprocal Rank, Precision, Recall, and Mean Average Precision.Source: §3.2 · pp. 8–9
  • Coverage counts catalog items appearing in any top-n list, while novelty is approximated by average recommended-item popularity, with lower popularity treated as greater discovery potential.Coverage also indicates whether recommendations vary by session or tend to repeat across users.Source: §3.2 · p. 9
  • Running times were reported for selected datasets; neural methods used a GPU and non-neural methods a CPU, while nearest-neighbor methods learned no model but could build prediction-time data structures.The authors note that the hardware difference makes the measured neural running times conservative for comparing computational complexity.Source: §3.2 · p. 9
  • Stability was assessed by comparing daily evaluation with retraining after each day against using the initial model throughout, interpreting relative accuracy drop as an indicator when starting accuracy is comparable.The comparison included only items already present in the original training data.Source: §3.2 · pp. 9–10

4 RESULTS

Across the evaluated datasets, nearest-neighbor methods are often strongest on accuracy, while algorithm rankings and coverage vary by dataset and method. Neural methods generally require more computation, and stability without retraining also differs across algorithms and datasets.

4.1 Accuracy Results

  • Nearest-neighbor methods lead most e-commerce accuracy comparisons, while RSC15 is an exception where narm slightly outperforms vstan on MAP, Precision, and Recall.The best neural and non-neural differences are often around or below 1%.Source: §4.1 · p. 10, Table 6 · p. 11
  • Across e-commerce datasets, vstan is the best neighborhood-based method for most measures, making it a reasonable baseline for future comparisons.Source: §4.1 · p. 10
  • Across music datasets except 8TRACKS, nearest-neighbor methods favor Precision, Recall, MAP, and Hit Rate, while ct leads MRR; narm leads Precision and MRR on 8TRACKS.The ranking of algorithms varies across datasets, though narm performs consistently well on most datasets and measures.Source: §4.1 · p. 12, Table 7 · p. 13

4.2 Coverage and Popularity

  • ct has much higher popularity bias than other methods, whereas gru4rec almost consistently recommends the most unpopular or novel items; neighborhood methods often fall between them.The experiments do not confirm that neighborhood methods generally focus more on popular items than neural methods; sr-gnn can have similar or stronger popularity bias.Source: §4.2 · p. 12
  • gru4rec often has the highest coverage, while neighborhood-method coverage ranges from nearly as high to low and other neural methods show no clear ranking.Other neural methods’ coverage is often within the neighborhood-method range and sometimes very low.Source: §4.2 · pp. 12, 14

4.3 Scalability

  • Neural methods generally have much higher computational complexity than non-neural methods, and several algorithms, including ct, have scalability issues as recommendable items increase.nextitnet also ran out of memory on some datasets.Source: §4.3 · p. 14
  • On ZALANDO, the best neural method, sr-gnn, needs 10,000 times more training time than the best non-neural method, vstan, which needs half the recommendation time.Source: §4.3 · pp. 14–15, §4.4 · p. 15

4.4 Stability With Respect to New Data

  • Without daily retraining, accuracy decreases vary by dataset, and nearest-neighbor approaches are generally more stable than association-rule or context-tree baselines.On DIGI, decreases range from 0–10%; on NOWP, they range from about 15–50% for HR and 15–85% for MRR. Neural-method stability also varies: narm is most stable on DIGI, while csrm leads on NOWP; nearest-neighbor methods appear quite stable overall.Source: §4.4 · pp. 15–16

5 OBSERVATIONS FROM A USER STUDY

The user study compared how people perceived recommendations from different session-based methods in an online radio setting. It found that simple methods, especially sknn, were well perceived, while users valued Spotify’s discovery support and optimizing only for likes could be misleading.

5.1 Research Questions and Study Setup

  • The between-subjects study recruited 250 participants, assigning 50 to each of five treatments: ar, cagh, sknn, gru4rec, and Spotify.Participants used an online radio application, selecting a seed track and rating subsequent track excerpts; playlist updates followed like actions.Source: §5.1 · pp. 16–17

5.2 User Study Outcomes

  • 6.48 likes went to ar, ahead of sknn (5.63), cagh (5.38), gru4rec (5.36), and Spotify (4.48).The study also found a correlation of r=0.89 between track popularity in MPD and likes.Source: §5.2 · p. 18
  • Spotify recommended the fewest already-known tracks (7.00%), while cagh (10.83%) and sknn (10.13%) recommended the most.sknn and cagh continuations were perceived as most suitable; ar was considered the least playlist-matching method.Source: §5.2 · pp. 18–19
  • The study found that sknn recommendations were well perceived across multiple dimensions, complementing its strong offline performance.The post-task results also rated sknn significantly better than gru4rec, ar, and Spotify on overall liking.Source: §5.2 · pp. 19–20
  • Ar’s highest like count was misleading: it was consistently worse on nearly all other dimensions and often produced very poor individual-track ratings.Its playlist-independent track-rating distribution had a mode of 1 on the 1–7 scale.Source: §5.2 · pp. 19–20
  • Spotify was slightly less appreciated than sknn, yet users reported no difference in intention to reuse or recommend the systems.Spotify was perceived as significantly better at recommending unknown tracks that users liked.Source: §5.2 · p. 20

6 RESEARCH LIMITATIONS

The authors qualify their comparisons because historical interaction logs may reflect prior recommender behavior, and the analysis did not investigate those biases. They also leave explanations for complex models’ results unresolved and note constraints on the user study’s setting and measures.

Potential Data Biases

  • Historical logs may reflect existing recommenders or item promotion, so offline results may not fully represent system effectiveness.The authors did not analyze or account for these potential biases, focusing instead on comparisons under standard evaluation protocols.Source: §6 · p. 20

Empirical Nature of the Work

  • Why complex models did not consistently outperform simpler methods remains unresolved; possible explanations include unimportant sequence order or overfitting.A detailed analytical investigation of the six complex models was left for future work.Source: §6 · p. 21

User Study Limitations

  • The study offers user feedback, but its findings may still reflect a nonrepresentative population and an artificial, paid radio setting.The authors note that each post-task quality dimension was measured with a single questionnaire item.Source: §6 · p. 21

7 CONCLUSIONS AND WAYS FORWARD

The authors conclude that neural methods have not yet delivered consistent gains over simpler heuristics for session-based recommendation. They advocate broader quality evaluation, reproducible comparisons, and further study of hybrid methods across application domains.

7 CONCLUSIONS AND WAYS FORWARD

  • Deep learning techniques often do not outperform much simpler heuristic methods, indicating that progress in session-based recommendation remains limited.The authors see potential for more effective neural recommendation methods in the future.Source: §7 · p. 21
  • The shared session-rec framework includes preprocessing, evaluation, and hyperparameter-optimization code to support reproducible comparisons.The authors also highlight that inconsistent preprocessing and baseline choices can make results inconclusive.Source: §7 · pp. 21–22
  • Optimizing only for accuracy can be insufficient; coverage, diversity, novelty, and efficiency may matter for service adoption and success.The authors argue that more user studies and field tests are needed to understand successful recommendations in specific domains.Source: §7 · p. 22
  • Hybrid methods using user or item side information are promising, but whether their benefits generalize beyond individual domains remains an open question.The compared approaches often rely on distinct datasets or target specific settings such as cold-start news recommendation.Source: §7 · p. 22

Related papers

Loading 1910.12781v2…