PaperlayerView exact revision on arXiv

Source-linked AI summary

Linear Recurrent Units for Sequential Recommendation

Zhenrui Yue, Yueqi Wang, Zhankui He, Huimin Zeng, Julian McAuley, Dong Wang

arXiv:2310.02367v2cs.IR

TL;DR

Sequential recommenders based on self-attention incur time and space costs for incremental updates, while recurrent models impede parallel training. LRURec combines linear recurrence with recursive parallelization and architectural modifications; the paper reports superior results over state-of-the-art baselines on multiple real-world datasets.

  • Problem

    Self-attentive recommenders recompute sequence-level item-to-item weights after each new action, while recurrent nonlinear units hinder parallelizable training.

    Source: §1 · pp. 1–2

  • Method

    LRURec uses linear recurrence with matrix diagonalization and recursive parallelization, alongside architectural improvements including layer normalization and position-wise feed-forward networks.

    Source: §3.2 · p. 3, §3.4 · p. 5, §6 · p. 9

  • Results

    LRURec consistently achieves superior results over state-of-the-art baselines in extensive experiments on multiple real-world datasets.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results challenge the necessity of self-attention in sequential recommenders while highlighting the importance of techniques such as layer normalization.

    Source: §1 · p. 2

    The reported numbers should not be directly compared with results from works using k-core preprocessing because the paper’s preprocessing differs and yields more realistic, sparser distributions.

    Source: §5.1 · p. 6

Abstract

from arXiv · show

State-of-the-art sequential recommendation relies heavily on self-attention-based recommender models. Yet such models are computationally expensive and often too slow for real-time recommendation. Furthermore, the self-attention operation is performed at a sequence-level, thereby making low-cost incremental inference challenging. Inspired by recent advances in efficient language modeling, we propose linear recurrent units for sequential recommendation (LRURec). Similar to recurrent neural networks, LRURec offers rapid inference and can achieve incremental inference on sequential inputs. By decomposing the linear recurrence operation and designing recursive parallelization in our framework, LRURec provides the additional benefits of reduced model size and parallelizable training. Moreover, we optimize the architecture of LRURec by implementing a series of modifications to address the lack of non-linearity and improve training dynamics. To validate the effectiveness of our proposed LRURec, we conduct extensive experiments on multiple real-world datasets and compare its performance against state-of-the-art sequential recommenders. Experimental results demonstrate the effectiveness of LRURec, which consistently outperforms baselines by a significant margin. Results also highlight the efficiency of LRURec with our parallelized training paradigm and fast inference on long sequences, showing its potential to further enhance user experience in sequential recommendation.

1 INTRODUCTION

Self-attentive recommenders improve recommendation accuracy and training efficiency, but sequence-level updates make incremental inference costly. LRURec uses linear recurrence and recursive parallelization to target efficient training and RNN-like incremental inference.

1 INTRODUCTION

  • SARs recompute all item-to-item weights after each new action, incurring time and space costs for incremental user-representation updates.RNNs instead store a fixed-length hidden-state vector and update it stepwise, but their recurrent nonlinear units impede parallel training.Source: §1 · pp. 1–2
  • LRURec is presented as outperforming SARs in recommendation accuracy and training efficiency while matching RNNs’ inference efficiency through fixed-length user-vector updates.The introduction attributes parallelizable training to recursive parallelization.Source: §1 · p. 2
  • LRURec combines linear recurrent units and recursive parallelization with layer normalization, residual connections, and position-wise feed-forward networks.The paper presents these techniques as addressing linear recurrence’s lack of nonlinearity and improving recommendation accuracy and training dynamics.Source: §1 · p. 2
  • The paper reports that LRURec consistently outperforms baseline methods by a large margin on multiple real-world datasets.It also frames the findings as challenging the necessity of self-attention while highlighting techniques such as layer normalization.Source: §1 · p. 2

2 RELATED WORK

Prior sequential recommenders trade off parallel training, inference efficiency, and recommendation performance. LRURec brings linear recurrence to sequential recommendation, while adapting efficient language-modeling ideas to recommendation’s sequence lengths and task demands.

2.1 Sequential Recommendation

  • Transformer recommenders [16] [31] offer parallel training and improved accuracy, but their sequence-level inference is slower than RNN-based models [14, 19].RNNs update the latest hidden state, whereas transformer- and MLP-based models recompute sequence-level relevance when a new action arrives.Source: §2.1 · p. 2
  • LRURec applies linear recurrent units [24] to sequential recommendation, aiming to combine transformer-like training efficiency with RNN-like inference efficiency.Source: §2.1 · p. 2

2.2 Efficient Language Modeling

  • Efficient language-modeling approaches include AFT or RWKV [26] [39], S4 [5], linear recurrent units [24], and RetNet [32], which pursue long-sequence modeling with efficient training or inference.The passage describes RetNet as having recurrent and parallel formulations, enabling parallel training with low inference costs.Source: §2.2 · pp. 2–3
  • The paper argues that language-modeling solutions may overfit or incur latency in recommendation, and that long-range optimization may not suit recommendation’s short action sequences.LRURec is proposed with a designed LRU block and recursive parallelization to perform across sequence lengths with training and inference efficiency.Source: §2.2 · p. 3

3 METHODOLOGY

LRURec models sequential recommendation with a linear recurrent unit and recursive parallelization, then adds architectural components to address missing nonlinearity and improve training dynamics. Its setup predicts the next item from a chronological action history.

3.1 Setup

  • The model predicts the next item from a user’s chronological action history, training by minimizing the negative log-likelihood of the ground-truth item.The model scores items in the item space and aims to assign the highest score to the next action.Source: §3.1 · p. 3

3.2 Linear Recurrent Unit

  • LRU removes recurrence nonlinearity to serialize hidden-state updates, diagonalizes A to reduce matrix computation and improve stability, and uses polar complex parameters optimized in real space.The polar form expresses Λ using magnitude and phase; the parameterization enables the scale to be computed as e^-ν.Source: §3.2 · p. 3

3.3 Recursive Parallelization

  • Recursive parallelization pads sequences to a power of two, splits them into parallel subsequences, and reduces forward passes to log2(L), improving training and inference efficiency.Scaled additions of hidden features account for recurrence across subsequences.Source: §3.3 · pp. 4–5

3.4 Overall LRURec Model

  • The model combines item embeddings, normalized LRU blocks with PFFN and residual sublayers, and a next-item predictor that shares the embedding matrix to reduce model size and overfitting.The PFFN is used to compensate for the recurrence’s lack of nonlinearity and model nonlinear transition patterns.Source: §3.4 · p. 5

4 DISCUSSION

LRURec’s discussion attributes its gains to recency weighting and added modeling capacity, and reports lower complexity for full-sequence and incremental inference than the cited recurrent and transformer approaches.

4 DISCUSSION

  • Table 2 presents dataset statistics after preprocessing.Source: Table 2 · p. 6
  • LRURec’s recency weighting favors recent interactions, while PFFN and hierarchical blocks improve long-range modeling, especially on long sequences and dense data.On ML-1M, PFFN and stacking an LRU block yield up to 11.24% and 3.43% performance improvements, respectively; on sparse Beauty, each contributes less than 1%.Source: §4 · pp. 5–6
  • Full-sequence processing costs O(log(L)H^2), and incremental inference costs O(H^2), versus O(LH^2) for typical RNNs and O(L^2H + LH^2) for original transformers.With H = 64 in the experiments, LRURec’s parameter count is comparable to RNN-based recommenders and does not grow with the number of users.Source: §4 · p. 6

5 EXPERIMENTS

LRURec consistently outperforms benchmark baselines, including on long histories, while efficiency tests report rapid training convergence and fast incremental inference. Ablations and hyperparameter tests show how component contributions and tuning vary across datasets.

5.1 Experimental Setup

  • Evaluation spans six recommendation datasets and compares factorization/Markov, RNN, transformer, and MLP baselines using Recall@k and NDCG@k.Splits use leave-last-out, excluding users and items with fewer than five interactions; maximum lengths are 200 for ML-1M, 1000 for XLong, and 50 elsewhere. Scores are not directly comparable with studies using k-core preprocessing.Source: §5.1 · p. 6

5.2 Overall Performance

  • LRURec beats baselines across datasets and metrics (Table 3), averaging 4.39% improvement; average NDCG@10 gains are 6.73% versus 3.08% for Recall@10.Gains average 10.48% on sparse Sports versus 1.32% on dense ML-1M, with 17.43% on Sports NDCG@10.Source: Table 3 · p. 7, §5.2 · p. 7

5.3 Long-Range Modeling Performance

  • On XLong's ∼1k-length sequences, LRURec achieves 8.53% average improvement across metrics over the best-performing baseline, SASRec.Average |λ| is ∼0.8 on XLong versus ∼0.3 on Beauty and Video; Sports has relatively high |λ| despite short sequences.Source: Table 4 · p. 7, §5.3 · pp. 7–8, Table 5 · p. 7

5.4 Ablation Studies

  • Removing PFFN drops ML-1M average performance by 11.24%, while one-layer LRURec outperforms the second-best backbone variant by 7.70% on Recall@10.Layer normalization and residual connections contribute strongly on sparse Beauty; PFFN improves non-linear transition modeling on dense ML-1M.Source: Table 6 · p. 8, §5.4 · p. 8

5.5 Model Efficiency

  • LRURec reaches early stopping at ∼23k training steps, versus over 50k for SASRec and over 80k for GRU4Rec; at length 50, it handles ×7.3 more input examples than SASRec.Incremental inference throughput for LRURec is independent of sequence length.Source: §5.5 · p. 8

5.6 Hyperparameter Sensitivity

  • LRURec is robust to weight-decay changes, but optimal dropout varies by dataset: 0.2 for dense ML-1M and 0.6 for sparse Video and Beauty.Weight-decay performance changes little until the value reaches 1.Source: Figure 4 · p. 9, §5.6 · p. 9

6 CONCLUSION

Across experiments on multiple real-world datasets, LRURec achieves superior results over state-of-the-art baselines. Its design combines linear recurrence with matrix diagonalization and recursive parallelization, alongside improvements for recommendation performance and inference efficiency.

  • LRURec consistently achieves superior results over state-of-the-art baselines across multiple real-world datasets.Source: §6 · p. 9
  • LRURec uses linear recurrence with matrix diagonalization to efficiently capture user transition patterns.Source: §6 · p. 9
  • Its recursive parallelization framework significantly accelerates both training and inference.Source: §6 · p. 9
  • LRURec incorporates further improvements targeting recommendation performance and inference efficiency.Source: §6 · p. 9
Loading 2310.02367v2…