Lightweight Self-Attentive Sequential Recommendation

1 INTRODUCTION

• Item embeddings dominate memory use in latent recommenders, while many compositional embedding methods target static rather than sequential recommendation.This matters for edge deployment, where lightweight recommenders can reduce communication latency and retain personal data on users’ devices.Source: §1 · pp. 1–2
• Feeding compositional embeddings into conventional sequential DNNs can retain excessive parameters and computation, motivating a compact sequence-mining design.The paper identifies timeliness of on-device inference as a concern.Source: §1 · p. 2
• LSAN uses substantially smaller base embedding tables and a quotient-remainder scheme to compose unique item vectors that change with sequence-specific context.The context-aware composition accounts for factors such as seasonal changes and adjacent items.Source: §1 · p. 2
• LSAN separates global and local preference modeling into self-attention and convolution branches, avoiding reliance on redundant multi-head attention units.The paper notes that self-attention can overemphasize adjacent-item relationships, while multi-head attention is used to acquire global information.Source: §1 · p. 2
• Experiments on three benchmark datasets show LSAN has advantageous effectiveness and memory efficiency compared with state-of-the-art baselines.The introduction reports this comparison without giving metric values.Source: §1 · p. 2

2 PROBLEM FORMULATION

• Given a user’s ordered sequence of T interacted items, the goal is to produce a top-K ranking of items for the next time step, T + 1.Each item has a sequence-position index and a unique fixed global index in the item set.Source: §2 · p. 2

3 METHODOLOGY

• LSAN has a dynamic context-aware compositional embedding layer and a twin-attention network for learning global and local preferences without redundant multi-head self-attention modules.The paper presents these as its two main components.Source: §3 · p. 2

3.1 Dynamic Context-aware Compositional Embedding

• LSAN replaces the O(|V|D) item table with N base tables whose dimensions are far smaller than the item set, selecting one base vector from each table for each item.The selected vectors are attentively fused into a single compositional embedding.Source: §3.1 · pp. 2–3
• The product of base-table sizes must cover the item set, and quotient-remainder indexing assigns each item a distinct combination without extra learnable parameters.The first table’s index is computed as the item’s global index modulo that table’s size.Source: §3.1 · p. 3
• For each item, LSAN defines context from the previous and current item categories and the discrete time slot, then uses that context to weight selected base embeddings.The time slot is defined as an hour of the day; a padding category is used for the first item.Source: §3.1 · pp. 3–4
• LSAN injects the context embedding into the composed item embedding through a nonlinear operation before stacking item representations into a sequence matrix.The operation uses concatenation and an MLP mapping from 2D to D.Source: §3.1 · p. 4

3.2 Modelling Long- and Short-term User Preferences with Twin-Attention

• Twin-attention pairs a global self-attention branch with a local convolution branch, separating long- and short-term preference modelling.The design is intended to reduce the redundancy of using excessive self-attention units for both patterns.Source: §3.2 · p. 4, §3.2.2 · p. 4

3.2.1 Convolution Branch for Local Patterns.

• Depth-wise convolution extracts local dependencies within an L-sized sliding window using O(LD) parameters rather than O(LD^2).Each output row represents an item’s interactions with nearby items in the sequence.Source: §3.2.1 · p. 4

3.2.2 Self-attention Branch for Global Patterns.

• Position-aware scaled dot-product self-attention models long-range dependencies among items regardless of their distance in the sequence.The item embeddings are paired with positional context before attention is applied.Source: §3.2.2 · p. 4

3.2.3 Enhancing Expressiveness with Parallelism.

• With the same total of 2H heads, twin-attention uses O(H(LD + 3D^2)) parameters versus O(6HD^2) for self-attention and can yield stronger performance.LSAN does not require H > 1 for its twin-attention design to learn global and local preferences.Source: §3.2.3 · p. 4

3.3 Prediction Layer

• A point-wise feed-forward network adds non-linearity to the twin-attention output.The described output has shape T × D, with GeLU used as the non-linearity.Source: §3.3.1 · pp. 4–5

3.3.1 Point-wise Feed-forward Network.

• The final representation is used to estimate a probability for each item, forming a |V|-dimensional distribution for user u.The item scores are sorted in descending order and the top K become the recommendations.Source: §3.3.1 · p. 5

• The learning objective uses cross-entropy to measure next-item prediction error and applies L2 regularization with coefficient λ.Each training target y_s is a one-hot vector for the ground-truth next item.Source: §3.4 · p. 5

4 EXPERIMENTS

• The experiment section evaluates accuracy and model size, then investigates the effects of LSAN’s key components and hyperparameters.Source: §4 · p. 5

4.1.1 Dataset.

• The four benchmark datasets are highly sparse and include Amazon Beauty, Sports, and Toys, plus Yelp check-in data.Interactions are grouped by user chronologically, and users or items with fewer than five interactions are discarded.Source: §4.1.1 · p. 5

4.2 Evaluation Metrics

• Evaluation uses leave-one-out splits and reports HR@{5, 10, 20} and nDCG@{5, 10, 20} averaged over users after ranking against the full item set.The last interaction is the test instance, the second-last is validation, and earlier interactions are used for training.Source: §4.2 · p. 5

4.3 Baseline Methods

• The baseline set includes FPMC, GRU4Rec, Caser, SASRec, and BERT4Rec, spanning matrix-factorization/Markov, recurrent, convolutional, and self-attentive approaches.BERT4Rec adds a Cloze objective and bidirectional self-attention to SASRec.Source: §4.3 · p. 5

4.4 Implementation Details

• All methods use embedding dimension 128; LSAN uses one twin-attention layer, two heads per branch, and N = 2 while m1 varies for compression tuning.The reported setup uses Adam with batch size 256, learning rate 0.001, and L2 strength 1e−5.Source: §4.4 · pp. 5–6

4.5 Overall Performance Comparison

• Among sequential baselines, FPMC performs worst across metrics, while SASRec and BERT4Rec have clear margins over the other baselines.The authors note that BERT4Rec does not outperform SASRec under full-test-set evaluation, unlike in its original paper, and suggest possible bias in naive negative-sampled tests.Source: §4.5 · p. 6
• Both LSAN and LSAN_full.emb surpass the best baseline, SASRec, with significant margins on most metrics, and both show reduced model size.The compositional-embedding LSAN performs better than the full-sized-embedding version on all datasets.Source: §4.5 · p. 6

4.6 Impact of Embedding Compression Rate

• At m1=2, LSAN surpasses SASRec on most metrics with around 60% of its parameters across all datasets; at m1=3, it remains comparable on Beauty, Toys, and Yelp with around 45%.Performance drops clearly when m1 exceeds 3, which the authors attribute to under-parameterisation.Source: §4.6 · p. 6, Table 3 · p. 8

4.7 Ablation Study

• Removing temporal information from compositional embeddings causes significant performance drops on most datasets, while replacing twin-attention with plain self-attention drops performance on three datasets.The authors interpret these ablations as evidence for temporal dynamics and for modelling long- and short-term preferences with limited attention heads.Source: §4.7 · p. 6, Table 4 · p. 8

4.8.1 Impact of Dimension Size.

• The best dimension is D=64 on the most sparse Sports dataset and D=128 on the other datasets; larger dimensions such as 256 may not improve performance because of overfitting.A dimension of 16 cannot preserve sufficient latent item information, and performance increases as dimension grows from that small setting.Source: §4.8.1 · pp. 6–7, Figure 2 · p. 7

4.8.2 Impact of Attention Head Number.

• Most datasets perform best with 2 self-attention heads and 2 convolution heads, which together make up LSAN’s twin-attention heads.The reported trend is that more attention heads improve performance.Source: §4.8.2 · p. 7, Figure 2 · p. 7

4.8 Hyper-parameter Analysis

• A one-layer LSAN performs best, while stacking more twin-attention layers causes performance to drop across datasets, possibly because of overfitting on sparse data.The authors state that twin-attention captures varied sequential information with one layer.Source: Figure 2 · p. 7, §4.8.3 · p. 8

4.9 Attention Weight Visualisation

• On Yelp, LSAN’s self-attention heat maps show global patterns, unlike SASRec’s local-pattern concentration; LSAN assigns local-pattern modelling to its convolution branch.The figure compares average attention weights for three samples over the last 10 items across attention heads.Source: §4.9 · p. 8, Figure 3 · p. 8

5.1 Sequential Recommendation

• Early sequential recommendation mainly used Markov chains; FPMC combined matrix factorisation and Markov-chain modelling to learn item-to-item transition probabilities for next-item prediction from the latest interaction, while GRUs extracted high-order sequential information.Subsequent RNN-based approaches used attention, memory networks, copy mechanisms, or reinforcement learning. Convolutional methods captured local dependencies among items.Source: §5.1 · pp. 8–9
• Transformer-based self-attention supports parallel sequence modelling; SASRec and BERT4Rec apply self-attentive architectures to sequential recommendation.The paper describes large item-embedding tables as a memory cost and cites concerns that multi-head self-attention can emphasize local dependencies over global preferences.Source: §5.1 · p. 9

5.2 Lightweight Deep Learning Models

• Lightweight-model approaches include pruning, quantisation, knowledge distillation, and compositional embeddings; prior embedding-compression methods often target static recommendation or require per-item codes.LSAN’s context-aware temporal composition is presented as addressing the omission of changing user interests in static recommendation methods.Source: §5.2 · p. 9

6 CONCLUSION

• LSAN’s two parallel branches specialize in short- and long-term user preference modelling.The conclusion describes LSAN as a lightweight twin-attention sequential recommender.Source: §6 · p. 9
• Temporal context-aware compositional embeddings reduce memory cost while preserving the intrinsic temporal dynamics of sequential data.Source: §6 · p. 9
• Experiments on four real-world datasets demonstrate LSAN’s effectiveness and efficiency.Source: §6 · p. 9