Source-linked AI summary
Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer
Zhiwei Liu, Ziwei Fan, Yu Wang, Philip S. Yu
TL;DR
Transformer-based sequential recommenders perform poorly on short sequences because those histories provide limited informative context. ASReP reverse-pre-trains a transformer to generate pseudo-prior items, then fine-tunes it left to right on augmented sequences; experiments report improved performance across two real-world datasets.
Problem
Transformer-based sequential recommenders struggle with short sequences, where limited history provides insufficient informative context for encoding.
Method
ASReP reverse-pre-trains a transformer to predict prior items, prepends recursively generated pseudo-prior items to short sequences, and fine-tunes left to right for next-item prediction.
Results
ASReP outperforms transformer-based baselines across metrics, with average relative improvements over SASRec of 19.1% on Beauty and 21.8% on Phones.
Takeaways & Limitations
The results support pseudo-prior augmentation as an effective approach for improving sequential recommendation, particularly on short sequences.
Abstract
from arXiv · showhide
Sequential Recommendation characterizes the evolving patterns by modeling item sequences chronologically. The essential target of it is to capture the item transition correlations. The recent developments of transformer inspire the community to design effective sequence encoders, \textit{e.g.,} SASRec and BERT4Rec. However, we observe that these transformer-based models suffer from the cold-start issue, \textit{i.e.,} performing poorly for short sequences. Therefore, we propose to augment short sequences while still preserving original sequential correlations. We introduce a new framework for \textbf{A}ugmenting \textbf{S}equential \textbf{Re}commendation with \textbf{P}seudo-prior items~(ASReP). We firstly pre-train a transformer with sequences in a reverse direction to predict prior items. Then, we use this transformer to generate fabricated historical items at the beginning of short sequences. Finally, we fine-tune the transformer using these augmented sequences from the time order to predict the next item. Experiments on two real-world datasets verify the effectiveness of ASReP. The code is available on \url{https://github.com/DyGRec/ASReP}.
1 INTRODUCTION
Sequential recommendation models chronological item sequences to capture transitions, but transformer encoders struggle with short histories because they provide limited context. ASReP addresses this by adding pseudo-prior items while preserving sequential correlations.
- Sequential recommendation models chronological user-item sequences to infer the next item from evolving behavioral patterns.
- Transformer encoders such as SASRec and BERT4Rec effectively model item-transition correlations from sequence representations.
- Nearly 75% of Amazon Beauty sequences contain fewer than 7 items, and very short sequences have poorer Recall@5 than sequences of length at least 20.
- Short-sequence augmentation must preserve original sequential correlations, while item attributes may not represent complex item transitions.
- ASReP generates pseudo-prior items with a reverse-direction transformer, then fine-tunes on augmented sequences for next-item prediction.
2 PRELIMINARY
The paper formulates sequential recommendation as next-item prediction from chronological user-item sequences. Its transformer backbone combines item and position embeddings with multi-head attention and a feed-forward network.
- 2.1 Problem Definition: Each user sequence lists interacted items in chronological order, and its length equals the number of items in that sequence.
- 2.1 Problem Definition: Sequential recommendation estimates the probability of candidate next items conditioned on a user's training sequence.
- 2.2 Transformer for SR: The transformer represents each sequence position by adding an item embedding to a position embedding.
- 2.2 Transformer for SR: For fixed-length inputs, longer sequences are truncated to their last n items and shorter sequences are zero-padded.
- 2.3 Transformer for SR: Multi-head attention learns item importance at each position, after which a ReLU-based feed-forward network produces position-wise sequence embeddings.
3 PROPOSED MODEL
ASReP uses a SASRec-style transformer in three stages: reverse pre-training, recursive augmentation of short sequences, and left-to-right fine-tuning for next-item prediction.
- 3 PROPOSED MODEL: ASReP adopts the SASRec transformer backbone and comprises reverse pre-training, short-sequence augmentation, and left-to-right fine-tuning.
- 3.1 Reversely Pre-training: Reverse pre-training masks the left prior item and trains the transformer to predict it from the sequence in the right-to-left direction.
- 3.2 Short Sequences Augmentation: The reverse-trained transformer recursively generates k pseudo-prior items and prepends them to sequences whose length is at most M.
- 3.3 Left-to-Right Fine-tuning: Fine-tuning uses augmented sequences from left to right to predict the next item, masking that item to maintain causality and sequential correlations.
- 3 PROPOSED MODEL: The framework diagram depicts reverse prior-item prediction, pseudo-prior generation for short sequences, and next-item prediction with augmented sequences.
4 EXPERIMENTS
Experiments on two Amazon datasets evaluate ASReP against sequential and static baselines, examine parameter sensitivity, and compare performance across sequence lengths. ASReP outperforms the baselines overall and is especially effective for short sequences, while larger augmentation settings improve performance with higher time cost.
- Datasets and Protocols: The evaluation uses Beauty and Phones datasets, with timestamps defining item order and the most recent items reserved for testing and validation.The datasets are publicly available Amazon 5-core collections with predominantly short user sequences.
- Datasets and Protocols: Recall@5, NDCG@5, and MRR are evaluated by ranking the ground-truth item against 100 randomly sampled negative items.Table 1 reports performance comparisons across these three metrics.
- Overall Comparison (RQ1): ASReP outperforms all baselines across Recall@5, NDCG@5, and MRR, with average relative gains over SASRec of 19.1% on Beauty and 21.8% on Phones.It also improves over the second-best model by 8.4% and 8.1% on Beauty and Phones, respectively.
- Parameter Sensitivity (RQ2): Recall@5 improves as the short-sequence threshold M increases on both datasets, while increasing k helps little when M is small.Increasing both parameters also raises time cost, requiring a trade-off between performance and efficiency.
- Performance w.r.t. Sequence Length (RQ3): ASReP significantly outperforms SASRec and BERT4Rec when sequence length is below 20, whereas BERT4Rec performs best at lengths of at least 20.Because most sequences are short, ASReP’s short-sequence gains improve its overall performance.
5 CONCLUSIONS
The paper proposes ASReP to improve sequential recommendation by augmenting short sequences with pseudo-prior items generated through reverse pre-training. Overall comparisons and detailed analyses verify its effectiveness, particularly for short sequences.
- Conclusions: ASReP reversely pre-trains a transformer to generate pseudo-prior items, then fine-tunes it left-to-right to predict the next item.The framework targets sequential recommendation with short sequences.
- Conclusions: Experiments verify ASReP’s effectiveness, with detailed analyses showing significant performance improvements for short sequences.