PaperlayerView exact revision on arXiv

Source-linked AI summary

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer

Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, Peng Jiang

arXiv:1904.06690v2cs.IRcs.LG

TL;DR

Sequential recommendation models often rely on rigid left-to-right encoding, limiting how they represent users’ historical behavior. BERT4Rec uses bidirectional self-attention trained with a Cloze objective to predict masked items from both-sided context, and it outperforms state-of-the-art baselines on four real-world datasets.

  • Problem

    Sequential recommendation models commonly assume rigid left-to-right behavior sequences, limiting the use of context from both sides when representing users’ histories.

  • Method

    BERT4Rec models behavior sequences with bidirectional self-attention and trains by predicting masked items using their left and right context through a Cloze task.

  • Results

    BERT4Rec outperforms state-of-the-art baselines consistently on four real-world datasets.

  • Takeaways & Limitations

    Bidirectional context and Cloze training provide an effective approach for sequential recommendation across four benchmark datasets.

  • Takeaways & Limitations

    BERT4Rec has quadratic per-layer computational complexity in sequence length and does not incorporate rich item features or explicitly model users across multiple sessions.

Abstract

from arXiv · show

Modeling users' dynamic and evolving preferences from their historical behaviors is challenging and crucial for recommendation systems. Previous methods employ sequential neural networks (e.g., Recurrent Neural Network) to encode users' historical interactions from left to right into hidden representations for making recommendations. Although these methods achieve satisfactory results, they often assume a rigidly ordered sequence which is not always practical. We argue that such left-to-right unidirectional architectures restrict the power of the historical sequence representations. For this purpose, we introduce a Bidirectional Encoder Representations from Transformers for sequential Recommendation (BERT4Rec). However, jointly conditioning on both left and right context in deep bidirectional model would make the training become trivial since each item can indirectly "see the target item". To address this problem, we train the bidirectional model using the Cloze task, predicting the masked items in the sequence by jointly conditioning on their left and right context. Comparing with predicting the next item at each position in a sequence, the Cloze task can produce more samples to train a more powerful bidirectional model. Extensive experiments on four benchmark datasets show that our model outperforms various state-of-the-art sequential models consistently.

1 INTRODUCTION

User interests evolve with historical behavior, but conventional left-to-right sequential recommenders limit historical representations and cannot directly use bidirectional context without information leakage. BERT4Rec addresses these issues with deep bidirectional self-attention trained by a Cloze objective that predicts randomly masked items from surrounding context.

  • Motivation: Users’ interests are dynamic and evolving, so sequential recommendation methods use historical interactions to predict successive items [15].For example, purchasing a Nintendo Switch may increase the likelihood of subsequently purchasing Joy-Con controllers.
  • Problem: Left-to-right unidirectional models restrict each historical item to information from previous items, weakening sequence representations despite their effectiveness [14] [15].The limitation is illustrated in Figure 1c and 1d.
  • Problem: Bidirectional next-item prediction causes information leakage because each item can indirectly see the target item, making future prediction trivial.This prevents the network from learning useful representations.
  • Training objective: The Cloze objective randomly replaces items with a special [mask] token and predicts their IDs from surrounding context, avoiding leakage while learning bidirectional representations.The paper evaluates the bidirectional architecture and Cloze objective quantitatively on four benchmark datasets.
  • Proposed approach: BERT4Rec models user behavior sequences with a deep bidirectional self-attention network trained through the Cloze task [6].The paper presents this as the first study to introduce a deep bidirectional sequential model and Cloze objective into recommendation systems.

2 RELATED WORK

Prior recommendation methods range from collaborative filtering and deep preference models to sequential architectures based on Markov chains, recurrent networks, convolutions, memory networks, and attention. BERT4Rec differs from closely related SASRec and RNN-based methods by using a bidirectional Cloze-trained model rather than left-to-right next-item prediction.

  • General Recommendation: Collaborative filtering methods model preferences from interaction histories using matrix factorization or item-based neighborhoods, but they ignore the order of user behaviors and therefore are not sequential recommendation methods.Matrix factorization projects users and items into a shared vector space and estimates preference through their inner product [26, 27, 41].
  • Sequential Recommendation: Sequential recommendation has progressed from Markov-chain models such as MDP-based recommendation and FPMC to recurrent models including GRU4Rec and related RNN variants,, [15].RNN-based methods encode previous records into a user-preference vector using recurrent architectures and different loss functions [7, 14, 15, 28, 37, 56, 58].
  • Attention Mechanism: Attention is used both as an added component, such as attention-enhanced GRU models [28], and as the sole basis of Transformer and BERT architectures for sequence modeling, [6].Transformer and BERT rely solely on multi-head self-attention, motivating purely attention-based approaches to sequential data [30, 32, 38, 46? ].
  • Attention Mechanism: SASRec applies a Transformer decoder with causal masking for left-to-right next-item prediction, whereas BERT4Rec learns bidirectional representations through the Cloze task.The architectural distinction is explicitly contrasted with RNN-based methods: BERT4Rec is bidirectional and Cloze-trained, while SASRec and RNN methods are unidirectional and sequentially predict the next item.

3 BERT4REC

BERT4Rec models sequential recommendations with bidirectional Transformer self-attention, replacing left-to-right sequence encoding with representations that integrate information across all positions. It trains this bidirectional model through a Cloze objective that predicts randomly masked items from their surrounding context.

  • 3 BERT4REC: The model stacks L bidirectional Transformer layers built from multi-head self-attention and position-wise feed-forward networks, with residual connections and layer normalization supporting deeper networks.Self-attention directly captures item-item interactions across the sequence, while stacking layers learns more complex transition patterns.
  • 3 BERT4REC: BERT4Rec uses bidirectional self-attention to model behavior sequences, capturing dependencies at arbitrary distances and producing more powerful sequence representations than left-to-right architectures.Its Transformer layers exchange information across all positions in parallel, providing a global receptive field unlike recurrent stepwise encoding and limited-range convolution.
  • 3 BERT4REC: BERT4Rec injects learnable positional embeddings into item embeddings so the order-sensitive sequence information remains available to the Transformer.The positional embedding matrix imposes a maximum sequence length, with inputs truncated to the last N items when necessary.
  • Training: Instead of predicting each next item from left-to-right prefixes, BERT4Rec randomly masks a proportion ρ of sequence items and predicts their original identities using the Cloze objective.This objective enables bidirectional conditioning while avoiding direct access to the target item; SASRec instead predicts the next item with a unidirectional causal architecture.

4 EXPERIMENTS · 4.1 Datasets · 4.2 Task Settings & Evaluation Metrics

Experiments evaluate BERT4Rec on four diverse real-world datasets using standardized implicit-feedback sequences and leave-one-out next-item recommendation. Performance is measured with HR, NDCG, and MRR under the specified ranking protocol.

  • 4.1 Datasets: The model is evaluated on four representative real-world datasets spanning substantially different domains and sparsity levels.
  • 4.1 Datasets: The datasets include Amazon Beauty, derived from Amazon product reviews by McAuley et al. [34], and Steam, collected from a large video-game distribution platform.
  • 4.1 Datasets: Ratings and reviews are converted into implicit interaction feedback, then grouped by user and chronologically sorted to form behavior sequences.
  • 4.1 Datasets: Following common practice [12], preprocessing retains users with at least five feedbacks, with processed-dataset statistics summarized in Table 1.
  • 4.2 Task Settings & Evaluation Metrics: The leave-one-out next-item task holds out each user’s final interaction for testing, the penultimate interaction for validation, and earlier interactions for training.
  • 4.2 Task Settings & Evaluation Metrics: Each test ground-truth item is paired with 100 random items, and ranking quality is reported using HR, NDCG, and MRR.
  • 4.2 Task Settings & Evaluation Metrics: HR and NDCG are reported at k = 1, 5, 10; with one ground-truth item per user, HR@k equals Recall@k, scales with Precision@k, and MRR equals MAP.

4.3 Baselines & Implementation Details

BERT4Rec is evaluated against representative popularity, factorization, neural interaction, Markov-chain, and sequential recommendation baselines using author code or TensorFlow reimplementations. Hyperparameters are tuned for optimal baseline results, while BERT4Rec uses specified TensorFlow, Adam, initialization, regularization, and architecture settings.

  • Baselines: The comparison includes POP, BPR-MF, NCF [12], FPMC, and GRU4Rec [15] as representative baselines.These baselines respectively cover popularity ranking, pairwise matrix-factorization ranking, MLP interaction modeling, MF combined with first-order Markov chains, and GRU-based sequential recommendation.
  • Implementation: NCF, GRU4Rec, GRU4Rec+, Caser, and SASRec use code from their authors, whereas BPR-MF and FPMC are implemented in TensorFlow.The passage specifies author-provided implementations for the former models and TensorFlow reimplementations for the latter two.
  • Implementation: Baseline hyperparameters are selected from specified hidden dimensions, ℓ2 regularization values, and dropout rates, with other settings followed from authors’ recommendations or tuned on validation sets.Results are reported for each baseline under its optimal hyperparameter configuration.
  • Implementation: BERT4Rec is implemented in TensorFlow with truncated-normal initialization, Adam optimization at learning rate 1e-4, weight decay 0.01, linear learning-rate decay, and gradient clipping above ℓ2 norm 5.The implementation uses β1 = 0.9 and β2 = 0.999 for Adam.

4.4 Overall Performance Comparison

BERT4Rec achieves the best performance across four benchmark datasets and all evaluation metrics, with substantial average gains over the strongest baselines. The comparisons indicate that sequential modeling, self-attention, bidirectional context, and the Cloze objective contribute to performance and modeling behavior.

  • Overall Performance Comparison: Sequential methods consistently outperform non-sequential baselines, while Caser benefits from high-order Markov chains and SASRec outperforms GRU4Rec and GRU4Rec+ through self-attention.Caser is especially strong on dense ML-1m, but its limited scalable order causes weaker performance than GRU4Rec+ and SASRec on sparse datasets.
  • Overall Performance Comparison: BERT4Rec performs best on all four datasets and evaluation metrics, improving average HR@10 by 7.24%, NDCG@10 by 11.03%, and MRR by 11.46% over the strongest baselines.Table 2 reports statistically significant improvements over baselines with p < 0.01.
  • Analysis on bidirection and Cloze: BERT4Rec with one masked item significantly outperforms SASRec on all metrics, isolating the benefit of jointly using left and right context.The comparison uses Beauty and ML-1m with hidden dimensionality d = 256; under one-mask training, the key difference is bidirectional conditioning.
  • Attention Analysis: Attention patterns vary across heads and layers: some heads favor left or right items, while later layers focus more on recent items and may attend to [mask].The analysis visualizes average attention weights for the last 10 items on Beauty using representative heat-maps.

4.5 Impact of Hidden Dimensionality d

Increasing hidden dimensionality does not necessarily improve recommendation performance, with model performance tending to converge as d grows and overfitting likely affecting sparse datasets. Because BERT4Rec performs satisfactorily for d≥64, subsequent analyses use d=64.

  • 4.5 Impact of Hidden Dimensionality d: Performance tends to converge as hidden dimensionality increases, and larger d does not necessarily improve results, especially on sparse Beauty and Steam datasets.The passage attributes this pattern probably to overfitting.
  • 4.5 Impact of Hidden Dimensionality d: Self-attention methods, including SASRec and BERT4Rec, achieve superior performance even with relatively small hidden dimensionality.
  • 4.5 Impact of Hidden Dimensionality d: Because BERT4Rec achieves satisfactory performance with d≥64, the following analysis reports results with d=64.

4.6 Impact of Mask Proportion ρ

Mask proportion ρ must balance sufficient masked items against trainability, with performance generally declining beyond ρ>0.6. The optimal ρ depends strongly on dataset sequence length, favoring larger values for short sequences and smaller values for long sequences.

  • Impact of Mask Proportion ρ: Mask proportion ρ should be neither too small nor too large: small values provide insufficient learning signal, while large values require guessing too many items from few contexts.For ML-1m and Beauty, ρ=0.6 corresponds to predicting 98=⌊163.5×0.6⌋ and 5=⌊8.8×0.6⌋ items per sequence, respectively, making training harder for ML-1m.
  • Impact of Mask Proportion ρ: Performance decreases as ρ increases beyond 0.6 across datasets, while ρ=0.2 outperforms ρ=0.1 on all datasets.Figure 4 evaluates mask proportions from 0.1 to 0.9 on d=64.
  • Impact of Mask Proportion ρ: The optimal ρ depends on sequence length: short-sequence Beauty and Steam prefer ρ=0.6 and ρ=0.4, whereas long-sequence ML-1m and ML-20m prefer ρ=0.2.The study attributes this dependence to the larger number of items that must be predicted in long sequences at the same mask proportion.

4.7 Impact of Maximum Sequence Length N

The optimal maximum sequence length depends strongly on dataset sequence length: Beauty performs best with N = 20, whereas ML-1m performs best with N = 200. Despite quadratic per-layer complexity, GPU parallelization makes self-attention effective for longer sequences.

  • 4.7 Impact of Maximum Sequence Length N: Beauty achieves its best recommendation performance with N = 20, while ML-1m performs best with N = 200.The proper maximum length is highly dependent on the dataset’s average sequence length.
  • 4.7 Impact of Maximum Sequence Length N: The dataset-dependent optima suggest that recent items matter more in short-sequence datasets, whereas less recent items matter in long-sequence datasets.
  • 4.7 Impact of Maximum Sequence Length N: Although BERT4Rec has per-layer complexity O(n^2d), GPU parallelization makes its self-attention layer effective for longer sequences.Table 4 reports recommendation performance and training speed across maximum sequence lengths.

4.8 Ablation Study

The ablations show that BERT4Rec’s components have dataset- and sequence-length-dependent effects. Positional embeddings and architectural depth are especially important for long sequences, while regularization components matter more on small datasets.

  • (1) PE: Positional embeddings are crucial on long-sequence datasets, whose performance decreases dramatically when they are removed because masked targets otherwise share the same hidden representation.Without positional embeddings, each item representation depends only on item embeddings, making predictions ill-posed; the problem worsens as the number of masked items increases.
  • (2) PFFN: PFFN benefits long-sequence datasets such as ML-20m by integrating information from multiple attention heads.The analysis links this benefit to the greater preference of long-sequence datasets for multiple heads.
  • (3) LN, RC, and Dropout: LN, residual connections, and dropout primarily alleviate overfitting and are more effective on small datasets such as Beauty.On ML-20m with L = 4, removing residual connections decreases NDCG@10 by about 10%.
  • (4) Number of layers L: Increasing the layer number L improves performance especially on large datasets such as ML-20m, whereas L = 4 declines on Beauty because of overfitting.The result supports using deeper self-attention to learn more complex item-transition patterns.
  • (5) Head number h: Long-sequence datasets such as ML-20m benefit from larger head numbers h, while short-sequence datasets such as Beauty prefer fewer heads.This pattern is consistent with the reported empirical finding [48] that larger h is essential for capturing long-distance dependencies with multi-head self-attention.

5 CONCLUSION AND FUTURE WORK

The paper introduces BERT4Rec, a deep bidirectional sequential recommendation model trained with a Cloze task that predicts masked items from both left and right context. Experiments on four real-world datasets show that it outperforms state-of-the-art baselines, while future work will incorporate rich item features and explicit user modeling across multiple sessions.

  • 5 CONCLUSION AND FUTURE WORK: BERT4Rec uses deep bidirectional self-attention for sequential recommendation and trains by predicting masked items from both left and right context.The Cloze task is the paper’s central training mechanism for the bidirectional sequential model.
  • 5 CONCLUSION AND FUTURE WORK: BERT4Rec outperforms state-of-the-art baselines across four real-world datasets.The conclusion reports extensive experimental support for the model’s effectiveness.
  • 5 CONCLUSION AND FUTURE WORK: Future work will incorporate rich item features, such as product category and price or movie cast, instead of modeling only item ids.The proposed features are intended to enrich the model’s item representations.
  • 5 CONCLUSION AND FUTURE WORK: Another future direction is adding a user component for explicit user modeling when users have multiple sessions.This extension targets settings in which a single user generates multiple sessions.
Loading 1904.06690v2…