PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupled Side Information Fusion for Sequential Recommendation

Yueqi Xie, Peilin Zhou, Sunghun Kim

arXiv:2204.11046v1cs.IRcs.AI

TL;DR

Sequential recommendation methods often integrate heterogeneous side information before attention, but this can constrain attention expressiveness and gradient flexibility while mixing unrelated correlations. DIF-SR moves fusion into the attention layer, decouples item and attribute attention calculations, and adds auxiliary attribute predictors; experiments across four datasets show consistent superiority over established SR approaches and compatibility with existing attention-based models.

  • Problem

    Early integration of item and side-information embeddings limits attention-matrix rank and gradient flexibility while introducing mixed correlations among heterogeneous information resources.

  • Method

    DIF-SR generates separate keys and queries for each item and side-information representation, fuses their attention matrices, and uses auxiliary attribute predictors during multi-task training.

  • Results

    DIF-SR outperforms basic and competitive side-information-integrated SR models on four public datasets and can be incorporated into attention-based SR models.

  • Takeaways & Limitations

    Decoupled attention provides a reusable way to incorporate side information while increasing attention representation capacity and enabling flexible learning of its relative importance.

  • Takeaways & Limitations

    Integrated embeddings in prior attention solutions still suffer from compound attention space and degraded attention-matrix rank and training-gradient flexibility.

Abstract

from arXiv · show

Side information fusion for sequential recommendation (SR) aims to effectively leverage various side information to enhance the performance of next-item prediction. Most state-of-the-art methods build on self-attention networks and focus on exploring various solutions to integrate the item embedding and side information embeddings before the attention layer. However, our analysis shows that the early integration of various types of embeddings limits the expressiveness of attention matrices due to a rank bottleneck and constrains the flexibility of gradients. Also, it involves mixed correlations among the different heterogeneous information resources, which brings extra disturbance to attention calculation. Motivated by this, we propose Decoupled Side Information Fusion for Sequential Recommendation (DIF-SR), which moves the side information from the input to the attention layer and decouples the attention calculation of various side information and item representation. We theoretically and empirically show that the proposed solution allows higher-rank attention matrices and flexible gradients to enhance the modeling capacity of side information fusion. Also, auxiliary attribute predictors are proposed to further activate the beneficial interaction between side information and item representation learning. Extensive experiments on four real-world datasets demonstrate that our proposed solution stably outperforms state-of-the-art SR models. Further studies show that our proposed solution can be readily incorporated into current attention-based SR models and significantly boost performance. Our source code is available at https://github.com/AIM-SE/DIF-SR.

1 INTRODUCTION

DIF-SR addresses limitations of early side-information integration in sequential recommendation by decoupling item and attribute attention calculations. Across four datasets, it outperforms basic and competitive side-information-aware SR methods and can enhance existing attention-based models.

  • Motivation: Early integration creates a rank bottleneck that limits attention-score representation capacity and mixes heterogeneous correlations, causing unrelated information to attend together.The bottleneck is tied to the multi-head query-key down-projection size d_h.
  • Approach: DIF-SR moves side-information fusion into the attention layer and separately generates keys and queries for each attribute and the item representation.The resulting attention matrices are fused after decoupled calculation.
  • Approach: Decoupled attention enables higher-rank attention matrices, flexible gradients, and reduced disturbance from mixed heterogeneous embeddings.These properties are intended to improve side-information fusion and item-representation learning.
  • Results: DIF-SR outperforms basic SR and competitive side-information-integrated SR methods on Beauty, Sports, Toys, and Yelp.The method is also designed for incorporation into existing self-attention-based SR models.
  • Contributions: The framework combines DIF attention with an auxiliary attribute prediction training scheme that can boost attention-based recommender systems.The contribution includes theoretical and empirical analyses, ablations, and interpretability studies.

2 RELATED WORK

Prior sequential recommendation work progressed from classical sequence models to neural and self-attention approaches. Side-information methods either separate representations, use pretraining, or integrate attributes before attention, but earlier approaches do not directly support attentive aggregation with side information.

  • Sequential Recommendation: Early sequential recommendation methods relied on Markov Chain assumptions and Matrix Factorization, while later work adopted CNNs and RNNs for complex sequence patterns.The passage introduces the transition toward neural sequential recommendation models.
  • Attention-Based SR: Self-attention-based sequential recommendation became a mainstream approach with competitive performance.Recent improvements include incorporating side information such as item attributes and ratings.
  • Side Information Fusion: FDSA uses separate self-attention blocks for item and feature representations, while S3-Rec uses pretraining tasks to incorporate side information.These methods place side-information use at different stages of the recommendation pipeline.
  • Side Information Fusion: FDSA and S3-Rec do not effectively and directly use side information to support attentive aggregation of item representations and prediction.Their independent-learning and pretraining strategies limit direct interaction with item self-attention.
  • Early Integration: Recent methods such as p-RNN and ICAI-SR integrate side-information embeddings into item representations before the attention layer.The cited approaches use direct concatenation or item-attribute aggregation before attentive learning.

3 PROBLEM FORMULATION

The problem formulation represents each user by a chronologically ordered interaction sequence and uses item and side-information data to define next-item prediction. Side information may describe users, items, or actions and provides extra predictive information.

  • Interaction Sequence: For a user u, the historical interactions are represented as a chronologically ordered sequence S_u = [v_1, v_2, ..., v_n].Each v_i denotes the user’s ith interaction.
  • Side Information: Side information consists of attributes of users, items, and actions that provide extra information for prediction.It supplements the item interaction sequence in the sequential recommendation setting.
  • Prediction Task: Given S_u, the task is to predict the item I_pred that user u will interact with most highly.The formulation frames sequential recommendation as next-item prediction.

4 METHODOLOGY

DIF-SR replaces early side-information fusion with decoupled attention calculations for item and attribute representations, then combines their attention matrices. Its design targets higher attention expressiveness, more flexible gradients, and direct interaction between side information and item representation.

  • Overall Architecture: DIF-SR uses stacked DIF attention and feed-forward blocks that update item representations from item embeddings and auxiliary side information embeddings.The auxiliary side information embeddings remain fixed across layers to reduce computation and avoid overfitting.
  • Prior Attention Solutions: Prior SASRecF-style attention fuses side information into item representations before computing key, query, and value.This early fusion directly influences item representation learning but can invade the item representation.
  • Prior Attention Solutions: NOVA separates value computation from its integrated representation, but its integrated key and query still create compound attention spaces and limit rank and gradient flexibility.The method keeps value non-invasive while retaining early integration for key and query.
  • DIF Attention: DIF-SR generates separate key and query attention calculations for the item and each attribute, then fuses the resulting attention matrices.Attribute-specific projections use dimensions d_fj ≤ d, with fusion functions including addition, concatenation, and gating.
  • Theoretical Analysis: Theoretical analysis states that early fusion bounds attention-matrix rank by d_h, whereas DIF-SR breaks this bottleneck by fusing decoupled item and attribute attention scores.The analysis also shows that simple-addition early fusion can force shared gradients, while DIF-SR supports more flexible gradient behavior.
  • Prediction Module with AAP: AAP applies auxiliary attribute predictors to final representations so item learning incorporates useful side information during training.The paper reports that AAP further improves performance, especially when combined with DIF.

5 EXPERIMENTS

The experiments evaluate DIF-SR on four real-world datasets using established sequential-recommendation baselines, full-ranking evaluation, and Recall@K and NDCG@K metrics. They address overall performance, integration into attention-based models, component effects, and attention-matrix visualization.

  • Research questions: The experiments ask whether DIF-SR surpasses state-of-the-art baselines, whether DIF and AAP improve existing attention models, and how framework components and hyperparameters affect performance.A fourth question examines whether attention-matrix visualization supports the rationale for DIF’s attention fusion.
  • Datasets: Experiments use four real-world datasets: Amazon Beauty, Sports, Toys, and Yelp.Amazon datasets use fine-grained product categories and position information as attributes; Yelp uses business categories and position information.
  • Evaluation: Leave-one-out full-ranking evaluation reserves each user’s final two interactions for validation and testing, using Recall@10, Recall@20, NDCG@10, and NDCG@20.Earlier interactions are used for training, and the models are evaluated under full ranking for fair comparison.
  • Baselines: The study compares strong basic sequential-recommendation methods with recent side-information-integrated methods.Baselines include GRU4Rec, Caser, BERT4Rec, SASRec, SASRecF, and S3-Rec.
  • Implementation: All baselines and DIF-SR use the RecBole framework and the same general training setting, with grid search applied to relevant hyperparameters.Models are trained with Adam for 200 epochs using batch size 2048 and learning rate 1e-4.

5.2 Overall Performance (RQ1)

DIF-SR consistently outperforms both basic and side-information-integrated sequential-recommendation models across all four datasets and evaluation metrics. Its decoupled attention calculation and auxiliary attribute predictors are presented as mechanisms supporting this improvement.

  • Baseline analysis: Simple early fusion does not always improve GRU4RecF and SASRecF over versions without side information.By contrast, carefully designed fusion strategies in S3Rec, NOVA, and ICAI achieve better and competitive performance.
  • Overall comparison: DIF-SR consistently outperforms state-of-the-art sequential-recommendation and side-information-integrated models on all four datasets across all evaluation metrics.The comparison covers both basic SR baselines and recent side-information-aware methods.
  • Interpretation: DIF enhances self-attention expressiveness through higher-rank attention matrices, avoidance of mixed correlations, and more flexible training gradients.AAP further strengthens interactions between side-information and item representations during training.
  • Generalization: The study reports that DIF-SR can be incorporated into self-attention-based sequential-recommendation models while retaining its performance advantages.This motivates the enhancement study on representative attention-based architectures.

5.3 Enhancement Study (RQ2)

The enhancement study tests DIF and AAP as additions to SASRec and BERT4Rec. Both enhanced models significantly outperform their original versions, with the reported gains measured across three datasets.

  • BERT4Rec enhancement: DIF-BERT4Rec achieves 18.46% and 36.16% average relative improvements on Recall@10 and NDCG@10, respectively, across three datasets.These gains compare the enhanced model with the original BERT4Rec.
  • SASRec enhancement: DIF-SASRec achieves 12.42% and 22.64% average relative improvements on Recall@10 and NDCG@10, respectively, across three datasets.These gains compare the enhanced model with the original SASRec.
  • Overall enhancement: The enhanced SASRec and BERT4Rec models significantly outperform their original models.The study uses SASRec and BERT4Rec as unidirectional and bidirectional representative self-attention models, respectively.

5.4 Ablation and Hyper-parameter Study (RQ3)

The ablation and hyper-parameter studies show that DIF is the framework’s most effective component, while AAP further helps when paired with DIF. Performance is generally robust across attribute sizes and fusion functions, with λ values of 5 or 10 performing best.

  • Effectiveness of Different Components: DIF is the most effective component, as removing it causes a much larger performance loss than removing AAP.The comparison is reported on Sports and Yelp datasets.
  • Effectiveness of Different Components: AAP does not always improve early-fusion models because those models do not enable attributes to influence item-to-item attention.AAP further improves performance when combined with DIF.
  • Effectiveness of Different Kinds of Side Information: Item-related attributes substantially improve prediction on Yelp, while combining two attributes provides a further slight improvement.The studied item-related attributes are city and category, alongside position information.
  • Impact of Hyperparameters and Fusion Function: λ values of 5 or 10 achieve the best performance across all four datasets, with λ=10 preferred for Beauty and Yelp.The evaluation uses Recall@20 and NDCG@20.
  • Impact of Hyperparameters and Fusion Function: Performance is consistent across most attribute embedding sizes, allowing model complexity to be reduced by choosing a smaller d_f.The smaller dimension is typically below the item-embedding dimension.
  • Impact of Hyperparameters and Fusion Function: DIF-SR remains robust across different fusion functions, so simple addition does not harm model capacity.The study explicitly evaluates the fusion function F.

5.5 Visualization of Attention Distribution (RQ4)

The visualization separates item and attribute attention matrices before showing their fused matrix. Different attributes exhibit distinct preferences, while fusion adaptively combines their contributions.

  • Attention Matrix Visualization: Each visualization contains two rows for the same head across different layers, with decoupled item and attribute matrices followed by the fused matrix.The fused matrix computes the output item representation for each layer.
  • Attention Matrix Visualization: Different attributes’ decoupled attention matrices show different preferences for capturing data patterns.This provides an interpretable view of heterogeneous side-information behavior.
  • Attention Matrix Visualization: The fused attention matrix adaptively adjusts each side-information contribution and synthesizes crucial patterns.Its inputs are the decoupled attention matrices for the item and attributes.

6 CONCLUSION

DIF-SR addresses limitations of early side-information integration by moving fusion into the attention layer and adding auxiliary attribute prediction. Theoretical analysis and experiments on four public datasets support its effectiveness and extensibility.

  • Conclusion: DIF-SR moves side-information fusion from the input stage to the attention layer to increase attention representation power and gradient flexibility.The decoupled attention mechanism separately handles item and side-information representations.
  • Conclusion: Auxiliary attribute predictors use multi-task training to promote interaction between side information and item representation.The predictors operate on the final representation.
  • Conclusion: Theoretical analysis and experiments on four public datasets show that DIF-SR outperforms basic and competitive side-information-aware SR models.The DIF attention mechanism and AAP training scheme can also be incorporated into attention-based recommender systems.
  • Conclusion: DIF attention and AAP-based training can be readily incorporated into attention-based SR models to benefit side-information fusion.The conclusion also identifies exploring side information at other model stages as future work.

A PROOFS

The proofs establish rank and gradient properties of the DIF attention mechanism using matrix-rank reasoning and constructed identity-matrix cases. They define gradients for item and attribute embeddings under the relevant objectives.

  • Theorem Proofs: The first theorem claim uses rank(MN) ≤ min(rank(M), rank(N)) with W_Q∈R^(d×d_h) to derive the attention-rank bound.The proof begins from the query projection dimensionality.
  • Theorem Proofs: The second theorem claim constructs W_Q=W_K and embedding matrices with identity blocks and zero rows to demonstrate the proposed rank behavior.The construction uses identity submatrices over selected coordinate ranges.
  • Theorem Proofs: The constructed matrix expression uses I for identity matrices and 0 for all-zero matrices.This notation specifies the structure of the proof’s special case.
  • Gradient Proofs: Theorem 4.2 defines gradients for item and attribute embeddings under the objectives in Equations (11) and (12).The proof separately introduces gradients for E_ID and each E_fj.
Loading 2204.11046v1…