Source-linked AI summary
Non-invasive Self-attention for Side Information Fusion in Sequential Recommendation
Chang Liu, Xiaoguang Li, Guohao Cai, Zhenhua Dong, Hong Zhu, Lifeng Shang
TL;DR
Sequential recommendation under BERT lacks a systematic way to use diverse side information, while direct fusion can yield limited or adverse effects. NOVA-BERT keeps item representations separate and uses side information to guide self-attention, achieving state-of-the-art performance across experimental and industrial datasets with negligible computational and model-size overhead.
Problem
BERT was designed for one input type, leaving effective use of diverse side information in sequential recommendation insufficiently studied.
Method
NOVA-BERT uses side information to compute attention guidance while keeping item-ID representations in an independent vector space.
Results
NOVA-BERT stably outperforms other methods across three datasets and all reported metrics, with improvements ranging from 1.48% to 13.51% in HR@10 across tasks.
Takeaways & Limitations
Side information can improve sequential recommendation when used as attention guidance rather than directly altering item representations.
Takeaways & Limitations
The contribution of different side-information types depends on data quality, making side-information selection dataset dependent.
Abstract
from arXiv · showhide
Sequential recommender systems aim to model users' evolving interests from their historical behaviors, and hence make customized time-relevant recommendations. Compared with traditional models, deep learning approaches such as CNN and RNN have achieved remarkable advancements in recommendation tasks. Recently, the BERT framework also emerges as a promising method, benefited from its self-attention mechanism in processing sequential data. However, one limitation of the original BERT framework is that it only considers one input source of the natural language tokens. It is still an open question to leverage various types of information under the BERT framework. Nonetheless, it is intuitively appealing to utilize other side information, such as item category or tag, for more comprehensive depictions and better recommendations. In our pilot experiments, we found naive approaches, which directly fuse types of side information into the item embeddings, usually bring very little or even negative effects. Therefore, in this paper, we propose the NOninVasive self-attention mechanism (NOVA) to leverage side information effectively under the BERT framework. NOVA makes use of side information to generate better attention distribution, rather than directly altering the item embedding, which may cause information overwhelming. We validate the NOVA-BERT model on both public and commercial datasets, and our method can stably outperform the state-of-the-art models with negligible computational overheads.
1 Introduction
Sequential recommendation models users’ changing interests, but BERT’s single-input design makes effective side-information fusion difficult. NOVA addresses this by using side information to guide attention while keeping item representations separate.
- Sequential recommendation predicts users’ next-item interests from historical behaviors while modeling their varying interests over time.
- Neural and transformer-based approaches, including BERT, have advanced sequential recommendation through self-attention and bidirectional processing.
- BERT was designed for one input type, and prior invasive side-information fusion produced scarce, limited, or adverse improvements.
- NOVA uses side information as an auxiliary signal for learning attention distributions instead of fusing it directly into item representations.
- The NOVA-BERT framework efficiently employs various side information and applies the NOVA mechanism to compounded sequential data.
- Experiments and deployment evaluations, including visualization analysis, are conducted to assess NOVA-BERT’s effectiveness and interpretability.
2 Related Works
Related work has increasingly adopted neural and transformer frameworks for sequential recommendation, while attempts to use side information have generally relied on invasive fusion. These approaches often provide limited benefits and may depend on heuristics or specific side-information types.
- Sequential recommendation research has progressed from CNN and RNN models toward transformer-based frameworks such as BERT.
- Previous methods under CNN, RNN, attention, and BERT frameworks attempted to incorporate side information with limited investigation of how it should be added.
- Invasive approaches directly merge side information with item representations using operations such as summation, concatenation, or gated fusion.
- Experiment results indicate that simple fusion approaches cannot effectively leverage rich side information across various scenarios.
- Item boosting incorporates dwell time through the loss function but relies heavily on heuristics and is limited to behavior-related side information.
3 Methodology
This methodology defines sequential recommendation with item- and behavior-related side information, reviews BERT-based modeling and invasive fusion, and introduces NOVA to use side information in attention while preserving item-ID representations.
- Problem Statement: Sequential recommendation predicts a user’s next interaction from chronological historical behavior, with side information categorized as item-related or behavior-related.Item-related information describes intrinsic item properties, while behavior-related information accompanies an interaction, including action type, execution time, feedback, or position.
- BERT and Invasive Self-attention: BERT4Rec represents items as embeddings, randomly masks items during training, and uses multi-head self-attention to recover item IDs.BERT’s attention uses query, key, and value components to generate contextual representations for sequence items.
- BERT and Invasive Self-attention: Conventional invasive fusion encodes side information separately and directly merges it with item-ID embeddings before feeding the mixed representation into BERT.The reviewed fusion operations include summation, concatenation, and gating, while the resulting integrated embeddings form the user-history input.
- Non-invasive Self-attention (NOVA): NOVA preserves a pure item-ID embedding branch and computes query and key from integrated representations while computing value from item-ID embeddings.This separates the representation used to form attention distributions from the value representation propagated through the model.
- Non-invasive Self-attention (NOVA): Unlike invasive fusion, NOVA treats side information as an auxiliary input fused into queries and keys, rather than altering the item representation itself.The comparison explicitly contrasts invasive fusion with NOVA’s query-and-key-only fusion of item-related and behavior-related side information.
4 Experiments
Experiments on public MovieLens and private APP datasets evaluate NOVA-BERT’s effectiveness, component contributions, robustness, and interpretability. NOVA-BERT consistently outperforms invasive fusion methods and BERT4Rec, while its gains vary with dataset scale and side-information type.
- Effectiveness: NOVA-BERT outperforms all other methods on three datasets and all evaluation metrics.The comparison includes BERT4Rec and invasive modifications using side information.
- Effectiveness: 13.51% HR@10 improvement over BERT4Rec occurs on ML-1m with NOVA gating, compared with 3.19% on ML-20M and 1.48% on APP.All invasive methods perform worse than the baseline on ML-1m, while noninvasive gains decrease on larger and denser datasets.
- Robustness: NOVA-BERT consistently outperforms baselines across fusion functions, although the best function depends on the dataset.Gating generally performs strongly, possibly because its gating mechanism is trainable; fusion choice can therefore be tuned as a hyper-parameter.
- Side-information contributions: Combining item-related and behavior-related side information yields greater improvement than either type alone on ML-1m.The evaluated item-related features are year and genre, while behavior-related information is rating; complete side information produces the most significant improvement.
- Interpretability: NOVA-BERT attention concentrates approximately along the diagonal, forming a stronger locality pattern than the BERT4Rec baseline.The authors report that side information produces more explicit attention in early layers and helps learn targeted attention distributions.
- Interpretability: The attention visualization cannot establish that NOVA-BERT’s pattern is absolutely better because deep neural networks have limited interpretability.The observed pattern is described as correlated with better performance, not as definitive proof of superiority.
5 Deployment and Cost of NOVA-BERT
NOVA-BERT surpasses deployed methods in online tests while adding almost no computational or model-size overhead. Its inference time is approximately that of original BERT.
- NOVA-BERT consistently surpasses current deployed methods in real-world online tests.
- NOVA-BERT has almost no extra computational overhead and the same model size as the invasive method.Table 5 evaluates the models using FLOPs and sizes, with addition as the fusion function.
- NOVA-BERT’s inference time is approximately the same as original BERT because it supports parallel computing and GPU acceleration.
6 Conclusion and Future Work
The paper presents NOVA-BERT and its non-invasive self-attention mechanism for using side information in sequential recommendation. It reports state-of-the-art performance with negligible computation and model-size overhead, while identifying fusion design as future work.
- NOVA-BERT uses side information as directional guidance while keeping item representations undoped in their vector space.
- The model achieves state-of-the-art performance on experimental datasets and industrial applications with negligible computation and model-size overhead.
- Future work will investigate whether fusing side information at every layer is optimal and whether stronger fusion functions can improve the approach.