Source-linked AI summary
Behavior Sequence Transformer for E-commerce Recommendation in Alibaba
Qiwei Chen, Huan Zhao, Wei Li, Pipei Huang, Wenwu Ou
TL;DR
Industrial recommendation systems commonly use embedding-and-MLP models that overlook the order of users’ behavior sequences. This paper proposes BST, which adds Transformer self-attention to model sequential dependencies before MLP prediction. BST outperforms the WDL and DIN baselines offline, remains close in average response time, and is deployed for Taobao recommendation.
Problem
Embedding-and-MLP recommendation models commonly concatenate features while overlooking sequential signals in ordered user behavior sequences.
Method
BST adds Transformer self-attention to the embedding stage to learn sequentially informed representations of clicked items before MLP prediction.
Results
BST raises offline AUC from 0.7734 for WDL and 0.7866 for DIN to 0.7894, with average response time close to both baselines.
Takeaways & Limitations
BST was deployed in Taobao’s ranking stage and provides recommendation service for hundreds of millions of consumers every day.
Takeaways & Limitations
The model retains hand-crafted cross features because their usefulness had been demonstrated in the authors’ scenarios before deep learning.
Abstract
from arXiv · showhide
Deep learning based methods have been widely used in industrial recommendation systems (RSs). Previous works adopt an Embedding&MLP paradigm: raw features are embedded into low-dimensional vectors, which are then fed on to MLP for final recommendations. However, most of these works just concatenate different features, ignoring the sequential nature of users' behaviors. In this paper, we propose to use the powerful Transformer model to capture the sequential signals underlying users' behavior sequences for recommendation in Alibaba. Experimental results demonstrate the superiority of the proposed model, which is then deployed online at Taobao and obtain significant improvements in online Click-Through-Rate (CTR) comparing to two baselines.
1 INTRODUCTION
Alibaba’s Taobao ranking task must predict clicks among millions of candidate items, but standard embedding-and-MLP models do not adequately represent ordered user behavior. BST addresses this gap by applying Transformer self-attention to behavior sequences and is evaluated offline and online for deployment.
- Taobao’s rank stage predicts whether users will click candidate items among millions of items using historical behaviors and other features.
- The embedding-and-MLP paradigm embeds raw features into low-dimensional vectors and feeds them through fully connected layers for click prediction.Representative industrial systems include Google’s WDL and Alibaba’s DIN.
- WDL concatenates behavior-related features without modeling order, while user click sequences contain sequential signals relevant to future clicks.Examples include cellphone accessories after buying an iPhone and shoes after buying trousers.
- BST applies Transformer self-attention during embedding to learn item representations that incorporate dependencies among items in user behavior sequences.The resulting representations are then fed into MLPs to predict responses to candidate items.
- Offline experiments and online A/B tests report BST’s superiority over existing methods, and the model is deployed in Taobao’s ranking stage.The deployment provides recommendation services for hundreds of millions of consumers daily.
2 ARCHITECTURE
BST models CTR from a user’s behavior sequence and other features by embedding inputs, applying Transformer self-attention to sequence items, and predicting with an MLP.
- CTR formulation: BST formulates CTR prediction as estimating the probability that a user clicks target item v_t given sequence S(u) and other features.Other features include user profile, context, item, and cross features.
- Embedding Layer: The embedding layer maps input features to fixed-size low-dimensional vectors, representing sequence items with item_id, category_id, and positional features.The model uses these sparse item features because selecting all item features is too expensive for behavior-sequence representation.
- Embedding Layer: Positional embeddings encode each clicked item’s temporal distance from recommendation time as pos(v_i) = t(v_t) − t(v_i).The paper reports that this position representation outperforms the sine and cosine functions used in prior work in its scenarios.
- Transformer layer: The Transformer layer uses self-attention, multi-head attention, and point-wise feed-forward networks to learn deeper item representations from relations within behavior sequences.Self-attention receives item embeddings, projects them into query, key, and value matrices, and the model stacks attention blocks to represent complex sequence relations.
- Transformer layer: One Transformer block performs better than two or three blocks in the reported experiments, while larger depths were not evaluated for efficiency.The observed comparison is b = 1 versus b = 2, 3; the authors leave larger b for future work.
- MLP layers and Loss function: BST concatenates other-feature embeddings with the Transformer output for the target item, uses three fully connected layers, and applies sigmoid output with cross-entropy training.The sigmoid output represents the predicted probability of a sample being clicked.
3 EXPERIMENTS
Experiments evaluate BST against WDL, DIN, and WDL(+Seq) using offline AUC and online CTR, while also examining deployment efficiency and Transformer depth.
- 3.1 Settings: The Taobao dataset uses seven days for training and one day for testing, with offline AUC and online CTR and response time as evaluation measures.Baselines are WDL, DIN, and WDL(+Seq), which averages embeddings of previously clicked items.
- 3.2 Results Analysis: 0.7894 AUC: BST improves offline performance over WDL at 0.7734 and DIN at 0.7866.The comparison supports the effectiveness of Transformer-based sequential modeling over the listed baselines.
- 3.2 Results Analysis: BST’s average response time is close to WDL and DIN, supporting deployment of the Transformer model in large-scale recommendation systems.The result addresses efficiency alongside predictive performance.
- 3.2 Results Analysis: b = 1 achieves the best offline AUC, so BST is deployed with one self-attention block.The authors attribute this to simpler sequential dependencies in user behavior sequences than in machine translation sentences and do not try larger b for efficiency.
4 RELATED WORK
Earlier deep learning CTR methods emphasize feature combinations or neural-network architectures, while BST addresses the sequential nature of user behavior; DIN also uses attention for behavior sequences.
- 4 RELATED WORK: WDL, DeepFM, XDeepFM, and Deep and Cross networks focus on feature combinations or neural-network architectures for CTR prediction.These methods are presented as related deep learning approaches preceding BST.
- 4 RELATED WORK: DIN models users’ behavior sequences with an attention mechanism, addressing a limitation shared by earlier feature-focused methods.BST is situated among methods that incorporate sequential behavior information into recommendation.
5 CONCLUSION
The paper presents Transformer-based Taobao recommendation, reports its ability to model user behavior sequences, and describes production deployment serving hundreds of millions of users in China.
- 5 CONCLUSION: BST applies the Transformer to Taobao recommendation and uses its capability to capture sequential relations in user behavior sequences.The paper presents technical details and evaluates the approach through extensive experiments.
- 5 CONCLUSION: The proposed model is deployed in Taobao’s production environment, which provides recommendation service for hundreds of millions of users in China.The conclusion includes deployment details in addition to experimental findings.