Source-linked AI summary
Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction
Pi Qi, Xiaoqiang Zhu, Guorui Zhou, Yujing Zhang, Zhe Wang, Lejian Ren, Ying Fan, Kun Gai
TL;DR
Long sequential user behavior is valuable for CTR prediction, yet existing methods struggle to capture candidate-specific interests and meet industrial latency and storage constraints at lifelong sequence lengths. The paper proposes SIM, which cascades general and exact search units to retrieve relevant behaviors and model their precise relationship with a candidate item. SIM is deployed in Alibaba’s display advertising system, where it brings reported business gains and supports behavior sequences up to 54000.
Problem
Existing CTR models struggle to use lifelong sequential behavior because long sequences create latency, storage, computation, and candidate-specific interest-capture challenges.
Method
SIM cascades a General Search Unit that retrieves candidate-relevant behavior subsequences with an Exact Search Unit that models their precise relationship to the candidate item.
Results
SIM has been deployed in Alibaba’s display advertising system since 2019, bringing 7.1% CTR and 4.4% RPM lift and modeling behavior sequences up to 54000.
Takeaways & Limitations
SIM provides an industrially deployed approach for modeling lifelong user behavior while serving the main traffic in Alibaba’s display advertising system.
Takeaways & Limitations
The search unit shares the same formula and parameters among all users; future work proposes user-specific models for organizing each user’s lifelong behavior.
Abstract
from arXiv · showhide
Rich user behavior data has been proven to be of great value for click-through rate prediction tasks, especially in industrial applications such as recommender systems and online advertising. Both industry and academy have paid much attention to this topic and propose different approaches to modeling with long sequential user behavior data. Among them, memory network based model MIMN proposed by Alibaba, achieves SOTA with the co-design of both learning algorithm and serving system. MIMN is the first industrial solution that can model sequential user behavior data with length scaling up to 1000. However, MIMN fails to precisely capture user interests given a specific candidate item when the length of user behavior sequence increases further, say, by 10 times or more. This challenge exists widely in previously proposed approaches. In this paper, we tackle this problem by designing a new modeling paradigm, which we name as Search-based Interest Model (SIM). SIM extracts user interests with two cascaded search units: (i) General Search Unit acts as a general search from the raw and arbitrary long sequential behavior data, with query information from candidate item, and gets a Sub user Behavior Sequence which is relevant to candidate item; (ii) Exact Search Unit models the precise relationship between candidate item and SBS. This cascaded search paradigm enables SIM with a better ability to model lifelong sequential behavior data in both scalability and accuracy. Apart from the learning algorithm, we also introduce our hands-on experience on how to implement SIM in large scale industrial systems. Since 2019, SIM has been deployed in the display advertising system in Alibaba, bringing 7.1\% CTR and 4.4\% RPM lift, which is significant to the business. Serving the main traffic in our real system now, SIM models user behavior data with maximum length reaching up to 54000, pushing SOTA to 54x.
1 INTRODUCTION
Long sequential user behavior is valuable for CTR prediction but difficult to model at industrial scale because existing methods face computation, storage, and candidate-specific interest-capture limits. SIM addresses this with cascaded search over lifelong behavior data.
- 23% of Taobao users clicked more than 1000 products in the last 5 months, underscoring the scale of historical behavior data.
- Existing approaches generally model only hundreds of sequential behaviors, while MIMN extends this limit to 1000 through a memory matrix and serving-system co-design.MIMN decouples user modeling from CTR prediction, reducing online storage relative to raw behavior sequences.
- MIMN becomes unable to precisely capture candidate-specific interests when behavior sequences grow to 10000 or more.
- DIN searches user behaviors for candidate-specific interests, but its search formula incurs unacceptable computation and storage costs on long sequences.
- SIM uses a General Search Unit to retrieve a candidate-relevant Sub user Behavior Sequence from arbitrary-long raw behavior data before precise interest modeling.The paper presents this as a cascaded search paradigm for long sequential behavior data.
- SIM is presented as a two-stage paradigm intended to improve both scalability and accuracy for lifelong sequential behavior modeling.The paper also reports industrial implementation experience and deployment in Alibaba’s display advertising system.
2 RELATED WORK
User-interest modeling methods learn representations from historical behavior using architectures including CNNs, RNNs, Transformers, and Capsules. Long-term behavior adds useful information but creates latency, storage, and noise challenges, motivating specialized long-term modeling approaches.
- User Interest Model: User Interest Models learn latent user-interest representations from historical behaviors using CNN, RNN, Transformer, and Capsule architectures.
- Long-term User Interest: Long-term historical behavior can improve CTR prediction, but its growing length burdens online-serving latency and storage while introducing noise for point-wise prediction.
3 SEARCH-BASED INTEREST MODEL
SIM models very long user behavior sequences through two cascaded, candidate-aware search units: General Search Unit retrieves relevant behaviors, and Exact Search Unit models precise interest from them.
- Overall Workflow: SIM first uses the General Search Unit to retrieve top-K behaviors relevant to a candidate item from an arbitrarily long sequence with sub-linear time complexity.The retrieved sequence is shorter than the original behavior history, enabling efficient search under time and computation constraints.
- Overall Workflow: The Exact Search Unit applies an attention-based model to the filtered sub-sequence, allowing complex architectures such as DIN and DIEN after sequence reduction.It further captures precise user interest from the selected behaviors.
- General Search Unit: The General Search Unit supports hard-search, which selects behaviors from the same category as the candidate item for online serving.Hard-search is non-parametric and aggregates matching behaviors into the sub-sequence sent to the Exact Search Unit.
- General Search Unit: Soft-search embeds behaviors and uses sublinear-time maximum inner product search to reduce over ten thousand behaviors to hundreds of candidate-relevant behaviors.The soft-search parameters are trained with an auxiliary CTR task on long-term behavior data, while behavior and target representations are combined for the following MLP.
- Exact Search Unit: The Exact Search Unit incorporates temporal distances for selected behaviors and uses multi-head attention to represent diverse long-term user interest before CTR prediction.The resulting representation is fed into the MLP, and the General and Exact Search Units are trained simultaneously under cross-entropy loss.
4 IMPLEMENTATION FOR ONLINE SERVING
SIM’s online serving design addresses the storage, latency, and throughput demands of lifelong behavior modeling through indexed search and resource-aware deployment choices.
- Serving challenges: SIM’s serving system targets real-time CTR prediction under storage and latency pressure from long sequential user behavior data.The industrial system serves more than 1 million users per second at peak, making long-term model deployment challenging.
- Deployment choice: Hard search was selected for deployment because its top-K results closely resembled soft search while requiring a better balance between performance gain and resource consumption.Soft search performed slightly better offline, but the observed similarity of retrieved behaviors supported choosing hard search for production.
- Search-based serving: The system uses a hard-search module to retrieve effective behaviors associated with the target item from long sequential behavior data.Its user behavior tree organizes each user’s behaviors by category, enabling category-based retrieval for the target item.
- Search-based serving: The user behavior tree is a distributed two-level index using user IDs, category IDs, and behavior items, with size reaching 22 TB.This Key-Key-Value structure is designed to provide high-throughput queries.
- Latency optimization: The user behavior tree index can be built offline, keeping general-search response time short while allowing other user features to be computed in parallel.Offline construction saves most of the latency cost for online serving.
5 EXPERIMENTS
The experiments evaluate SIM on public and industrial datasets, compare it with established CTR models, and analyze its two-stage search, time embeddings, deployment trade-offs, and online impact.
- Datasets: Experiments use two public datasets and one industrial dataset, with dataset statistics reported in Table 1.The industrial dataset comes from Alibaba’s online display advertising system.
- Competitors and experiment setup: SIM is compared with DIN, average-pooling Long DIN, MIMN, and hard- or soft-search SIM variants using AUC.The setup follows related work and uses Adam, exponential learning-rate decay, fixed network dimensions, and embedding dimension 4.
- Results on Public Datasets: Long-term behavior models outperform DIN, while SIM outperforms other long-term interest models; adding time embeddings provides further improvement.The results support filtering relevant historical behaviors and modeling diverse interests conditioned on target items.
- Ablation Study: Filter strategies substantially outperform average pooling, and the second attention-based search stage further improves performance over one-stage search.Filtered sequences are usually much shorter than the originals, limiting the serving burden of attention operations.
- Ablation Study: Time embeddings improve results, indicating that user behaviors from different periods contribute differently.
- Results on Industrial Dataset: SIM improves MIMN by 0.008 AUC on the industrial dataset; hard-search is selected because it is more efficient and system friendly despite soft-search performing better.Hard-search behavior covers 75% of the behavior reserved by soft-search, and SIM serves behavior sequences up to 54000 with only a 5ms latency increase over MIMN for over ten thousand behaviors.
- Results on Industrial Dataset: Online A/B testing reports gains for SIM over MIMN in Alibaba’s display advertising system, where SIM serves the main scene traffic.The system deployment began in 2019.
- Results on Industrial Dataset: SIM’s clicks differ from DIEN mainly in the long-term portion, where SIM accounts for a larger proportion of clicks.The short-term distributions are nearly identical because both models use short-term behavior features from the last 14 days.
6 CONCLUSIONS
SIM exploits more than ten thousand sequential user behaviors in industrial CTR prediction by first reducing them to relevant behaviors and then modeling precise interest. It improves business performance while sharing search formulas and parameters across users.
- SIM targets lifelong sequential behavior data exceeding ten thousand behaviors in real industrial CTR prediction.
- Its two-stage design uses General Search Unit to reduce behaviors to hundreds, followed by Exact Search Unit to model precise interest from relevant behaviors.
- SIM is implemented in Alibaba’s display advertising system and serves the main traffic with significant business improvement.
- The search unit shares the same formula and parameters among all users, motivating future user-specific models for organizing evolving lifelong interests.