Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction

Qi Pi; Weijie Bian; Guorui Zhou; Xiaoqiang Zhu; Kun Gai

Source-linked AI summary

Practice on Long Sequential User Behavior Modeling for Click-Through Rate Prediction

Qi Pi, Weijie Bian, Guorui Zhou, Xiaoqiang Zhu, Kun Gai

arXiv:1905.09248v3May 22, 2019cs.IR

TL;DR

Long sequential user behavior improves CTR interest modeling but makes real-time serving increasingly costly in latency and storage. The paper co-designs a UIC serving module with the incremental MIMN memory model, achieving long-sequence handling and gains over the prior product model in online testing.

Problem
Industrial CTR systems need to exploit long user behavior sequences, but complex models face approximately linear latency and storage growth as sequence length increases.
Method
The paper co-designs a decoupled UIC server for event-driven interest updates with an incremental MIMN memory model for long sequential behavior.
Results
7.5% CTR and 6% RPM gains over DIEN were observed in online A/B testing after deployment in Alibaba’s display advertising system.
Takeaways & Limitations
The UIC–MIMN solution handles sequential behavior lengths scaling to thousands while improving model performance and system efficiency.
Takeaways & Limitations
The online system assumes real-time behavior-event volume cannot significantly exceed real-time CTR prediction request volume.

Abstract

from arXiv · show

Click-through rate (CTR) prediction is critical for industrial applications such as recommender system and online advertising. Practically, it plays an important role for CTR modeling in these applications by mining user interest from rich historical behavior data. Driven by the development of deep learning, deep CTR models with ingeniously designed architecture for user interest modeling have been proposed, bringing remarkable improvement of model performance over offline metric.However, great efforts are needed to deploy these complex models to online serving system for realtime inference, facing massive traffic request. Things turn to be more difficult when it comes to long sequential user behavior data, as the system latency and storage cost increase approximately linearly with the length of user behavior sequence. In this paper, we face directly the challenge of long sequential user behavior modeling and introduce our hands-on practice with the co-design of machine learning algorithm and online serving system for CTR prediction task. Theoretically, the co-design solution of UIC and MIMN enables us to handle the user interest modeling with unlimited length of sequential behavior data. Comparison between model performance and system efficiency proves the effectiveness of proposed solution. To our knowledge, this is one of the first industrial solutions that are capable of handling long sequential user behavior data with length scaling up to thousands. It now has been deployed in the display advertising system in Alibaba.

1 INTRODUCTION

Long sequential behavior data can improve user-interest modeling but creates major latency and storage challenges in industrial CTR systems. The paper addresses these challenges through a co-designed serving and learning solution combining UIC and MIMN.

Motivation: Industrial CTR models use historical user behavior to model interests, with pooling-based and sequential-modeling architectures representing two main approaches.Pooling methods summarize behaviors as independent signals, while sequential methods model them as ordered signals using LSTM or GRU operations.
Challenge: Sequences of 1000 or more behaviors make online deployment difficult because latency and storage costs increase approximately linearly with sequence length.Earlier sequential deployment work handled a maximum sequence length of 50.
Solution: The paper co-designs a machine-learning algorithm and online serving system by separating user-interest modeling from the overall CTR prediction process.The proposed design introduces UIC for serving and MIMN for incremental long-sequence modeling.
Solution: UIC maintains each user’s latest interest representation and updates it from real-time behavior events rather than traffic requests, making interest computation latency free for CTR prediction.MIMN is designed to work incrementally with UIC.
Outcome: Combining UIC and MIMN theoretically supports user-interest modeling with unlimited sequential behavior length, with experiments showing superior model performance and system efficiency.The solution is described as one of the first industrial systems handling sequences scaling to thousands.
Outcome: The work reports a deployed industrial practice that co-designs learning and serving for long sequential behavior modeling in CTR prediction.The contribution includes experiments on public and Alibaba industrial datasets and discussion of deployment issues.

2 RELATED WORK

Prior CTR and user-interest research uses deep models, feature engineering, and memory networks, but the cited approaches leave gaps in adaptive long-term interest modeling and practical deployment.

Deep CTR Models: Deep CTR methods replace traditional feature engineering with neural networks that capture feature interactions.The related work describes deep-learning-based CTR prediction as a major development in the area.
Long-term User Interest: Long-term user-interest methods model general, categorical, or profile-based interests, but the cited approaches rely on feature engineering rather than adaptive end-to-end learning.TDSSM is cited as jointly modeling long-term and short-term user interests.
Memory Networks: Memory networks have been applied to user-interest modeling, but the cited methods neglect long-term interest modeling and practical deployment issues.The paper positions its memory-based approach against these limitations.

3 REALTIME CTR PREDICTION SYSTEM

The industrial CTR system must process behavior-heavy features under strict latency and throughput constraints. The proposed UIC server decouples interest updates from requests, while MIMN stores and updates user interests incrementally for long sequences.

RTP System: Real-time CTR prediction receives candidates and returns probability scores under a typical 10-millisecond latency limit.The system includes feature management, model management, and prediction-server components.
Behavior Features: User behavior features constitute nearly 90% of the system’s features, making them a major source of information and system volume.The remaining features are user demography and ad features.
Serving Challenges: Serving long behavior sequences requires storing and fetching large distributed in-memory feature sets, creating latency and throughput pressure.DIEN reached the RTP system’s performance edge at sequence length 150, making length 1000 difficult to serve directly.
Serving Challenges: At sequence length 1000, storing behavior data for more than 600 million users would consume 6 TB, with storage increasing linearly with sequence length.At length 150, the system uses about 1 TB and stores product and related feature identifiers.
MIMN Architecture: MIMN uses a memory-based architecture whose external memory is stored and incrementally updated by UIC as new user behaviors arrive.Its architecture combines a sequential-interest sub-network with a traditional Embedding&MLP sub-network.
User Interest Center: UIC maintains each user’s latest interest representation and updates it on real-time behavior events rather than requests, removing interest computation from request latency.UIC reduced DIEN latency for length 1000 from 200 ms to 19 ms at 500 QPS.

4 MULTI-CHANNEL USER INTEREST MEMORY NETWORK

The paper addresses the difficulty of learning from long sequential behavior data by combining memory-based user-interest modeling with an online serving design. MIMN and UIC incrementally maintain user interests, improve memory utilization, and support sequences scaling to thousands.

Challenges of Learning From Long Sequential User Behavior Data: Long sequential modeling is difficult because recurrent models can fail on long sequences, while attention requires storing raw behaviors and has computation cost that grows linearly with sequence length.These constraints create storage and serving pressure for real-time inference.
Multi-Channel User Interest Memory Network: MIMN uses a fixed-size external memory and UIC incrementally updates that memory for each new user behavior.The design captures evolving user interests without storing the entire behavior sequence.
Neural Turing Machine: NTM performs memory read and write operations through a controller, using behavior embeddings to address memory and add or erase information.Read operations produce a weighted memory summary, while write operations update memory using addressing, add, and erase vectors.
Memory Utilization Regularization: Memory utilization regularization rebalances write weights across slots, reducing update-weight variance and helping the memory tensor store more source behavior information.The transfer matrix depends on accumulated slot utilization and learned parameters.
Memory Induction Unit: MIMN adds a Memory Induction Unit that updates selected user-interest channels with GRU-based states using both NTM memory and behavior embeddings.The multi-channel GRU parameters are shared without increasing parameter volume.
Implementation for Online Serving: The UIC server hosts the heavy NTM and MIU user-interest sub-network, while incremental memory states are updated for real-time CTR prediction without storing user behavior data.In the reported system, longtime behavior storage was reduced from 6T to 2.7T.

5 EXPERIMENTS

Experiments evaluate MIMN on public and industrial datasets, compare it with CTR baselines, and examine its components and serving behavior. Results report gains in AUC, online business metrics, and support for much longer sequences with the UIC co-design.

Datasets and Experimental Setup: Experiments compare MIMN with established CTR models on two public datasets and an industrial Alibaba advertising dataset.The compared models include pooling-, attention-, recurrent-, and memory-based architectures.
Results on Public Datasets: MIMN beats all compared models with a significant gain over AUC on the public-dataset evaluation.The authors attribute this result to memory-based modeling of diverse and evolving interests in long behavior sequences.
Ablation Study: The best memory-slot setting is 4 slots for Amazon and 8 slots for Taobao, with performance depending on behavior-sequence length.Larger memory capacity can hurt shorter-sequence datasets when some randomly initialized slots remain underused.
Ablation Study: Memory Utilization Regularization balances storage use and improves model performance, while the Memory Induction Unit captures high-order interest information.The induction unit derives sequential relationships among interests from the basic NTM memory.
Results on Industrial Dataset: MIMN improves DIEN by 0.01 AUC on Alibaba’s industrial dataset and, with UIC, maintains constant latency and throughput while handling sequences scaling to thousands.DIEN’s deployed product model exploited only 50 behavior events because of latency and throughput pressure.
Results on Industrial Dataset: Online A/B testing reports 7.5% CTR and 6% RPM gains for MIMN over DIEN in Alibaba’s display advertising system.The test ran from 2019-03-30 to 2019-05-10 after deployment of the proposed solution.

6 CONCLUSIONS

The paper co-designs a decoupled UIC serving system with a memory-based MIMN model to exploit long sequential behavior data for CTR prediction. This solution targets storage bottlenecks, supports incremental implementation, and is reported to outperform other state-of-the-art models.

The proposed solution co-designs a decoupled UIC server for real-time user-interest inference with a memory-based MIMN model.UIC addresses online serving, while MIMN can be implemented incrementally.
Storage is identified as the main computational bottleneck when capturing long-term interests from very long behavior sequences.
The memory-based MIMN model is reported to outperform other state-of-the-art models.
The work proposes modeling with extremely long sequential user behavior data and identifies this direction as a new research space.Future work includes learning algorithms, training systems, and online serving systems.