PaperlayerView exact revision on arXiv

Source-linked AI summary

TWIN: TWo-stage Interest Network for Lifelong User Behavior Modeling in CTR Prediction at Kuaishou

Jianxin Chang, Chenbin Zhang, Zhiyi Fu, Xiaoxue Zang, Lin Guan, Jing Lu, Yiqun Hui, Dewei Leng, Yanan Niu, Yang Song, Kun Gai

arXiv:2302.02352v2cs.IR

TL;DR

Conventional lifelong behavior modeling has inconsistent relevance metrics across its two search stages. TWIN aligns the stages’ metrics and extends target attention to longer behavior sequences; it outperforms SIM Hard and SIM Soft across three Kuaishou business scenarios.

  • Problem

    Lifelong user behavior modeling is a key component of CTR prediction, and conventional two-stage methods have inconsistent relevance metrics across stages.

    Source: §1 · p. 2

  • Method

    TWIN uses a consistency-preserved general search unit with the same target-behavior relevance metric as exact-search target attention, extending attention through feature splitting and efficient infrastructure.

    Source: §1 · p. 2, §3.3.1 · p. 5, §5 · p. 9

  • Results

    TWIN remarkably outperforms SIM Hard and SIM Soft in all three representative scenarios; Kuaishou considers a 0.1% Watch Time increase an effective improvement.

    Source: §4.8 · p. 9

  • Takeaways & Limitations

    TWIN’s computational bottleneck was accelerated by 99.3%, enabling deployment serving the main traffic of hundreds of millions of active users on Kuaishou.

    Source: §5 · p. 9

    The attention formulation assumes there has been no interaction between the user and the target item.

    Source: §3.3.3 · p. 5

Abstract

from arXiv · show

Life-long user behavior modeling, i.e., extracting a user's hidden interests from rich historical behaviors in months or even years, plays a central role in modern CTR prediction systems. Conventional algorithms mostly follow two cascading stages: a simple General Search Unit (GSU) for fast and coarse search over tens of thousands of long-term behaviors and an Exact Search Unit (ESU) for effective Target Attention (TA) over the small number of finalists from GSU. Although efficient, existing algorithms mostly suffer from a crucial limitation: the \textit{inconsistent} target-behavior relevance metrics between GSU and ESU. As a result, their GSU usually misses highly relevant behaviors but retrieves ones considered irrelevant by ESU. In such case, the TA in ESU, no matter how attention is allocated, mostly deviates from the real user interests and thus degrades the overall CTR prediction accuracy. To address such inconsistency, we propose \textbf{TWo-stage Interest Network (TWIN)}, where our Consistency-Preserved GSU (CP-GSU) adopts the identical target-behavior relevance metric as the TA in ESU, making the two stages twins. Specifically, to break TA's computational bottleneck and extend it from ESU to GSU, or namely from behavior length $10^2$ to length $10^4-10^5$, we build a novel attention mechanism by behavior feature splitting. For the video inherent features of a behavior, we calculate their linear projection by efficient pre-computing \& caching strategies. And for the user-item cross features, we compress each into a one-dimentional bias term in the attention score calculation to save the computational cost. The consistency between two stages, together with the effective TA-based relevance metric in CP-GSU, contributes to significant performance gain in CTR prediction.

1 INTRODUCTION

Long-term behavior modeling supports personalized CTR prediction, but conventional two-stage systems can use inconsistent relevance metrics in GSU and ESU. TWIN aligns those stages with target attention and extends it to long behavior sequences through feature splitting and efficient projection strategies.

1 INTRODUCTION

  • The GSU’s coarse relevance metric can miss relevant behaviors and retrieve behaviors that ESU considers irrelevant, undermining ESU’s target attention.TA is effective but computationally expensive, so the conventional pipeline uses GSU to pre-filter long sequences before ESU.Source: §1 · p. 2
  • TWIN’s CP-GSU uses the same target-behavior relevance metric as ESU’s target attention, making the two stages consistent.TWIN extends TA from ESU to CP-GSU by splitting behavior features, precomputing and caching inherent features, and compressing user-item cross features into bias terms.Source: §1 · p. 2
  • TWIN’s consistency between stages and TA-based relevance metric in CP-GSU contribute to significant CTR prediction gains.The paper also reports that TWIN outperforms SIM Hard and SIM Soft in online tests across three representative business scenarios.Source: §1 · p. 2
  • 99.3%: TWIN reduces the linear-projection computational bottleneck; the paper reports offline experiments on Kuaishou’s 46 billion-scale dataset and online A/B tests.TWIN is deployed on Kuaishou’s recommendation system, serving the main traffic of 346 million active users every day.Source: §1 · p. 2

2 RELATED WORK

Prior work develops CTR models and long-term behavior methods, including two-stage retrieval followed by target attention. TWIN extends target attention to the retrieval stage and synchronizes its embeddings and attention parameters across stages.

2.1 Click-Through-Rate Prediction

  • CTR models progress from feature-interaction methods to deep architectures, while user behavior models use target attention, temporal modeling, sessions, multiple interest vectors, and transformers.The cited examples include FM and FFM for feature interactions, and DIN, DIEN, DSIN, MIND, DMIN, BST, SASRec, and BERT4Rec for behavior modeling.Source: §2.1 · p. 3

2.2 Long-Term User Behavior Modeling

  • Long-term behavior methods move from offline summaries and memory matrices toward two-stage retrieval, but existing GSU strategies can remain computationally burdensome or inconsistent with ESU.SIM variants use category matching or embedding inner products; ETA uses hashing and Hamming distance, while SDIM uses hash collisions.Source: §2.2 · p. 3
  • TWIN extends the target-attention structure to GSU and synchronizes embeddings and attention parameters from ESU, maintaining end-to-end training.The paper distinguishes this from indexing methods that approximate relevance scores with codebooks; TWIN uses exact distance calculation after GSU pre-filtering.Source: §2.2 · pp. 3–4

3 TWIN IN KUAISHOU CTR PREDICTION

This section outlines the CTR system, TWIN’s behavior-modeling module, and the infrastructure strategies used for online deployment.

3 TWIN IN KUAISHOU CTR PREDICTION

  • The section first reviews CTR preliminaries and then describes Kuaishou’s CTR system architecture.The notations used are summarized in Table 2.Source: §3 · p. 4
  • It next details TWIN, the consistency-preserved lifelong user behavior modeling module.Source: §3 · p. 4
  • The section concludes with accelerating strategies intended to support deployment on Kuaishou’s main traffic.Source: §3 · p. 4

3.1 Preliminaries

The preliminaries formulate CTR prediction as binary classification from user, item, and context features. The predicted CTR is the sigmoid-transformed predictor output, and training minimizes negative log-likelihood.

3.1 Preliminaries

  • CTR prediction is formulated as binary classification using a predictor from feature vectors that concatenate user, item, and context features.The label indicates whether the user clicks the item.Source: §3.1 · p. 4
  • The predicted CTR is obtained by applying the sigmoid function to the predictor output.The sigmoid scales the predictor’s output to the interval from 0 to 1.Source: §3.1 · p. 4
  • The model is trained by minimizing negative log-likelihood.Source: §3.1 · p. 4

3.2 The Architecture of the CTR Prediction

Kuaishou’s CTR architecture embeds categorical inputs and combines TWIN’s lifelong behavior modeling with short-term behavior modeling and other task-specific signals. TWIN uses two cascading stages, with CP-GSU and ESU applying the same relevance metric.

3.2.1 The Embedding Layer.

  • Categorical inputs are encoded as one-hot or multi-hot vectors and transformed into low-dimensional embeddings.Id features use dimension 64; other features, such as video topic and played timestamp, use dimension 8.Source: §3.2.1 · pp. 4–5

3.2.2 The Deep Networks.

  • TWIN first retrieves 100 relevant long-term behaviors with CP-GSU, then applies attention over those finalists in ESU using the same relevance metric.The upper network mixes TWIN’s output with short-term behavior modeling and other task modelings.Source: §3.2.2 · p. 5
  • The architecture includes a short-term behavior module that extracts interests from the 50 most recent behaviors, complementing TWIN’s long-term modeling.Other task modelings also represent user, video, and context features.Source: §3.2.2 · p. 5

3.3 TWIN: TWo-stage Interest Network

TWIN extends target attention to long behavior sequences while preserving the same target-behavior relevance metric in CP-GSU and ESU. It splits behavior features to reduce the cost of the attention computation.

3.3 TWIN: TWo-stage Interest Network

  • TWIN uses the same target-behavior relevance metric in CP-GSU and ESU, extending target attention from hundreds of behaviors to at least tens of thousands.The paper identifies linear projection of the behavior matrix as the main computational bottleneck.Source: §3.3 · p. 5, §3.3.1 · p. 5

3.3.1 Behavior Feature Splits and Linear Projection.

  • Behavior features split into user-independent inherent features and user-item cross features, enabling separate linear projections.Inherent features include video id, author, topic, and duration; cross features include click timestamp, play time, and page position.Source: §3.3.1 · p. 5
  • Each cross feature is compressed to one dimension through a simplified projection equivalent to restricting its weights to a diagonal block matrix.The paper notes that inherent-feature projections can be gathered through caching, while cross-feature projections cannot be cached because they depend on user-video interactions.Source: §3.3.1 · p. 5

3.3.2 Complexity Analysis.

  • Because C is much smaller than H and d_out, the proposed computation enables consistent MHTA in CP-GSU and ESU.The inherent-feature projection is precomputed and gathered in O(L), independent of H, while cross-feature projection costs O(L × C).Source: §3.3.2 · p. 5

3.3.3 Target Attention in TWIN.

  • The deployment precomputes and caches inherent-feature projections for an 8-billion-video candidate pool, refreshing them every 15 minutes to minimize caching-related accuracy loss.The embedding server’s cached projections cover 97% of online requests.Source: Figure 3 · p. 6
  • TWIN scores target-behavior relevance using the target’s projected inherent features against behavior keys, with compressed cross features acting as learnable-weighted bias terms.CP-GSU selects the 100 highest-scoring behaviors from 10^4 behaviors; ESU then performs weighted average pooling over those finalists.Source: §3.3.3 · pp. 5–6

3.4 System Deployment

TWIN is deployed in Kuaishou’s ranking system through nearline training, offline precomputation of inherent-feature projections, and online relevance scoring. The system serves the platform’s main traffic.

3.4.1 Training System.

  • Nearline training processes daily watch and interaction logs and updates model parameters using interactions from less than eight minutes earlier.The message queue synchronizes the latest parameters to offline inference and online serving every five minutes.Source: §3.4.1 · p. 6

3.4.2 Offline Inferring.

  • An offline projector refreshes inherent-feature projections for an 8-billion-video candidate pool every 15 minutes, and the embedding server provides lookup access covering 97% of online requests.The pool is limited by cutting off tailed videos.Source: §3.4.2 · p. 6

3.4.3 Online Serving.

  • Online serving retrieves cached inherent-feature projections, computes the remaining relevance-score components, and sends the 100 highest-scoring behaviors to ESU.This reduces the linear-projection bottleneck for 10^4 behaviors by 99.3% in practice; ESU calculations on 100 behaviors run online with up-to-date parameters.Source: §3.4.3 · p. 6
  • TWIN serves Kuaishou’s main ranking traffic for 346 million active users, with a peak request rate of 30 million videos per second.Source: §3.4.3 · p. 6

4 EXPERIMENT

The experiments evaluate TWIN on Kuaishou industrial data and compare it with lifelong behavior-modeling baselines. The dataset is intended to provide long user behavior histories at industrial scale.

4 EXPERIMENT

  • The evaluation plan includes offline comparison with state-of-the-art methods and asks how TWIN performs in real online recommendation systems.The experiment section also lists consistency, sequence-length effects, and component or implementation comparisons as research questions.Source: §4 · pp. 6–7

4.1 Dataset

  • The Kuaishou dataset averages 14,500 watched videos per user over six months and caps behavior sequences at 100,000.It is constructed from daily user logs labeled by clickthroughs; the platform records 46 billion video plays per day.Source: §4.1 · p. 7

4.2 Baselines

  • The experiments compare TWIN with pooling, DIN, SIM Hard, ETA, SDIM, SIM Cluster variants, and SIM Soft.The baselines use differing retrieval strategies, including category or cluster matching, hashing, and embedding inner products.Source: §4.2 · p. 7

4.3 Experimental Setting

Offline comparisons favor TWIN, while the consistency analysis shows that its GSU retrieves far more behaviors that ESU considers important than SIM Hard does.

4.3 Experimental Setting

  • The offline evaluation averages results over five days, using 23 hours for training and the following hour for testing; two-stage models use 10,000 GSU inputs and retrieve 100 behaviors.Source: §4.3 · p. 7

4.4 Overall Performance (RQ1)

  • TWIN improves over the best compared model by 0.29% in AUC and 0.51% in GAUC, exceeding the 0.05% threshold noted as sufficient for online gains.Source: §4.4 · p. 7, Table 4 · p. 8
  • ETA and SDIM also use end-to-end training, but their GSU retrieval strategies remain less precise and inconsistent with ESU’s relevance metric.Source: §4.4 · p. 8
  • Performance improves as GSU retrieval granularity becomes finer, from category-based SIM Hard through clustered variants to per-behavior SIM Soft.Source: §4.4 · p. 8

4.5 Consistency Analysis (RQ2)

  • TWIN returns 94 of the 100 behaviors in ESU’s real top-100, versus 40 for SIM Hard; a 15-minute caching refresh delay limits practical consistency.Source: Figure 4 · p. 8, §4.5 · p. 8

4.6 Effects of Behavior Length (RQ3)

TWIN remains strongest as behavior sequences grow, and the ablations indicate benefits from consistency and from retaining user-item cross features while compressing attention computation.

4.6 Effects of Behavior Length (RQ3)

  • The length experiment changes GSU input sequence length while keeping its output at 100 behaviors.Source: §4.6 · p. 8
  • TWIN performs best across sequence lengths, and its performance gap over other methods grows as the sequence length increases.Source: §4.6 · p. 8, Figure 5 · p. 9

4.7 Ablation Study (RQ4)

  • TWIN w/o Para-Con beats SIM Soft but trails TWIN, indicating that both structure and parameter consistency help, with structure consistency contributing more.Source: §4.7 · p. 9, Figure 6 · p. 9
  • TWIN performs almost the same as TWIN w/ Raw MHTA, while removing user-item cross-feature biases significantly reduces GAUC and saves little computation.Source: §4.7 · p. 9, Figure 6 · p. 9

4.8 Online Result (RQ5)

Online A/B tests on Kuaishou’s short-video platform show that TWIN outperforms SIM Hard and SIM Soft across three representative business scenarios, with significant Watch Time gains.

4.8 Online Result (RQ5)

  • TWIN remarkably outperforms SIM Hard and SIM Soft in all three scenarios, and its Watch Time gains meet Kuaishou’s threshold for significant business improvement.Source: §4.8 · p. 9

5 CONCLUSION

TWIN addresses inconsistent retrieval across two-stage lifelong behavior modeling by extending MHTA from ESU to CP-GSU over sequences of 10^4–10^5 behaviors.

5 CONCLUSION

  • TWIN extends MHTA from ESU’s sequence length of 100 to CP-GSU lengths of 10^4–10^5, using efficient mechanisms that reduce the computational bottleneck by 99.3%.Source: §5 · p. 9
Loading 2302.02352v2…