PaperlayerView exact revision on arXiv

Source-linked AI summary

TWIN V2: Scaling Ultra-Long User Behavior Sequence Modeling for Enhanced CTR Prediction at Kuaishou

Zihua Si, Lin Guan, ZhongXiang Sun, Xiaoxue Zang, Jing Lu, Yiqun Hui, Xingchao Cao, Zeyu Yang, Yichen Zheng, Dewei Leng, Kai Zheng, Chenbin Zhang, Yanan Niu, Yang Song, Kun Gai

arXiv:2407.16357v2cs.IRcs.AI

TL;DR

CTR systems lack an effective way to model behavior histories spanning a user’s full life cycle. TWIN-V2 combines offline clustering with cluster-aware target attention, and the paper reports improvements over TWIN alongside large-scale deployment at Kuaishou.

  • Problem

    Existing methods do not fully model life-cycle behavior histories, which can reach 1 million behaviors in an app.

    Source: §1 · p. 2

  • Method

    TWIN-V2 compresses life-cycle histories by clustering similar items offline and uses cluster-aware target attention to model interests from clustered behaviors online.

    Source: §1 · p. 2, §3.2 · p. 3

  • Results

    TWIN-V2 improves over the second-best TWIN model by 0.0013 in AUC and 0.0024 in GAUC.

    Source: §4.2 · pp. 6–7

  • Takeaways & Limitations

    TWIN-V2 reduces life-cycle behavior storage and computation overhead by 90% and serves Kuaishou’s main traffic of around 400 million daily active users.

    Source: §5 · p. 7

Abstract

from arXiv · show

The significance of modeling long-term user interests for CTR prediction tasks in large-scale recommendation systems is progressively gaining attention among researchers and practitioners. Existing work, such as SIM and TWIN, typically employs a two-stage approach to model long-term user behavior sequences for efficiency concerns. The first stage rapidly retrieves a subset of sequences related to the target item from a long sequence using a search-based mechanism namely the General Search Unit (GSU), while the second stage calculates the interest scores using the Exact Search Unit (ESU) on the retrieved results. Given the extensive length of user behavior sequences spanning the entire life cycle, potentially reaching up to 10^6 in scale, there is currently no effective solution for fully modeling such expansive user interests. To overcome this issue, we introduced TWIN-V2, an enhancement of TWIN, where a divide-and-conquer approach is applied to compress life-cycle behaviors and uncover more accurate and diverse user interests. Specifically, a hierarchical clustering method groups items with similar characteristics in life-cycle behaviors into a single cluster during the offline phase. By limiting the size of clusters, we can compress behavior sequences well beyond the magnitude of 10^5 to a length manageable for online inference in GSU retrieval. Cluster-aware target attention extracts comprehensive and multi-faceted long-term interests of users, thereby making the final recommendation results more accurate and diverse. Extensive offline experiments on a multi-billion-scale industrial dataset and online A/B tests have demonstrated the effectiveness of TWIN-V2. Under an efficient deployment framework, TWIN-V2 has been successfully deployed to the primary traffic that serves hundreds of millions of daily active users at Kuaishou.

1 INTRODUCTION

Existing methods do not cover users’ full life-cycle behavior histories, which can reach 1 million behaviors. TWIN-V2 addresses this gap with offline clustering and cluster-aware attention, and the paper reports improved recommendations and deployment at Kuaishou.

1 INTRODUCTION

  • Existing methods had no solution for modeling a user’s entire life cycle, which can encompass up to 1 million in-app behaviors.Source: §1 · p. 2
  • The two-stage approach uses GSU to retrieve target-related behaviors and ESU to extract long-term interests through target attention.Source: §1 · p. 2
  • TWIN’s 10,000-behavior GSU input covers only the last 3–4 months of a user’s Kuaishou history, not the full life cycle.Source: §1 · p. 2
  • TWIN-V2 uses offline hierarchical clustering to group similar items into virtual items, then applies cluster-aware target attention online with scores reweighted by cluster size.Source: §1 · p. 2
  • The paper reports that TWIN-V2 improves efficiency and interest modeling, with offline and online validation and deployment serving around 400 million daily active users.Source: §1 · p. 2

2 RELATED WORK

Prior CTR work spans target-attention and sequence methods, while two-stage systems retrieve target-related history before modeling longer-term interests. TWIN-V2 aims to extend history modeling to the life-cycle level while retaining efficiency.

2 RELATED WORK

  • CTR behavior modeling includes sequence pooling, target attention, temporal and multi-head attention, and memory-network approaches.Source: §2 · p. 2
  • TWIN-V2 can process histories up to 10^6 behaviors by adjusting clustering hyperparameters, which the paper says covers Kuaishou users’ full life cycle.Source: §2 · p. 3
  • Two-stage methods retrieve target-related history before modeling interests; ETA and SDIM use hashing, while TWIN uses the same target-behavior relevance metric in both stages.Source: §2 · p. 3
  • The paper’s goal is life-cycle-level history modeling with efficiency, and Table 1 compares TWIN-V2 with prior methods, including two-stage models and their maximum sequence lengths.Source: §2 · p. 3, Table 1 · p. 3

3 METHOD

TWIN-V2 compresses life-cycle behavior into clusters offline, then uses a two-stage online process to retrieve target-related clusters and extract long-term interests. Its cluster-aware attention adjusts cluster relevance and aggregates selected clusters into user-interest representations.

3.2 Overall Workflow

  • TWIN-V2 first compresses life-cycle behavior into clusters, then retrieves target-related clusters with GSU and extracts long-term interests with ESU.The CTR model forecasts click probability from sample features.Source: §3.2 · p. 3, §3.1 · p. 3

3.3.1 Hierarchical Clustering Over Life Cycle.

  • Hierarchical clustering groups items by play-time proportion, then recursively clusters behavior embeddings until each cluster contains at most γ items.Play-time grouping aims to keep completion ratios relatively consistent within clusters; adaptive clustering accommodates personalized history sizes, and embeddings come from the recommendation model.Source: §3.3.1 · p. 4

3.3.2 Extracting Cluster Representation.

  • Each cluster becomes a virtual item: numerical features are averaged, while the item closest to the cluster centroid represents categorical features.The resulting cluster representation is embedded for GSU and ESU relevance estimation.Source: §3.3.2 · pp. 4–5

3.3.3 Cluster-aware Target Attention.

  • Cluster-aware target attention adjusts relevance scores by cluster size, selects the top 100 clusters in GSU, and aggregates them in ESU using four attention heads.The authors state that 100 clusters cover more behaviors than 100 retrieved individual behaviors, broadening the interests ESU can model.Source: §3.3.3 · p. 5

3.4 Deployment of TWIN-V2

  • Offline processing periodically clusters life-cycle behavior, while online serving sends processed features through GSU and ESU to the CTR model.The clustering runner updates every two weeks; deployment compresses history to 10% of its original length, reducing storage costs by 90%. TWIN handles recent months while TWIN-V2 covers the full life cycle.Source: §3.4 · pp. 5–6

4 EXPERIMENT

The section evaluates TWIN-V2 through offline and online experiments using Kuaishou user histories, then compares it with CTR modeling baselines under shared evaluation and implementation settings.

4.1.1 Dataset.

  • User interactions from five consecutive Kuaishou days provide training and testing data, with each user's history retraced up to 100,000 behaviors.The first 23 hours of each day were used for training and the final hour for testing.Source: §4.1.1 · p. 6

4.1.2 Baselines.

  • Comparisons include Avg-Pooling, DIN, SIM Hard and Soft, ETA, SDIM, SIM Cluster variants, and TWIN, with other model factors held consistent.Table 3 reports overall comparisons; a 0.001-level improvement is considered significant for CTR prediction [11, 21].Source: §4.1.2 · p. 6, Table 3 · p. 6

4.1.3 Evaluation Metrics & Protocal.

  • Models are evaluated using AUC and GAUC, averaged across five consecutive daily test sets with means and standard deviations reported.Each day's first 23 hours are training data and its final hour is test data.Source: §4.1.3 · p. 6

4.1.4 Implementation Details.

  • TWIN-V2 clusters user histories up to 100,000 items, compressing them to about 10% and an approximately 10,000-item GSU input; two-stage models also use 10,000-item GSU inputs.DIN and Avg-Pooling use recent histories capped at 100; two-stage models retrieve 100 items for ESU. The batch size is 8192 and Adam's learning rate is 5e-6.Source: §4.1.4 · p. 6

4.2 Overall Performance

TWIN-V2 outperforms the reported baselines, and ablations support its adaptive clustering and cluster-size reweighting choices. Online tests also report gains in watch time and recommendation diversity versus TWIN.

4.2 Overall Performance

  • TWIN-V2 significantly outperforms other baselines; over second-best TWIN it gains 0.0013 in AUC and 0.0024 in GAUC.The authors report a larger relative improvement in GAUC than AUC and say gains are greater among highly active users, with that hypothesis validated in Appendix A.3.Source: §4.2 · pp. 6–7

4.3.1 Comparison of Different Hierarchical Clustering Methods.

  • Adaptive δ clustering achieves the highest cluster accuracy and shortest running time versus fixed-δ Binary and Balanced&Binary variants.Balanced&Binary moves items from the larger cluster toward the other centroid after each k-means iteration to equalize cluster sizes.Source: §4.3.1 · p. 7

4.3.2 Effectiveness of Cluster-aware Target Attention.

  • Online A/B tests report improved watch time and diversity of recommended results compared with TWIN.Source: Table 5 · p. 7
  • Removing cluster-size reweighting from GSU and ESU reduces performance, supporting adjustment of attention scores by cluster size.Figure 4's right side presents the reweighting ablation.Source: §4.3.2 · p. 7, Figure 4 · p. 7

4.4 Online Experiments

Online A/B tests in three Kuaishou scenarios evaluate TWIN-V2 on watch time and recommendation diversity. The reported results show improved watch time and more varied recommendations.

4.4 Online Experiments

  • TWIN-V2 improved watch time and recommendation diversity across three representative Kuaishou scenarios.The scenarios are Featured-Video Tab, Discovery Tab, and Slide Tab.Source: §4.4 · p. 7
  • The A/B test measured watch time as total user time spent and diversity as variety in recommended outcomes.The examples of diversity include richness in video types.Source: §4.4 · p. 7
  • The paper attributes more varied recommendations to longer behavior histories revealing a wider range of user interests.Source: §4.4 · p. 7

5 CONCLUSION

TWIN-V2 extends user-history modeling to lifecycle scale and compresses ultra-long behaviors. The paper reports lower storage and computation overhead, improved recommendation results, and deployment on Kuaishou’s main traffic.

5 CONCLUSION

  • TWIN-V2 accommodates up to 10^6 user behaviors in Kuaishou, extending history modeling to the full lifecycle.Source: §5 · p. 7
  • 90%: offline hierarchical clustering and feature extraction reduce storage and computational overhead for lifecycle behaviors.Source: §5 · p. 7
  • TWIN-V2’s cluster-aware target attention captures multifaceted interests, and experiments report more accurate and diverse recommendations than SOTA baselines.Source: §5 · p. 7
  • TWIN-V2 is deployed on Kuaishou’s main traffic, serving around 400 million active users daily.Source: §5 · p. 7

A.1 Summary of Notations

The appendix points readers to a table of important notations used in Section 3.

A.1 Summary of Notations

A.2 Behavior Feature Splits and Linear Projection

This section splits clustered behavior features to reduce the cost of multi-head target attention’s linear projection. It handles shared item features through caching and simplifies projection of user-item cross features.

A.2 Behavior Feature Splits and Linear Projection

  • The linear projection of clustered behavior features is identified as the computational bottleneck for applying MHTA to ultra-long sequences.Source: §A.2 · p. 9
  • K is split into inherent item features K_h and user-item cross features K_c, enabling separate linear projections.Inherent features are independent of the user or behavior sequence; cross features describe user-video interactions.Source: §A.2 · p. 9
  • Caching lets the system look up and gather K_hW_h because inherent item features are shared across users and behavior sequences.Source: §A.2 · p. 9
  • Because K_c features are not shared and each user watches a video at most once, the method simplifies their projection, compressing each cross feature to one dimension.With J cross features of embedding dimension 8, C = 8J; the simplified projection is equivalent to restricting W_c to a diagonal block matrix.Source: §A.2 · p. 9

A.3 Analysis of User Activity Levels

TWIN-V2 improves GAUC across all user-history groups, with larger absolute gains among users with more historical behaviors. Even users with shorter histories improve, which the authors attribute to clustered features representing a broader scope of behavior.

A.3 Analysis of User Activity Levels

  • The analysis compares GAUC for TWIN and TWIN-V2 across low-, medium-, and high-history user groups, alongside each group's share of users.Source: §A.3 · p. 9
  • TWIN-V2 improves GAUC across all user-history groups, with larger absolute increases among groups with more historical behaviors.The authors relate this pattern to broader life-cycle interests among users with more historical actions.Source: §A.3 · p. 9
  • Users with shorter histories also improve, which the authors attribute to cluster-aware target attention incorporating aggregated cluster features.These features make the behavior data supplied to GSU and ESU more comprehensive than TWIN's, according to the authors.Source: §A.3 · p. 9

Related papers

Loading 2407.16357v2…