PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Scaling Laws of CTR Model for Online Performance Improvement

Weijiang Lai, Beihong Jin, Jiongyan Zhang, Yiyuan Zheng, Jian Dong, Jia Cheng, Jun Lei, Xingxing Wang

arXiv:2508.15326v2cs.IR

TL;DR

CTR prediction must improve accuracy under stringent latency constraints, and high-performing scaling-law models are difficult to deploy online. The paper builds scalable SUAN and distills it into LightSUAN, reporting online CTR and CPM gains.

  • Problem

    CTR models face bottlenecks in improving accuracy under stringent latency requirements, while high-performing scaling-law models cannot be deployed online under inference-time constraints.

    Source: §1 · pp. 1–2

  • Method

    SUAN uses stacked UABs for scalable CTR modeling, while online distillation transfers high-grade SUAN knowledge to LightSUAN with sparse self-attention and parallel inference.

    Source: §1 · p. 2

  • Results

    2.81% CTR and 1.69% CPM gains online accompanied SUAN AUC scaling across three orders of magnitude and distilled LightSUAN outperforming one-grade-higher SUAN.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The study offers insight into factors such as model grade and regularizers that affect CTR scaling laws and explores distillation as a way for online services to benefit from them.

    Source: §5 · p. 9

    User-profile features are excluded from self-attention for efficiency because they lack obvious temporal or associative relationships.

    Source: §3.2.1 · p. 4

Abstract

from arXiv · show

CTR models play a vital role in improving user experience and boosting business revenue in many online personalized services. However, current CTR models generally encounter bottlenecks in performance improvement. Inspired by the scaling law phenomenon of LLMs, we propose a new paradigm for improving CTR predictions: first, constructing a CTR model with accuracy scalable to the model grade and data size, and then distilling the knowledge implied in this model into its lightweight model that can serve online users. To put it into practice, we construct a CTR model named SUAN (Stacked Unified Attention Network). In SUAN, we propose the UAB as a behavior sequence encoder. A single UAB unifies the modeling of the sequential and non-sequential features and also measures the importance of each user behavior feature from multiple perspectives. Stacked UABs elevate the configuration to a high grade, paving the way for performance improvement. In order to benefit from the high performance of the high-grade SUAN and avoid the disadvantage of its long inference time, we modify the SUAN with sparse self-attention and parallel inference strategies to form LightSUAN, and then adopt online distillation to train the low-grade LightSUAN, taking a high-grade SUAN as a teacher. The distilled LightSUAN has superior performance but the same inference time as the LightSUAN, making it well-suited for online deployment. Experimental results show that SUAN performs exceptionally well and holds the scaling laws spanning three orders of magnitude in model grade and data size, and the distilled LightSUAN outperforms the SUAN configured with one grade higher. More importantly, the distilled LightSUAN has been integrated into an online service, increasing the CTR by 2.81% and CPM by 1.69% while keeping the average inference time acceptable. Our source code is available at https://github.com/laiweijiang/SUAN.

1 Introduction

CTR prediction must balance high accuracy against stringent latency, while further performance gains are difficult. The paper proposes scalable SUAN and distills its knowledge into LightSUAN for online use.

1 Introduction

  • Accuracy and stringent latency conflict, and even a 0.001 AUC increase is considered significant, highlighting the difficulty of further CTR improvement.Source: §1 · pp. 1–2
  • The proposed paradigm scales CTR performance by increasing model grade and data size, with model grade defined as model size plus behavior-sequence length.Source: §1 · p. 2
  • High-grade CTR models can provide accuracy but incur computational overhead that does not satisfy inference-time constraints, motivating transfer to a lightweight model.Source: §1 · p. 2
  • SUAN stacks UABs with multiple attention mechanisms, while LightSUAN adds sparse self-attention and parallel inference and is trained through online distillation from high-grade SUAN.Source: §1 · p. 2
  • 2.81% CTR and 1.69% CPM gains online accompanied SUAN AUC scaling across three orders of magnitude and distilled LightSUAN outperforming one-grade-higher SUAN.The online service kept average inference time acceptable.Source: §1 · p. 2

2 Related Work

Related CTR work models feature interactions or user behavior sequences, while recommender-system research has explored scaling laws. The paper identifies online inference constraints as an underexplored barrier to benefiting from scalable CTR models.

2.1 CTR Models

  • CTR models span feature-interaction approaches such as DIN and CAN and behavior-sequence approaches; long-sequence systems such as SIM, ETA [8], and TWIN often use two-stage decomposition.Source: §2.1 · p. 2

2.2 Scaling Laws in Recommender Systems

  • Scaling-law work includes sequential recommenders LSRM and SRT, while CTR approaches include Wukong and HSTU; online benefit under inference constraints remained underexplored.Source: §2.2 · p. 3
  • This work adds both a key module for CTR scaling laws and an approach for online services to benefit from scaling-law CTR models.Source: §2.2 · p. 3

3.1 Overview

SUAN encodes a target-aware user behavior sequence with stacked UABs and predicts the click probability from the final candidate representation, profile, and other features.

3.1 Overview

  • The model input includes a user's behavior sequence, profile, and candidate features, with behaviors including item and timing information.Source: §3.1 · p. 3
  • SUAN comprises an input layer, a stack of UAB encoders that model the target-aware sequence, and a prediction layer.Source: §3.1 · p. 3
  • SUAN appends candidate regular features to the behavior sequence to form a target-aware sequence, helping the encoder model behavior–candidate relationships.Source: §3.1 · p. 3
  • The prediction layer feeds the final UAB candidate representation, flattened user-profile embedding, and other features to an MLP to predict click probability.Source: §3.1 · p. 3
  • The model is optimized with standard binary cross-entropy loss against the ground-truth label.Source: §3.1 · p. 3

3.2 Unified Attention Block

The UAB is a causal encoder built around self-attention, combining sequential and non-sequential features through multiple attention mechanisms. Its components normalize and transform behavior representations to capture feature importance from different perspectives.

3.2 Unified Attention Block

  • The UAB combines RMSNorm, self-attention, AFNet, and a feedforward network to discover feature importance from multiple perspectives.RMSNorm is described as a regularizer that reduces computation and maintains training stability.Source: §3.2 · p. 3

3.2.1 Self-attention Layer.

  • Self-attention uses relative positional and temporal bias to identify key behaviors and capture user interests.The model does not apply this layer to user profile features such as age and gender, which are described as lacking obvious temporal or associative relationships.Source: §3.2.1 · pp. 3–4

3.2.2 Adaptive Fusion Network.

  • AFNet uses cross-attention to relate behavior sequences to user profiles, then dual alignment attention to adaptively combine the resulting representations.The cross-attention layer uses the self-augmented sequence as query and the profile embedding as key and value; dual alignment uses global feature-distribution information to gate features.Source: §3.2.2 · pp. 4–5

3.2.3 Feedforward Network.

  • The feedforward network uses SwiGLU, combining Swish’s everywhere-differentiable property with the gating mechanism in GLU.The passage notes that similar feedforward networks are used in LLMs.Source: §3.2.3 · p. 5

3.3 Deployment Optimization

LightSUAN reduces SUAN’s inference burden with sparse self-attention and parallel inference. Online distillation trains it alongside a high-grade SUAN teacher using prediction and softened-logit losses.

3.3.1 Speedup Strategies.

  • Sparse self-attention combines local and dilated attention to capture recent and distant relationships while reducing attention connections.Local attention uses a window size k; dilated attention attends to every r-th element.Source: §3.3.1 · p. 5
  • Parallel inference batches m2 candidates per inference, reducing the per-user inference count from m1 to ceil(m1/m2).Offline training is modified to match the online input by appending m2 − 1 placeholders to single-candidate samples.Source: §3.3.1 · p. 5

3.3.2 Online distillation.

  • Online distillation trains a low-grade LightSUAN student alongside a high-grade SUAN teacher, using each model’s inputs and outputs.The teacher’s softened prediction serves as a soft label in a third binary cross-entropy loss, alongside supervised losses for both models.Source: §3.3.2 · p. 5
  • The distillation-loss weight λ is set to temperature t to counterbalance the loss’s gradient scaling as 1/t.This is intended to keep the distillation loss’s contribution appropriate during optimization.Source: §3.3.2 · p. 5

3.4 Complexity Analysis

The complexity analysis describes the attention-related time cost of SUAN and how sparse attention changes it. It also identifies UAB components as a primary source of extra learnable parameters.

3.4.1 Time complexity.

  • With full attention, the time complexity for l UABs is O(B l L^2 n1 d + B l L n2 n1 d), where B is batch size.The passage attributes SUAN’s time complexity primarily to self-attention and cross-attention within the UAB.Source: §3.4.1 · p. 5
  • In LightSUAN, sparse self-attention changes the stated time complexity to O(B l L(k + L/r)n1 d + B l L n2 n1 d).The passage begins describing an additional inference-time reduction from parallel inference but does not provide the full expression.Source: §3.4.1 · pp. 5–6

3.4.2 Space complexity.

  • The extra learnable parameters primarily come from UAB components, including self-attention, AFNet, and the feedforward network.The passage identifies these components as the main source of additional parameters beyond embeddings.Source: §3.4.2 · p. 6

4 Experiments

Across three datasets, SUAN significantly outperforms all competitors; ablations show performance declines when its modules are removed or changed. Experiments compare datasets and methods using offline AUC and online CTR, CPM, and inference time.

4.1.1 Datasets.

  • Experiments use Eleme1, Taobao2, and an industrial dataset; the industrial dataset includes up to 1,000 behavior records for each of 88 million users.Eleme1 contains 30-day user behaviors, and Taobao2 contains 22-day behaviors from nearly one million randomly chosen users.Source: §4.1.1 · p. 6

4.1.2 Competitors.

  • Eight competitors cover feature-interaction models DIN and CAN, plus six sequence-modeling methods spanning long-sequence and other approaches.SoftSIM, HardSIM, ETA, and TWIN target long sequences; BST and HSTU do not.Source: §4.1.2 · p. 6

4.1.3 Evaluation Metrics.

  • Offline evaluation uses AUC and relative improvement, while online A/B tests measure CTR, CPM, and inference time.DIN is the base model for calculating relative improvement.Source: §4.1.3 · p. 6, Table 2 · p. 7

4.1.4 Implementation Details.

  • For fairness, all models are configured with parameter counts of the same order of magnitude, dimension 8, and the same prediction-layer MLP [1024, 512, 256, 1] with Dice activation.On industrial and Taobao datasets, sequence length is 1,000 for Group II and 100 for Groups I and III; on Eleme, it is 50 for Group II and 10 for Groups I and III.Source: §4.1.4 · p. 6

4.2 Overall Performance

  • SUAN significantly outperforms all competitors on three datasets, with AUC differences from each comparison model statistically significant.All tested p-values were below 0.05.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Ablation Study

  • Every ablation variant degrades performance; removing the target-aware sequence causes substantial deterioration, while removing self-attention bias causes the greatest decline.The authors attribute the greatest decline to losing relative-position and temporal information between behaviors.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Scaling Law Analysis

The analysis tests whether SUAN’s AUC follows scaling relationships as model grade and data size increase, using industrial data because public datasets are limited. It reports power-law patterns for model size, sequence length, and data size, and examines regularizers associated with maintaining scaling behavior.

4.4 Scaling Law Analysis

  • Scaling-law analysis varies model grade and data size and evaluates model performance using AUC on the industrial dataset because public datasets are size-limited.Model grade comprises model size and behavior-sequence length; data size is the number of input samples.Source: §4.4 · p. 7

4.4.1 Impact of Scaling the Model Grade.

  • With fixed data size, SUAN’s AUC follows power laws as model size and sequence length vary, with fitted upper bounds near 0.72.The fitted curves have R^2 values close to 1 for model size; the sequence-length analysis also confirms a scaling law.Source: §4.4.1 · p. 7, Figure 4 · p. 8

4.4.2 Impact of Scaling the Data Size.

  • Across sample sizes spanning three orders of magnitude, SUAN’s data-size power law has a higher fitted upper bound and smaller growth exponent than HSTU and DIN.The authors interpret this pattern as more effective use of training samples and scalability across data scales.Source: §4.4.2 · pp. 7–8, Figure 4 · p. 8
  • Increasing data size yields a higher AUC upper bound and faster growth for SUAN than increasing model grade, according to the fitted scaling parameters.The data-size upper-bound coefficient is higher, and its exponent is lower, than the corresponding model-grade coefficients.Source: §4.4.2 · pp. 7–8, §4.4.1 · p. 7

4.4.3 Regularizers for Supporting Scaling Laws.

  • Only the size-4 variants replacing RMSNorm or pre-norm show performance degradation that undermines scaling; the authors attribute these results to training stability.They speculate that RMSNorm and pre-norm help prevent outliers and fluctuations from disrupting scaling laws.Source: §4.4.3 · p. 8

4.5 Deployment Optimization Analysis

Sparse self-attention improves inference speed without significant performance loss at dilated rates r <= 4, while online distillation improves LightSUAN performance. In the online A/B test, distilled LightSUAN increased CTR and CPM over the baseline with a 43ms average inference time.

4.5.1 The impact of sparse self-attention.

  • Inference speed increases as the dilated rate grows, with no significant performance drop when r <= 4; AUC also rises rapidly with longer sequences.Figure 5a compares relative inference speed against full attention at sequence length 100; Figure 5b reports AUC across dilated rates and sequence lengths.Source: §4.5.1 · p. 8, Figure 5 · p. 8

4.5.2 The impact of online distillation.

  • DistilSUAN-3 outperforms LightSUAN-1 by 6‰ and also exceeds SUAN-1, equipped with a larger model, and SUAN(L), which takes a longer input sequence, as shown in Table 2.Source: §4.5.2 · p. 8

4.6 Online A/B Test

  • CTR rose 2.81% and CPM rose 1.69% with distilled LightSUAN, at 43ms average inference time, versus the baseline.Original SUAN improved CTR by 2.67% and CPM by 1.63%, with average inference time rising from 33ms to 48ms.Source: §4.6 · p. 9

5 Conclusion

The paper explores scaling laws for CTR prediction through user-behavior modeling and presents knowledge distillation as a way for online services to benefit from such models.

5 Conclusion

  • The paper explores whether modeling user behaviors can make CTR prediction performance follow scaling laws.It presents knowledge distillation as a way for online services to benefit from CTR models with scaling laws.Source: §5 · p. 9
  • The study offers insights intended to support online performance improvement in industry.The conclusion also describes hands-on experience identifying factors and components that affect CTR-model scaling laws.Source: §5 · p. 9
  • The paper shares experience identifying factors such as model grade and components such as regularizers that affect CTR-model scaling laws.Source: §5 · p. 9
Loading 2508.15326v2…