Unleashing the Potential of Sparse Attention on Long-term Behaviors for CTR Prediction

1 Introduction

• Standard self-attention grows quadratically with sequence length, hindering efficient long-term behavior modeling and online deployment.The introduction notes that some approaches instead sample behaviors or focus on short-term histories, losing information and limiting model potential.Source: §1 · p. 1
• Sparse self-attention borrowed from vision or language often assumes uniform or locally continuous data, unlike non-uniform user behaviors.Sampling behaviors or focusing only on short-term histories also loses information and limits model potential.Source: §1 · pp. 1–2
• User behaviors vary between users and over time, with within-period behaviors reflecting interests and cross-period variation indicating preference shifts.The paper argues that capturing these dynamics can aggregate redundant behaviors and identify key behaviors for sparse attention.Source: §1 · p. 2
• SparseCTR uses personalized TimeChunking and global, transition, and local attention, with RelTemporal encoding relative time through learnable, head-specific biases.TimeChunking produces varied-length chunks with high within-chunk cohesion and low between-chunk coupling, enabling parallel processing; RelTemporal encodes duration, hour, and weekend information.Source: §1 · p. 2
• SparseCTR achieves state-of-the-art performance and consistently improves AUC as FLOPs increase across three orders of magnitude.The reported results also describe the model as efficient.Source: §1 · p. 2

2.1 Long-term Behavior Modeling

• Long-term CTR models use one-stage attention [9] [39] or two-stage subsequence retrieval [10] [11] [12] [40] [57], while deployment on long sequences remains costly.The discussion also covers scaling-law studies in recommendation [79] and models unifying retrieval and ranking, including HSTU [22] [49] [74] [78] [81].Source: §2.1 · p. 2

2.2 Sparse Self-attention Mechanisms

• Sparse attention methods select positional or structural subsets [2] [71] [80], use fixed-interval or random attention [5] [16] [73], or chunk sequences [44, 72].The section says these approaches may assume uniform or locally continuous distributions, while clustering methods such as Reformer disrupt sequence order; kernelized methods [18] [69] [36] [43] [45] approximate full attention and are difficult to extend with relative encoding.Source: §2.2 · p. 2

2.3 Relative Information Encoding

• Relative encoding captures distances between sequence elements; prior approaches include learnable embeddings [53], efficient position biases [14] [15] [48], and rotary encoding [23].Some relative-encoding methods add lookup operations or computational overhead, and efficient integration of temporal information remains underexplored.Source: §2.3 · pp. 2–3

3.1 Problem Formulation

• Historical behaviors B include features such as item ID and behavior time, while user profile features U can include age.The notation |B| and |U| denotes the numbers of behaviors and profile features, respectively.Source: §3.1 · p. 3
• Each target candidate has a binary click label, and the task is to predict its click-through probability from user, behavior, and candidate features.Candidate features can include its CTR over the last three days.Source: §3.1 · p. 3
• SparseCTR is the proposed model for the task, comprising an input layer, multiple SparseBlocks, and a prediction layer.Source: §3.1 · p. 3

3.2 Input Layer

• Causal masking lets each element attend only to preceding elements while keeping target candidates isolated, enabling all candidates to be predicted in one inference.This design addresses redundant computation from traditional frameworks that predict candidates individually.Source: §3.2 · p. 3
• Offline training aggregates pairwise candidate samples from one exposure into a listwise sample, then concatenates behavior and candidate sequences.Source: §3.2 · p. 3
• A uniform embedding table initializes embeddings for the mixed sequence and user features; behavior and candidate features are concatenated and mapped to dimension d.Source: §3.2 · p. 3

3.3 SparseBlock

• Each SparseBlock contains RMSNorm, an attention layer, and a Feedforward Network (FFN).The blocks are stacked to progressively extract higher-level, more abstract behavioral features.Source: §3.3 · p. 3
• RMSNorm stabilizes training with lower computational cost than standard LayerNorm, and a SwiGLU-based FFN enhances nonlinear representation capacity.The FFN combines Swish activation with GLU gating to capture complex feature interactions.Source: §3.3 · p. 3
• EvoAttention models global, transition, and local dependencies, while RelTemporal adds temporal information to attention computation.Together, these mechanisms model evolving interests and fine-grained temporal relationships in behavior sequences.Source: §3.3 · p. 3

3.4.1 Personalized Time-aware Chunking.

• TimeChunking divides each user sequence into the same number of varied-length chunks, preserving temporal personalization while enabling parallel processing.It selects the largest adjacent-behavior time intervals as segmentation points, naturally separating padding behaviors into a padding chunk.Source: §3.4.1 · pp. 3–4

3.4.2 Global Attention.

• Global attention aggregates keys and values within behavior chunks, then models dependencies across preceding chunks to capture long-term interests.The chunk-level aggregation operation is implemented as an MLP.Source: §3.4.2 · p. 4

3.4.3 Transition Attention.

• Transition attention selects the last m behaviors from each valid chunk and models dependencies among preceding transition behaviors.These tail behaviors can reflect current interests or potential shifts in user interest.Source: §3.4.3 · p. 4

3.4.4 Local Attention.

• Local attention models short-window interests while accounting for user profile features in sequence elements.The local behavior set combines a compressed representation of user features with behaviors in a local window.Source: §3.4.4 · pp. 4–5

3.4.5 Sparse Self-attention Fusion.

• A gating mechanism adaptively weights the global, transition, and local branch outputs to combine multi-granularity interest information.Each branch uses multi-head attention.Source: §3.4.5 · p. 5

3.5.1 Relative Time Encoding.

• Relative time encoding buckets absolute time differences between behavior timestamps, whose range is large and unevenly distributed.Learnable, head-specific bias coefficients are initialized geometrically to give attention heads different time sensitivities.Source: §3.5.1 · p. 5

3.5.2 Relative Hour Encoding.

• Relative hour encoding uses a sine function to represent cyclic hourly differences, including wraparound between hours such as 23 and 0.The hour difference is mapped from the time interval.Source: §3.5.2 · p. 5

3.5.3 Relative Weekend Encoding.

• Relative weekend encoding uses a piecewise function to represent periodic relationships between weekdays and weekends.Its learnable parameter is initialized in the same way as the relative-time coefficient.Source: §3.5.3 · p. 5

3.5.4 Relative Temporal Integration.

• RelTemporal assigns distinct learnable bias coefficients to temporal-information types and adds the resulting bias to each query-key pair in every attention branch.For global attention, the encoding uses each chunk's mean time.Source: §3.5.4 · p. 5

3.6 Prediction Layer

• The prediction layer extracts target-candidate embeddings from the mixed-sequence output of the l-th SparseBlock and fuses them with user features for MLP-based CTR prediction.The candidate and user-feature representations are fused before the MLP.Source: §3.6 · p. 6
• The model is optimized using binary cross-entropy over all target candidates.The loss is computed across samples and candidates in each batch.Source: §3.6 · p. 6
• The loss compares each candidate's predicted value with its ground-truth label across the batch.The batch contains N samples and |C| target candidates per sample.Source: §3.6 · p. 6

3.7.1 Time Complexity.

• EvoAttention gives SparseCTR time complexity O(B l(n|P|d + nm|P|d + nwd)), significantly below full attention’s O(B l n^2 d) when |P|, m, and w are much smaller than n.The cost is within the stacked l SparseBlocks; B is batch size.Source: §3.7.1 · p. 6

3.7.2 Space Complexity.

• SparseCTR’s learnable parameters mainly come from EvoAttention and FFN, and it introduces fewer additional parameters than full attention.Their stated complexities are O(4 l d^2 + 3 l d) and O(9 l d^2), respectively.Source: §3.7.2 · p. 6

4.1.1 Datasets; 4.1.2 Competitors

• Experiments use an industrial dataset with two years of behavior from over 86 million users, plus Alibaba1 and Ele.me datasets covering 22 and 30 days.Ten representative baselines include interaction-focused DIN and CAN and eight sequence-modeling methods divided into long-sequence and other groups.Source: §4.1.1 · p. 6, §4.1.2 · p. 6, Table 1 · p. 6

4.1.3 Evaluation Metrics; 4.1.4 Implementation Details

• Offline performance is evaluated with AUC and RelImpr, while online evaluation uses CTR, CPM, and inference time; models use same-order parameter counts and A100-80G GPUs.The setup uses embedding size 32, two encoder layers and eight heads for SparseCTR and Group III models, and sequence lengths 1024 or 50 by dataset.Source: §4.1.3 · p. 6, §4.1.4 · p. 6

4.2 Overall Performance

• SparseCTR outperforms all baselines, combining superior AUC with significantly lower computational complexity than Group III models.The AUC differences from baselines are reported as statistically significant; Table 2 presents the performance comparison.Source: §4.2 · p. 6, Table 2 · p. 7

4.3 Ablation Study

• Removing any EvoAttention or RelTemporal component worsens performance; global and transition branches and relative-time encoding matter most, while hour and weekend encodings have limited effects.Table 3 reports the ablations; removing local attention causes a smaller drop, with transition attention potentially compensating for it.Source: §4.3 · p. 7, Table 3 · p. 7

4.4.1 Comparison of Different Sparse Self-attention Methods; 4.4.2 Comparison of Different Chunking Methods

• EvoAttention beats DilatedAttention and NSAttention, while TimeChunking achieves optimal performance with minimal computational overhead.The comparisons address recommender-specific behavior distributions and personalized temporal chunking; Figure 2 presents the sparse-attention and chunking comparisons.Source: §4.4.1 · p. 7, §4.4.2 · p. 7, Figure 2 · p. 7

4.4.3 Comparison of Different Relative Encoding Methods; 4.5 Sparsity of Attention Analysis

• RelTemporal has the highest AUC while maintaining relatively high efficiency; across sparsity settings, EvoAttention beats full attention and reaches up to 8.9× speedup with less GPU memory.RelTemporal’s comparison includes positional and time-aware encodings; EvoAttention configurations are assessed on AUC, speed, and memory.Source: Figure 2 · p. 7, §4.4.3 · p. 8, §4.5 · p. 8

4.6 Scaling Law Analysis

• On the industrial dataset, SparseCTR preserves increasing AUC as FLOPs rise across three orders of magnitude, and jointly increasing sequence length and model scale yields greater efficiency gains.The AUC-versus-FLOPs analysis fits a power law; Figure 4 identifies FLOPs as determined by sequence length and model scale.Source: §4.6 · p. 8, Figure 4 · p. 8

4.7 Online A/B Test

• CTR improved by 1.72% and CPM by 1.41% in a 7-day online A/B test with 1% of traffic, while inference time remained 40 ms.The baseline’s full-attention component at sequence length 128 was replaced with SparseCTR at sequence length 1024.Source: §4.7 · p. 8

• SparseCTR exhibits an obvious scaling law and achieves superior online performance while supporting longer sequences and larger parameters under the same computational resources.Source: §5 · p. 8
• SparseCTR learns user interests from diverse temporal views through a three-branch sparse attention mechanism.Source: §5 · p. 8
• SparseCTR avoids computational complexity that scales quadratically with sequence length.Source: §5 · p. 8