PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Positional Encoding and Interaction Granularity for CTR Prediction

Xinxun Zhang, Yuxiang Wang, Xiaokai Zhou, Jiayue Li, Yi Li, Yue Liu, Jun Zhang, Xiaoling Zhou, Tieyun Qian, Xiao Yan, Jiawei Jiang

arXiv:2610.37140v1cs.IR

TL;DR

Transformer CTR models commonly use index-based positional encodings and token-level causal attention, although their suitability for CTR histories remains unclear. TRIP combines target-aware rank–time encoding with temporal-group interactions. Across four benchmarks, it outperforms the best-performing baseline on each dataset, with average relative AUC improvement of 1.47%.

  • Problem

    The suitability of common positional encodings and token-level causal interaction for CTR remains unclear, and empirical comparisons find that tested encodings do not beat NoPE while target-relative group bidirectionality outperforms both event-causal and global bidirectional attention.

    Source: §1 · p. 1

  • Method

    TRIP combines low-rank ordinal encoding and coarse-to-fine target-relative time encoding with bidirectional interaction within temporal groups and causal interaction across groups.

    Source: §1 · p. 2

  • Results

    Across four public CTR benchmarks, TRIP outperforms the best-performing baseline on each dataset, with 1.47% average relative AUC improvement and a 4.45% maximum on TAAC.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The findings support reconsidering both historical position representations and the granularity at which CTR models let behaviors interact.

    Source: §6 · p. 9

    TRIP’s fixed logarithmic temporal groups may not align with individual users’ interest transitions, and mean-pooling group summaries can obscure fine-grained event distinctions in later layers.

    Source: §3.4 · p. 5

Abstract

from arXiv · show

Click-through rate (CTR) prediction relies on historical behaviors and contextual features to estimate clicks on a target item. Existing Transformer-based CTR models often adopt index-based or temporal positional encodings and token-level causal interaction. However, we observe that most existing positional encodings do not outperform NoPE, while relaxing causal constraints within target-relative temporal groups also yields gains. These observations motivate a fundamental question: what positional encoding and interaction granularity does CTR prediction actually require? To address this question, we propose \textbf{TRIP}, which combines target-aware rank--time positional encoding with temporal-group interaction modeling. Specifically, Factorized Rank Encoding represents each behavior's ordinal position within the historical sequence through a low-rank factorization, while Target-Relative Time Encoding characterizes its temporal distance to the target on a coarse-to-fine scale. This joint design grounds each behavior's positional representation in its sequence context and temporal relation to the current target. Furthermore, Temporal Group-Causal Interaction enriches intra-group representations through bidirectional interaction and masked reconstruction while preserving inter-group causal order. Finally, behavior representations are condensed into compact group summaries for efficient group-level modeling in subsequent layers. Experiments on four public benchmarks show that TRIP improves AUC by up to 4.45\% relative to the strongest baseline.

1 Introduction

The suitability of common positional encodings and token-level causal attention for CTR histories remains unclear. TRIP combines target-aware rank–time encoding with temporal-group interaction, and improves on the strongest baseline across four public benchmarks.

1 Introduction

  • Existing Transformer CTR models often inherit index-based positional encodings and token-level causal attention from language modeling, but their suitability for CTR remains unclear.OneTrans, for example, combines historical behaviors and non-sequential features in a unified token sequence with learned absolute positions and causal Transformer blocks.Source: §1 · p. 1
  • On KuaiRand, none of six positional-encoding configurations beats its NoPE counterpart, while target-relative group bidirectionality beats event-causal and global bidirectional attention in AUC under all four encodings.These comparisons motivate reconsidering the positional information and interaction granularity used for CTR prediction.Source: §1 · p. 1, Figure 1 · p. 1
  • TRIP’s Factorized Rank Encoding represents ordinal position with rank-specific coefficients over shared basis vectors, while Target-Relative Time Encoding models target distance on a coarse-to-fine scale.Target-anchored coordinates form density-corrected learnable temporal potentials used as attention biases.Source: §1 · p. 2
  • Temporal Group-Causal Interaction enables bidirectional enrichment within target-relative temporal groups while preserving causal interaction across groups.Masked item reconstruction strengthens within-group contextual representations, which are condensed into compact group summaries for efficient modeling in subsequent layers.Source: §1 · p. 2
  • TRIP outperforms the best-performing baseline on all four datasets, with 1.47% average relative AUC improvement and a 4.45% maximum on TAAC.The joint rank–time encoding also transfers to other Transformer CTR models, improving relative AUC by up to 4.20% over eight alternative encodings on the same backbone.Source: §1 · p. 2

2 Preliminaries

The preliminaries frame CTR prediction as estimating whether a displayed target item will be clicked, given an impression’s features and observed behavior history. Historical events and non-sequential fields are tokenized into separate inputs for the model.

2 Preliminaries

  • Each CTR sample is an impression of a target item for a user at a prediction time, with associated user, item, and contextual features.The task is supervised binary classification, with the label indicating whether the displayed item is clicked.Source: §2 · p. 2
  • Historical behaviors provide sequential evidence of user interests, while the target item and impression context specify the prediction instance.Model parameters are learned from labeled interactions by minimizing binary cross-entropy loss.Source: §2 · p. 2
  • A historical event is tokenized from its item, associated features, and timestamp, while user, target-item, and contextual fields form non-sequential tokens.The resulting history and non-sequential token matrices have dimensions L × D and N_ns × D, respectively; N_ns need not equal the number of feature fields.Source: §2 · p. 2

3 Methodology

TRIP combines rank and target-relative time representations with temporal-group interactions for CTR prediction. It jointly contextualizes behaviors within groups, preserves causal flow across groups, and compresses behavior states for subsequent modeling.

3.1 Factorized Rank Encoding

  • FRE represents each history behavior’s sequence rank with a low-rank embedding, reducing positional parameters by 3.8–21.6× versus full learned APE under the benchmark configurations.Each rank uses coefficients over shared basis vectors; FRE is applied only to history tokens.Source: §3.1 · p. 3

3.2 Target-Relative Time Encoding

  • TRTE encodes each behavior’s temporal position relative to the target on a logarithmic scale that resolves recent events finely and distant history more coarsely.Increasing the subdivisions per logarithmic interval refines temporal resolution.Source: §3.2 · p. 3
  • TRTE offers learnable temporal potentials or temporal rotations to incorporate target-relative time into attention.The potential variant corrects for event counts within temporal intervals; the rotation variant applies temporal query/key rotations across Transformer blocks.Source: §3.2 · pp. 3–4

3.3 Temporal Group-Causal Interaction

  • TGCI groups history by target-relative temporal distance, enabling bidirectional interaction within groups while retaining causal information flow across groups.The same history can be regrouped when the prediction target changes.Source: §3.3 · pp. 4–5
  • Masked-item reconstruction trains contextualized event states to recover original item embeddings from local group context; masking and its reconstruction head are disabled at inference.Reconstruction and CTR prediction share the masked input and backbone pass during training.Source: §3.3 · p. 5
  • After within-group contextualization, TRIP mean-pools events into chronological group summaries that replace the original history, reducing its length from L events to G nonempty summaries.Subsequent causal Transformer blocks model interest evolution across summaries at lower computational cost.Source: §3.3 · p. 5

3.4 Optimization

  • TRIP jointly minimizes binary cross-entropy for CTR prediction and an auxiliary masked-item reconstruction loss.The joint objective includes a coefficient controlling the strength of reconstruction supervision.Source: §3.4 · p. 5
  • Fixed logarithmic temporal groups may not align with individual users’ interest transitions, while mean-pooled summaries can obscure fine-grained event distinctions in later layers.The paper identifies adaptive boundaries and selective event-level retention as possible ways to address these limitations.Source: §3.4 · p. 5

4 Experiments

Across four public benchmarks, TRIP-B is evaluated against eleven baselines and positional-encoding alternatives, with experiments also testing component contributions and group-summary efficiency. The results report consistent AUC gains, transferable rank–time encoding, and benefits from temporal grouping and compression.

4.1 Experimental Setup

  • Experiments use four public datasets and compare TRIP with eleven models and two FRE + TRTE variants against eight encoding configurations on three backbones.Models share hardware, optimizer, learning rate, batch sizes, and training duration; performance is reported as mean and standard deviation over three seeds.Source: §4.1 · pp. 5–6

4.2 Overall Performance

  • On KuaiRand, despite its AUC advantage, TRIP-B exhibits higher LogLoss; analysis attributes much of the gap to substantial overestimation of positive-feedback probabilities in certain low-positive-rate scenarios.Source: §4.2 · p. 6

4.3 Evaluation of Positional Encoding

  • FRE + TRTE transfers across three CTR backbones without TGCI, with its rotary variant improving TAAC AUC by 4.20%, 3.78%, and 3.65% on OneTrans, HyFormer, and MixFormer.The additive and rotary variants outperform the strongest competing encoding in 10 and 11 of 12 settings, respectively; at least one leads in every setting.Source: §4.3 · p. 6, Table 3 · p. 7

4.4 Ablation and Mechanism Analysis

  • Removing FRE or TRTE lowers AUC on all four datasets; on TAAC, AUC falls from 0.6337 to 0.6066 without FRE and 0.6073 without TRTE.Replacing the prediction-time reference also reduces AUC across datasets.Source: §4.4 · p. 7
  • Replacing group-causal interaction with token-level causal attention, removing group summaries, or removing reconstruction lowers mean AUC.The full model has the highest mean AUC among the displayed variants on all four datasets.Source: §4.4 · pp. 7–8, Figure 5 · p. 7
  • Temporal grouping achieves the highest mean AUC on all four datasets versus random and rank-contiguous groups, with group counts and sizes held constant.Removing density correction lowers mean AUC across datasets; learned temporal potentials vary by dataset and attention head, and are not final attention weights.Source: §4.4 · pp. 7–8, Figure 7 · p. 8, Figure 6 · p. 8

4.5 Efficiency of Group-Level Modeling

  • Group-summary compression raises AUC and reduces per-step training time by 6.3–32.8%.The first Transformer block contextualizes event-level inputs before compression; later layers process the shorter group-level sequence.Source: §4.5 · p. 8, Table 5 · p. 8

5 Related Work

CTR modeling has developed through feature-interaction and sequence-modeling approaches, increasingly unified in Transformer architectures. Suitable positional encoding and interaction granularity for unified CTR prediction remain unclear.

5 Related Work

  • Feature-interaction and sequence models increasingly converge in unified architectures, but suitable positional encoding and interaction granularity remain unclear for unified CTR prediction.The related work traces feature methods from factorization and explicit crosses to attention, and sequence methods from target-conditioned aggregation toward long-history processing.Source: §5 · p. 8
  • Sequential recommendation encodings represent sequence indices, event time, target relevance, or joint order–time relationships.Examples include multiscale timestamp rotations, combined index–time rotations, and target–history item dissimilarity; TRIP combines task-adaptive rank representations, prediction-time-anchored temporal encoding, and target-relative grouping.Source: §5 · pp. 8–9

6 Conclusion

The conclusion argues that CTR models need positional representations tied to rank, event time, and prediction time, alongside interaction granularity suited to target-relative temporal groups. Experiments on four public datasets and transfer experiments across three unified backbones support TRIP’s effectiveness and the portability of its rank–time design.

6 Conclusion

  • Existing positional encodings do not consistently improve CTR prediction, while target-relative temporal-group bidirectional interaction can outperform token-level causal and global bidirectional attention.Source: §6 · p. 9
  • TRIP jointly models sequence rank, event time, and the prediction-time reference, with intra-group bidirectional and inter-group causal interaction.Source: §6 · p. 9
  • Experiments on four public datasets demonstrate TRIP’s effectiveness, and positional-encoding transfer experiments across three unified backbones validate the portability of its rank–time design.Source: §6 · p. 9

A.1 Training and Evaluation Settings

The appendix describes shared training and evaluation choices, model configurations, and dataset-specific history processing. It also identifies the settings summarized in Table 6 without supplying their values here.

A.1 Training and Evaluation Settings

  • All models use AdamW, a fixed learning rate of 10^-3, weight decay 10^-2, one training epoch, and validation-selected checkpoints; results average three seeds.Runs use one NVIDIA H20 GPU, BF16 mixed precision, gradient-norm clipping at 1.0, and shared batch sizes of 1024 for training and 2048 for evaluation.Source: §A.1 · p. 10
  • TRIP uses four Transformer blocks, hidden dimension 64, four attention heads, and FFN dimension 256; Table 6 lists dataset-specific FRE, reconstruction, and time-encoding settings.Source: §A.1 · p. 10, Table 6 · p. 10
  • Histories retain positive-feedback events on KuaiRec and KuaiRand and all events on TAAC and TaobaoAd; valid events are chronologically ordered and truncated to the most recent L.Shorter histories are left-padded with valid events right-aligned, and all models use the same dataset-specific history construction.Source: §A.1 · p. 10

A.2 Module Implementation

The implementation specifies initialization and temporal-encoding choices for TRIP variants, plus reconstruction and summary handling across blocks.

A.2 Module Implementation

  • FRE initializes μ to zero, A with a normal distribution of standard deviation 1/K, and B with a truncated normal distribution of standard deviation 0.02, without PCA initialization.Source: §A.2 · p. 10
  • TRIP-B indexes a learnable temporal embedding table with a capped logarithmic function of target-relative distance, while TRIP-R uses fixed RoPE frequencies with base 104.Each head shares temporal potentials across prediction-side queries in TRIP-B.Source: §A.2 · p. 10
  • Groups with at least two events retain masked and visible items, and the reconstruction weight rises linearly to λ_rec over the first 5% of training steps.For TRIP-R, each summary uses the rounded mean event timestamp, from which target-relative coordinates are recomputed for subsequent blocks; FRE is not reapplied after compression.Source: §A.2 · p. 10

B.1 Additional Ablation Results

The appendix defines ablation variants and grouping controls, then reports analyses of learned rank geometry and density-corrected temporal preferences.

B.1 Additional Ablation Results

  • TGCA removal restores event-level causal attention while retaining groups, summaries, and reconstruction; other variants remove reconstruction, both components, or target-awareness.All ablated variants are compared with full TRIP-B.Source: §B.1 · p. 11
  • Figure 7’s grouping alternatives preserve group counts and the number of summaries, but differ in how labels or original group sizes are assigned to events.Random assignment uses a deterministic fixed-seed hash; rank-contiguous assignment partitions consecutive valid ranks according to permuted original group sizes.Source: §B.1 · p. 11

B.2 Learned Positional Representations

  • On TaobaoAd, FRE learns nonuniform rank relationships in TRIP-B, while TRIP-R shows more regular diagonal bands and pronounced rank-distance oscillations; other datasets show weaker oscillations.The analysis says low-rank sharing accommodates dataset-specific positional relationships and rank representations adapted to the temporal formulation.Source: §B.2 · p. 11
  • Occupancy correction changes the relative prominence of temporal windows, preserving head-specific preference differences while reducing event-multiplicity effects.Figure 10 averages biases over events within each interval using 8,192 test samples per dataset; the passage concludes TRTE separates head-specific temporal preference from sample-specific history density.Source: §B.2 · p. 11

B.3 Efficiency Analysis

On TaobaoAd, TRIP-B takes 72.63 ms/step, slightly less than OneTrans, while retaining dense attention; later group-summary sequences offset first-block overhead.

B.3 Efficiency Analysis

  • 72.63 ms/step for TRIP-B on TaobaoAd compares with 73.78 for OneTrans, 147.23 for HyFormer, 124.20 for MixFormer, and 123.88 for TokenFormer.The measurement uses an optimized temporal-bias implementation.Source: §B.3 · p. 11
  • TRIP retains OneTrans’s dense-attention formulation, with positional-encoding and reconstruction overhead in the first block offset by shorter group-summary sequences in later blocks.For N_blk blocks, attention cost is O((L+N_ns)^2D) + (N_blk −1)O((G+N_ns)^2D).Source: §B.3 · p. 11
  • History-side states and key/value projections can in principle be reused when candidates share a request timestamp and candidate-independent historical inputs.Historical tokens do not attend to non-sequential tokens.Source: §B.3 · p. 11

B.4 Hyperparameter Analysis

Hyperparameter sweeps show dataset-dependent preferences for rank and temporal resolution, benefits from moderate density correction, and a KuaiRand probability-calibration issue concentrated in one scenario.

B.4 Hyperparameter Analysis

  • Independent three-seed sweeps vary one hyperparameter at a time, and validation-selected settings need not match test-set peaks.Reference means can differ across sweeps and from Table 2.Source: §B.4 · p. 12
  • Factorization width preferences vary by dataset: TAAC mean AUC rises from 0.6095 at K=1 to 0.6340 at K=10, then declines at K=16.KuaiRec AUC ranges from 0.8502 to 0.8508 across tested widths, while K=2 already approaches KuaiRand’s best result.Source: §B.4 · p. 12
  • Increasing temporal resolution from m=1 to m=8 raises KuaiRand mean AUC from 0.7711 to 0.7781, but TAAC and TaobaoAd peak at m=1.The change subdivides TRTE’s logarithmic time scale without changing TGCI’s temporal grouping.Source: §B.4 · p. 12
  • All four datasets achieve their highest observed AUC with nonzero density correction: λ_cnt peaks at 0.5 on KuaiRec, TAAC, and TaobaoAd, and 0.75 on KuaiRand.KuaiRec AUC rises from 0.8488 without correction to 0.8510 at 0.5.Source: §B.4 · p. 12
  • Rank-specific variation matters more than the shared-offset scale, especially on TAAC and KuaiRand: zero rank-branch scale yields AUCs of 0.6068 and 0.7669 versus 0.6340 and 0.7777 at scale one.The shared-offset scale changes AUC by less than 0.001 on every dataset.Source: §B.4 · p. 12
  • Masking sweeps peak at ratios of 0.1, 0.05, 0.15, and 0.3 on KuaiRec, KuaiRand, TAAC, and TaobaoAd, respectively; a 0.5 ratio reduces AUC.Reconstruction-weight peaks are 0.01 on KuaiRand, TAAC, and TaobaoAd, and 0.05 on KuaiRec; increasing to 0.05 reduces AUC on the former three datasets.Source: §B.4 · pp. 12–13

B.5 KuaiRand Probability Calibration

  • KuaiRand TRIP-B predicts a mean positive-feedback probability of 46.74%, above the observed 37.67% rate.Tab 0, 19.03% of test samples, has 36.05% mean predicted versus 11.64% observed positive rate and contributes 69.93% of the sample-weighted net LogLoss gap relative to MixFormer.Source: §B.5 · p. 13
  • On KuaiRand, validation-fitted calibration reduces TRIP-B’s LogLoss gap to MixFormer from 0.0300 to 0.0011 while preserving its AUC advantage.TRIP-B’s LogLoss falls from 0.5778 to 0.5471 after positive-slope affine logit calibration.Source: §B.5 · p. 13
Loading 2610.37140v1…