PaperlayerView exact revision on arXiv

Source-linked AI summary

T-RoPE: Time-Aware Rotary Position Embedding for Sequential Recommendation

Yang Liu, Noel Loo, Ali Khanafer, Shuying Sun, Akshay Soni, Zhong Wu, Linjun Yang

arXiv:2609.30576v1cs.AIcs.IRcs.LG

TL;DR

In sequential recommendation, interaction indices encode event order but omit elapsed time, behavioral rhythms, and calendar phase. T-RoPE replaces index-only rotations with timestamp-aware rotations and related design changes. The paper reports consistent gains across public benchmarks and a large commercial dataset, alongside positive production lifts.

  • Problem

    Index-based RoPE represents event order but not elapsed gaps, multiscale periodicity, or absolute calendar phase.

    Source: §1 · pp. 1–2

  • Method

    T-RoPE retains RoPE’s interface while using timestamp-based rotations, learnable temporal coefficients, multiscale frequencies, shifted query alignment, and non-stationary keys.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    T-RoPE consistently outperforms baselines on public benchmarks and a large commercial dataset, with positive production lifts.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    The paper concludes that grounding rotary attention in time is principled and effective for recommendation within the evaluated settings.

    Source: §7 · p. 10

    The theoretical invariance result concerns query and key rotations by the same timestamp function under a common time shift.

    Source: §1 · p. 2, §4 · p. 6

Abstract

from arXiv · show

Large-scale recommenders increasingly adopt the sequential generative recipe behind large language models, bringing the Transformer into recommendation along with design choices made for text, including Rotary Position Embedding (RoPE). In language models, RoPE encodes token indices for relative position reasoning, but in recommendation, an interaction index records only event order, saying nothing about elapsed time, behavioral cycles across scales, or calendar phase. We revisit this choice and propose T-RoPE, a time-aware RoPE for sequential generative recommendation that replaces index-only rotation with timestamp-based angles, learnable temporal coefficients, multiscale frequency banks, shifted query alignment, and non-stationary key rotation. We prove that standard RoPE, even on timestamps, remains time-translation invariant and cannot distinguish seasonal contexts, and that T-RoPE breaks this invariance while preserving the RoPE interface. Across five public benchmarks, T-RoPE achieves the best result on every metric on every dataset, improving over the strongest baseline by 78--130\% in HR@10 on the sparse PixelRec data and 8--12\% across metrics on Amazon Books. On an industrial-scale e-commerce dataset with more than 6B interactions, it improves every metric over the HSTU + Time RAB backbone by 13--82\%, with ablations attributing the largest gains to multiscale frequencies ($+56\%$ NDCG@50) and non-stationary keys ($+4\%$). An online A/B test in the Shop app yields positive lifts in conversion rate ($+0.33\%$) and order count ($+0.63\%$). We also provide forward and backward algorithms whose added cost is linear in sequence length and head dimension, keeping time-aware RoPE practical for large generative recommenders.

1 INTRODUCTION

Index-based RoPE records event order but not elapsed gaps, multiscale rhythms, or calendar phase. T-RoPE changes timestamp-based query/key rotations to represent these signals, and the paper reports improvements across public and industrial settings and an online test.

1 INTRODUCTION

  • An interaction index records event order but loses whether events are seconds, weeks, or seasons apart.Index-based attention also lacks wall-clock frequency structure and is time-translation invariant, so it cannot distinguish the same history at different seasonal phases.Source: §1 · pp. 1–2
  • Prior temporal encodings capture only parts of the gap: TiSASRec and HSTU model discretized intervals or gaps, while TO-RoPE remains time-translation invariant.The paper argues that closing the gap requires changing how RoPE rotates queries and keys, rather than adding time only as a feature or bias.Source: §1 · p. 2
  • T-RoPE retains the RoPE interface while using timestamp-based angles, learned temporal coefficients, multiscale frequency banks, shifted query alignment, and non-stationary key rotation.Here, multiscale means an explicit continuous frequency bank covering multiple periods; learnable refers to temporal coefficients inside the encoding.Source: §1 · p. 2
  • The paper proves that shared query/key timestamp rotations remain time-translation invariant, while T-RoPE’s non-stationary key breaks that invariance.The invariant class includes standard RoPE applied to timestamps and TO-RoPE’s timestamp component.Source: §1 · p. 2
  • T-RoPE improves over temporal and RoPE-based baselines on every public benchmark and delivers positive production lifts across industrial and online settings.The reported settings include an industrial dataset with more than 6B interactions and an online A/B test in the Shop app.Source: §1 · p. 2
  • Forward and backward algorithms add cost linear in sequence length and head dimension.Source: §1 · p. 2

2 PRELIMINARIES

Generative recommendation uses timestamped interaction histories to predict the next item. Standard RoPE’s rotated query-key dot product depends on relative position offset, while Figure 1 contrasts this with timestamp-based and shifted-query geometry.

2 PRELIMINARIES

  • The model consumes a time-ordered history of timestamped events and autoregressively predicts the next item.Source: §2 · p. 2
  • A Transformer maps each item to query, key, and value vectors, and standard RoPE rotates them across D/2 two-dimensional planes.Source: §2 · p. 2
  • With integer-index angles, the rotated query-key dot product depends only on relative offset n − m.The figure contrasts uniform index positions with event-time rotations, where bursty events cluster and dormant gaps spread.Source: §2 · p. 3, Figure 1 · p. 3

3 THE T-ROPE METHOD

T-RoPE retains RoPE’s interface but uses timestamp-based, multiscale rotations aligned to prediction time, with non-stationary keys to represent calendar phase. Its components combine true elapsed time with learned temporal sensitivities and cycle frequencies.

3.1 TIMESTAMP-BASED ROTATION

  • Absolute Unix timestamps replace interaction indices, so rotation phases reflect actual event-time gaps rather than sequence offsets.The timestamp is used in seconds before rotation without centering or normalization.Source: §3.1 · p. 3

3.2 LEARNABLE TEMPORAL COEFFICIENTS

  • Learnable query and key coefficients adjust temporal-band strengths, with coefficients aligned across most dimensions but temporal sensitivity varying substantially by layer.Layers therefore learn distinct temporal biases rather than sharing one bias.Source: §3.2 · pp. 3–4

3.3 MULTISCALE FREQUENCY BANK

  • A geometrically spaced frequency bank covers multiple time scales, while learned coefficients upweight selected bands that can represent recurring patterns without named calendar events.Training favors some temporal bands over others, and different frequencies can align with different user cycles.Source: §3.3 · pp. 4–5

3.4 SHIFTED QUERY ALIGNMENT

  • Training assigns each query the next-event timestamp while keys retain historical timestamps; inference assigns the final query the current session time.This lets the same item history be evaluated under different prediction seasons.Source: §3.4 · p. 5, Figure 4 · p. 5

3.5 NON-STATIONARY KEY ENCODING

  • Rotating keys with the transpose of the standard rotation changes the attention phase from a timestamp gap to a timestamp sum, making attention sensitive to calendar shifts.The rotation remains norm-preserving and uses the same rotation primitive.Source: §3.5 · p. 5

3.6 COMPLETE T-ROPE FORMULATION

  • The complete formulation combines timestamp-aware, multiscale, prediction-aligned, non-stationary rotations; its added cost is linear in sequence length and head dimension.Across planes, the learned combination of prediction and historical timestamps represents local recency and repeated temporal structure.Source: §3.6 · pp. 5–6

4 THEORETICAL ANALYSIS

The analysis shows that timestamp substitution alone does not give standard RoPE access to absolute calendar phase: shared query and key coefficients leave attention dependent only on time gaps. T-RoPE’s non-stationary keys break that invariance and can represent periodic dependence on prediction time.

Time-translation invariance

  • Shared-coefficient timestamp RoPE remains time-translation invariant: attention depends only on the query–key time gap, not their shared calendar location.A common shift of both timestamps leaves the relative angle unchanged.Source: §4 · p. 6

Breaking the invariance

  • With nonzero query coefficients, T-RoPE’s non-stationary key makes attention change under common timestamp shifts that do not yield a multiple of 2π.The per-plane phase combines query and key timestamps additively rather than using only their difference.Source: §4 · p. 6

Seasonality representability

  • T-RoPE can represent sinusoidal dependence on prediction-time calendar phase with period P when a dimension’s effective query frequency matches a nonzero integer harmonic of P.The matching condition makes attention periodic in prediction time with period P.Source: §4 · p. 6

5 EXPERIMENTS

The experiments assess T-RoPE on five public benchmarks, an industrial Shop dataset, and an online Shop app A/B test. T-RoPE leads across the public benchmarks, while Shop ablations and the online test report further offline and production results.

5.1 VALIDATING T-ROPE ON PUBLIC DATASETS

  • Public comparisons use shared inputs, preprocessing, evaluation, and training settings across five existing methods and HSTU + T-RoPE.The benchmarks report HR@10, HR@50, NDCG@10, NDCG@50, and MRR.Source: §5.1 · p. 7
  • HSTU + T-RoPE achieves the best result on all five metrics on every public dataset, beating HSTU + Time RAB by 8–12% across metrics on Amazon Books and TO-RoPE by 78–130% in HR@10 on PixelRec.Across PixelRec splits, it also improves over TO-RoPE by 72–112% in HR@50.Source: §5.1 · pp. 7–8

5.2 ABLATION STUDY ON AN INDUSTRIAL-SCALE DATASET

  • On Shop, removing multiscale frequencies cuts NDCG@50 by 36%, while shifted query alignment adds 7% NDCG@50 at inference.The ablation table reports cumulative component removals as well as single-component removals from the full model.Source: §5.2 · p. 8, Table 3 · p. 8
  • Adding the non-stationary key improves recall@200 by 3.2% and NDCG@50 by 4.2% over the shifted model, outperforming it on all five metrics.Removing shifted-query rotation lowers recall@200 by 3.0%, while removing key rotation lowers NDCG@50 by 10.3%.Source: §5.2 · p. 8
  • Removing Time RAB lowers all metrics, with a 27.0% drop in MRR, while T-RoPE without RAB still improves NDCG@50 by roughly 50% over the baseline.The authors interpret the ablation as showing that RAB captures relative-recency signals not captured by T-RoPE, and that the mechanisms contribute independently.Source: §5.2 · pp. 8–9

5.3 ONLINE A/B TEST

  • In the Shop app A/B test, conversion rate lifts by 0.33% (p = 0.037) and orders increase by 0.63% (p = 0.094) against the production control.The authors describe the order result as directional evidence of downstream value.Source: §5.3 · p. 9, Table 4 · p. 9

6 RELATED WORK

Sequential recommendation has moved from recurrent and convolutional models to Transformer-based and autoregressive generative approaches. Related temporal methods capture portions of time information, while T-RoPE is presented as a drop-in encoding for Transformer backbones that adds absolute-time awareness.

Sequential recommendation

  • Sequential recommendation progressed from recurrent and convolutional models to Transformer attention, production-scale HSTU, and autoregressive generation over semantic tokens.Examples include SASRec, BERT4Rec, TIGER, ActionPiece, PSID, and HLLM.Source: §6 · p. 9
  • T-RoPE is described as a drop-in positional encoding compatible with Transformer backbones.The passage situates it among sequential generative recommenders over discrete semantic tokens.Source: §6 · p. 9

Temporal information in sequential recommendation

  • Temporal methods capture different signals: TiSASRec models bucketed elapsed gaps, while calendar-feature attention mixtures condition on features such as hour of day.Knowledge-guided methods also learn time-decay kernels over item relations.Source: §6 · p. 9
  • TO-RoPE blends indices and timestamps but remains time-translation invariant with shared query and key coefficients; T-RoPE uses non-stationary keys to break that invariance.The passage connects this change to absolute-time awareness for patterns such as seasonal purchase cycles.Source: §6 · p. 9

7 CONCLUSION

The conclusion presents T-RoPE as a time-aware rotary embedding for recommendation and reports consistent gains on public and commercial data, especially under high temporal heterogeneity. The accompanying statements describe the authors’ AI-use boundaries, governance of proprietary studies, and reproduction materials.

7 CONCLUSION

  • T-RoPE rotates attention according to event time rather than order, and the paper proves shared-geometry timestamp encodings are time-translation invariant while non-stationary keys break that invariance.The conclusion reports consistent baseline improvements on public benchmarks and a large commercial dataset, with the largest gains under high temporal heterogeneity.Source: §7 · pp. 9–10

AI USE STATEMENT

  • AI assistance was limited to sentence-level writing edits; the authors state it was not used for research, methodology, experiments, implementation, or analysis.The authors say they reviewed every AI-assisted edit and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • Industrial and online studies used authorized data under company privacy and experimentation processes, with aggregate results and no released user records or personally identifying information.The proprietary data and production infrastructure cannot be made public.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The paper points to method, protocol, implementation, proof, cost, and throughput appendices as reproduction support for public-benchmark results.Appendix D reports architecture, optimization settings, random seed, and hardware.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 PROOF OF THEOREM 1

The proofs characterize when T-RoPE attention is invariant to common timestamp shifts and show how its learned frequencies can represent calendar-periodic effects. The implementation adds learnable-angle gradients and computation that is linear in sequence length and head dimension.

A.1 PROOF OF THEOREM 1

  • Shared timestamp coefficients make the attention score depend only on the time gap, so standard RoPE on timestamps and TO-RoPE’s timestamp component remain time-translation invariant.Summing the plane and head contributions preserves this invariance.Source: §A.1 · p. 13
  • Independent query and key coefficients can break the standard-RoPE symmetry; opposite nonzero coefficients make the score depend on the timestamp sum.The shared-coefficient condition is therefore not the only route to invariance.Source: §A.1 · p. 13

A.2 PROOF OF THEOREM 2

  • When a dimension has a nonzero combined query–key coefficient, T-RoPE is not time-translation invariant; the learned degenerate case instead restores invariance.The deployed model does not reach that degenerate case.Source: §A.2 · pp. 13–14

A.3 PROOF OF COROLLARY 1

  • T-RoPE can express sinusoidal dependence on prediction time with period P when an effective query frequency matches a nonzero integer harmonic of P.The resulting sine and cosine terms are periodic in the prediction timestamp.Source: §A.3 · p. 14

B.1 FORWARD AND BACKWARD ALGORITHMS

  • The custom backward pass returns both input gradients and angle gradients because the rotation angles depend on trainable coefficients.Angle-gradient contributions are summed across attention heads.Source: §B.1 · pp. 14–15

B.2 PARAMETER AND FLOP OVERHEAD

  • T-RoPE adds 4NP forward arithmetic FLOPs for timestamp-dependent angles, with extra training work linear in sequence length and head dimension while attention remains quadratic in sequence length.Runtime trigonometric evaluations may also be needed when sine and cosine are computed on the fly.Source: §B.2 · p. 16

B.3 STATIC LATENCY ANALYSIS.

Static throughput snapshots show modest observed overhead for T-RoPE relative to baseline HSTU and throughput comparable to TO-RoPE.

B.3 STATIC LATENCY ANALYSIS.

  • T-RoPE has modest observed throughput overhead relative to baseline HSTU and is comparable to TO-RoPE.The snapshots report mean ± sample standard deviation in iterations per second.Source: §B.3 · p. 16, Table 5 · p. 16

C DATASET STATISTICS

The section reports statistics for the five public benchmarks used in the paper.

C DATASET STATISTICS

D HYPERPARAMETER SETTINGS

The appendix provides shared public-benchmark training defaults, with a sequence-length exception for ML-20M. It also visualizes how T-RoPE’s components organize multiscale event blocks by real-time spacing.

D HYPERPARAMETER SETTINGS

  • The public benchmarks share defaults of two Transformer blocks, 128 in-batch negatives, and a learning rate of 10^-3.The defaults apply to ML-20M, Amazon Books, Pixel200K, Pixel1M, and Pixel8M.Source: D HYPERPARAMETER SETTINGS · p. 16, Table 7 · p. 17
  • Public runs use AdamW, 128 sampled negatives, and SampledSoftmaxLoss; histories are truncated to each dataset’s maximum sequence length.Training uses one NVIDIA H100, with at least 10 and at most 100 epochs.Source: D HYPERPARAMETER SETTINGS · p. 16

E ADDITIONAL VISUALIZATIONS

  • The visualized day-, week-, month-, and year-scale blocks become increasingly grouped by real-time spacing as T-RoPE components are added, with full T-RoPE separating them by elapsed time.Figure 5 compares index RoPE, timestamp RoPE, shifted-query timestamp RoPE, and full T-RoPE.Source: E ADDITIONAL VISUALIZATIONS · p. 17, Figure 5 · p. 17

Related papers

Loading 2609.30576v1…