Source-linked AI summary
ChronicleRec: Pre-training Temporally Anchored Tokens for Lifelong User Modeling
Chengkai Huang, Yubin Sheng, Liang Guo, Haoxi Liu, Junwei Pan, Shangyu Zhang, Zhixiang Feng, Chao Zhou, Chengguo Yin, Lina Yao, Haijie Gu, Jie Jiang
TL;DR
Ultra-long behavior histories are costly to model directly, while truncation and candidate-specific retrieval limit access to long-range signals. ChronicleRec pre-trains compact, chronologically anchored tokens for reusable user representations; experiments report gains over recent-window and single-pass compression baselines while recovering most full-attention performance.
Problem
Full-history attention is costly, truncation loses long-range signals, and candidate-specific retrieval repeats history modeling for each candidate.
Source: §1 · p. 1Method
ChronicleRec pre-trains chronologically anchored, target-independent tokens using recency-aware merging, causal query interleaving, and multi-horizon compression.
Source: §1 · p. 2, §3.1 · p. 3Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Modeling ultra-long user behavior sequences is crucial for industrial recommendation and online advertising, yet directly feeding thousands of historical actions into ranking models is computationally prohibitive, while truncation discards long-range signals. Existing lifelong-interest methods retrieve target-relevant behaviors for each candidate, coupling long-sequence modeling with candidate scoring and repeated online cost. Recent target-independent compression methods enable cached user summaries, but often append query tokens at the sequence end and use bidirectional encoding, producing unordered and redundant summaries that overlook temporal structure. We propose ChronicleRec, a pre-train-and-transfer framework that compresses an ultra-long behavior sequence once into a chronologically ordered set of Chronicle Tokens. ChronicleRec applies a recency-aware multi-granularity merge, preserving recent behaviors while coarsening distant history. It then interleaves query tokens with the merged sequence and uses a causal encoder, so each query summarizes only the history before its temporal anchor. A multi-horizon design masks different recent-history windows across parallel branches to learn complementary long-range interests. The compressor is pre-trained with a mask-and-predict objective that reconstructs held-out recent behaviors from compressed older history, aligning historical signals with near-present intent. Since Chronicle Tokens are target-independent, they can be cached per user, decoupling ultra-long sequence modeling from online candidate scoring. Experiments on KuaiRand and Tencent AdLive show that ChronicleRec outperforms recent-window and single-pass compression baselines while approaching full-attention performance. Token analyses reveal temporally organized and complementary representations, and a seven-day online A/B test confirms significant production gains.
1 Introduction
Ultra-long histories contain long-term interests, but full-history attention is costly and recent-window truncation discards distant signals. ChronicleRec learns chronologically ordered, target-independent tokens to summarize history for reuse across candidates.
1 Introduction
- Full-history attention is computationally prohibitive at industrial scale, while recent-window truncation discards long-range interest signals.Lifelong histories can reach thousands or tens of thousands of actions.Source: §1 · p. 1
- Target-attention retrieval recomputes target-dependent history representations per candidate, while end-appended bidirectional summaries lose temporal order and can be redundant.The latter limitations are described for VISTA [4].Source: §1 · pp. 1–2
- ChronicleRec interleaves query tokens along the timeline and encodes them causally, producing ordered tokens that summarize history only through each token’s temporal anchor.Its design also uses recency-aware merging, multiple temporal horizons, and alignment-oriented pre-training.Source: §1 · p. 2
- The authors report that pre-trained Chronicle Tokens recover most full-attention performance at lower serving cost and are cacheable for online serving.They also report downstream ranking improvements and significant production business gains.Source: §1 · p. 2
2 Related Work
Prior lifelong-interest systems commonly retrieve target-relevant behaviors separately for each candidate. Compression-based methods cache user summaries, and ChronicleRec adds temporal structure and multiple horizons to that paradigm.
2.1 Lifelong User Interest Modeling
- Search-based lifelong-interest systems retrieve target-relevant behaviors per candidate, making long-sequence representation target-dependent and serving cost grow with candidate count.SIM uses general search followed by exact search; ETA, SDIM, and TWIN-style designs refine retrieval or reduce its cost.Source: §2.1 · p. 3
2.2 Compression-Based Paradigms for Ultra-Long Sequence Modeling
- Compression methods cache compact latent summaries, but ChronicleRec differs from VISTA [4] by using recency-aware merging, causal query interleaving, and branches for multiple temporal horizons.The related work also situates these methods alongside Q-Former [8], Perceiver-style architectures, and efficient attention such as linear attention [7].Source: §2.2 · p. 3
3 Methodology
ChronicleRec maps embedded lifelong histories to a compact, target-independent token set. Its stages preserve recent detail, anchor causal summaries in time, cover multiple horizons, and transfer the compressor to a shared ranking head.
3.1 Problem Formulation and Overview
- The target-independent compressor maps an embedded history and mask to P much fewer Chronicle Tokens than the history length, reusable across candidates.A shared ranking head consumes the resulting token set.Source: §3.1 · p. 3
3.2 Recency-Aware Multi-Granularity Merge
- The merge keeps the near segment at full resolution and pools mid and far segments with progressively coarser windows, producing an ordered sequence dense near the present.The sequence is masked for padding and compresses only behaviors that exist for short-history users.Source: §3.2 · p. 3
3.3 Causal Query-Token Interleaving
- Learnable queries are interleaved at timeline anchors and causally encoded, so each Chronicle Token summarizes only history up to its anchor.Anchors are denser near the present and increasingly sparse toward the past.Source: §3.3 · pp. 3–4
3.4 Multi-Branch Multi-Horizon Compression
- Parallel branches mask progressively longer recent windows to access different horizons, and a weighted alignment objective trains each branch to predict held-out target labels from its compressed prefix.The objective encourages branches to retain complementary predictive information; auxiliary heads are removed at inference.Source: §3.4 · p. 4
3.5 Downstream Ranking and Training
- A shared ranking head consumes Chronicle Tokens alone or alongside a short recent-behavior window, after compressor pre-training and end-to-end ranker training.The compressor may be frozen or fine-tuned, while variants share the same head.Source: §3.5 · p. 4
4 Experiment
The experiments compare history-compression strategies on KuaiRand and Tencent AdLive under a shared ranking setup. ChronicleRec uses recency-aware merging, causal queries, and multi-horizon branches to produce compact tokens.
4.1 Experimental Setup
- KuaiRand averages close to 12K events per user, while Tencent AdLive averages close to 1,070; histories are truncated to maximum lengths of 2,048 and 4,000, respectively.KuaiRand uses click as its positive label, as does Tencent AdLive.Source: §4.1 · p. 4
- The comparison includes full-history attention, recent-100-behavior attention, VISTA compression, ChronicleRec, a hybrid variant, and multi-branch ChronicleRec.All methods share the ranking head and differ in how the ultra-long history is converted to tokens.Source: §4.1 · pp. 4–5
- Multi-horizon learning uses branches with drop offsets 0, 100, 300, 600, and 1,000 to model different recent-history windows and predict present behavior.Each branch masks a progressively larger recent window before compression and is independently supervised.Source: Figure 3 · p. 5, §4.1 · p. 5
- GAUC is the primary accuracy metric, and training uses BCE loss with Adam for three epochs and early stopping.Models share the embedding layer, head architecture, and training schedule.Source: §4.1 · p. 5
- The compressor keeps the latest 100 events one-to-one, merges the next 400 with kernel/stride 8/4, and merges distant history with kernel/stride 10/10.A four-layer, four-head causal Transformer with 11 query tokens produces Chronicle Tokens.Source: §4.1 · p. 5, Figure 2 · p. 5
4.2 Main Results
On both datasets, multi-branch ChronicleRec approaches full-history attention while outperforming recent-window and single-pass compression baselines. On Tencent AdLive, interleaved causal queries achieve the best GAUC among the tested query placements and attention directions.
4.2 Main Results
- Multi-ChronicleRec reaches 0.5580 GAUC on KuaiRand and 0.8034 GAUC on AdLive, exceeding Short-Attn and VISTA while approaching Full-Attn.On AdLive, Full-Attn reaches 0.8071 GAUC and Short-Attn 0.7950; on KuaiRand, Full-Attn reaches 0.5601 and Short-Attn 0.5307.Source: §4.2 · pp. 5–6, Table 2 · p. 6
4.3 Query-Token Placement and Attention Direction
- On Tencent AdLive, interleaved-causal queries achieve 0.8034 GAUC, ahead of end-causal at 0.7890, interleaved-bidirectional at 0.7930, and end-bidirectional at 0.7951.Interleaving improves GAUC by 0.0144 over end placement under causal encoding; causal encoding improves it by 0.0104 for interleaved queries.Source: Table 3 · p. 6, §4.3 · p. 6
- Interleaved queries occupy different timeline locations, and causal encoding restricts each query to preceding history, yielding chronologically structured rather than repeated global summaries.The reported comparison supports combining the two design choices.Source: §4.3 · p. 6
4.4 Ablation Study
Ablations on Tencent AdLive show positive contributions from recency-aware merging, causal masking, multi-branch learning, deep supervision, and Chronicle Alignment.
4.4 Ablation Study
- Removing recency-aware merging lowers GAUC by 0.0076, while removing causal masking or multi-branch learning lowers it by 0.0074 or 0.0072, respectively.The ablations compare uniform-stride merging, bidirectional interleaving, and a single branch against the full model.Source: §4.4 · p. 6
- Removing deep supervision reduces GAUC by 0.0039, and removing Chronicle Alignment causes the largest drop, −0.0086 GAUC.Chronicle Alignment aligns the compressed historical representation with near-present user interest before downstream ranking.Source: §4.4 · p. 6
4.5 Long-Range Signal Analysis
With the most recent 1,000 behaviors masked, ChronicleRec still outperforms a recent-window baseline, indicating that older history carries useful interest signals. ChronicleRec also reduces full-attention training cost, and its target-independent tokens can be cached for candidate scoring.
4.5 Long-Range Signal Analysis
- 0.5499 GAUC with the most recent 1,000 behaviors masked exceeds Short-Attn's 0.5307 GAUC, while full-history ChronicleRec reaches 0.5580 GAUC.The table reports KuaiRand GAUC when recent behaviors are masked and only older history is used.Source: §4.5 · p. 7, Table 5 · p. 7
4.6 Complexity and Efficiency
- ChronicleRec uses 28.0 GB of GPU memory and 620.3 training minutes, about a 4.8× speedup over Full-Attn.Full-Attn uses 49.1 GB and nearly 3000 minutes; Hybrid-ChronicleRec uses 24.8 GB and 614.3 minutes, while Multi-ChronicleRec uses 34.8 GB and 855.1 minutes.Source: §4.6 · p. 7, Table 6 · p. 7
- Chronicle Tokens are target-independent, so they can be computed once per user, cached, and reused across candidate scoring.The long-history methods have comparable parameter counts of approximately 1.23 B.Source: §4.6 · p. 7
- ChronicleRec's compressed tokens show clearer temporal organization and lower cross-region similarity than VISTA's broadly similar token pairs.The analysis groups token positions into far, middle, and recent temporal regions.Source: Figure 4 · p. 7
4.7 Token Similarity Analysis
ChronicleRec's tokens show more temporal structure and less cross-region redundancy than VISTA's. With multi-horizon modeling, the downstream ranker also attends across a broader range of token positions.
4.7 Token Similarity Analysis
- VISTA tokens have high similarity across many non-local pairs, whereas ChronicleRec tokens form temporal regions with lower similarity across distant regions.This pattern indicates clearer temporal specialization and lower cross-region redundancy in ChronicleRec's compressed tokens.Source: §4.7 · pp. 7–8
- ChronicleRec's interleaved causal queries are anchored at distinct times and summarize only their historical context.The analysis associates this structure with tokens representing different stages of a user's behavioral trajectory.Source: §4.7 · p. 8
- With multi-horizon modeling, attention spans a broad range of temporal positions; without it, the most recent token receives roughly one third of the attention mass.The multi-horizon example retains middle- and long-term tokens close to or above the uniform-attention baseline.Source: §4.7 · p. 8, Figure 5 · p. 8
- The analyses indicate that ChronicleRec preserves temporal diversity and that the downstream ranker can use information from different history horizons.The effective-rank, token-similarity, and attention analyses provide complementary evidence.Source: §4.7 · p. 8
4.8 Cumulative Information Gain of Chronicle Tokens
Cumulative target-relevant information rises more steadily with causal interleaving than with end-appended bidirectional queries, and multi-horizon learning achieves the highest cumulative mutual information over most token positions.
4.8 Cumulative Information Gain of Chronicle Tokens
- A sustained cumulative-MI increase represents added target-relevant information beyond the preceding token prefix.Because the debiased MI estimate uses finite samples, the analysis focuses on the curve's overall trend rather than strict monotonicity.Source: §4.8 · p. 8
- Causal interleaving yields a steadier cumulative-MI increase that eventually exceeds the end-appended bidirectional variant.The end-appended variant rises rapidly over the first query slots but grows only modestly afterward, suggesting overlapping target-relevant information.Source: §4.8 · p. 8
- Causal interleaving with multi-horizon learning achieves the highest cumulative MI over most of the token range.The result is consistent with retaining target-relevant signals from different temporal contexts.Source: §4.8 · p. 8
4.9 Online A/B Test
In a seven-day Weixin Moments Ads deployment, ChronicleRec significantly increased GMV. The production model incorporated Chronicle Tokens into an existing MixFormer ranking architecture under real-time and latency constraints.
4.9 Online A/B Test
- The test ran in Weixin Moments Ads pCVR prediction, a real-time setting with massive online traffic and stringent latency constraints.The production model used a MixFormer architecture for heterogeneous non-sequential features and user behavior sequences.Source: §4.9 · p. 8
- ChronicleRec increased GMV by +1.61%, with a 95% confidence interval of [0.678%, 2.547%] that excludes zero.The authors report this as a statistically significant improvement in the seven-day online A/B test.Source: §4.9 · p. 9
- Chronicle Tokens entered both the NS-Token and S-Token pathways without changing the ranking model's main architecture.The authors report that the production gain demonstrates offline gains transferring to business value under production latency constraints.Source: §4.9 · p. 9
5 Conclusion
ChronicleRec learns compact, temporally anchored tokens from lifelong behavior histories and transfers them to rankers, with reported benchmark, analysis, and production gains.
5 Conclusion
- ChronicleRec outperforms recent-window and single-pass compression baselines, recovers most full-attention performance, and delivers significant business gains in production.Evidence includes KuaiRand, Tencent AdLive, token-level analyses, and a seven-day online A/B test.Source: §5 · p. 9
- The framework learns Chronicle Tokens through recency-aware merging, causal query-token interleaving, multi-horizon compression, and alignment-oriented pre-training.Source: §5 · p. 9
- The compact, cacheable user representation transfers to downstream rankers, decoupling lifelong-history representation learning from online candidate scoring.Source: §5 · p. 9