PaperlayerView exact revision on arXiv

Source-linked AI summary

Preference-Drift-Aware Subsequence Learning and Hierarchical Context Fusion for Long-Sequence Generative Recommendation

Fei Li, Qingyun Gao, Jianzhe Zhao, Guibing Guo, Beibei Kong, Lei Cheng, Chengxiang Zhuo, Zang Li

arXiv:2609.12556v1cs.IR

TL;DR

Long-sequence generative recommendation must balance the cost and accuracy of modeling growing histories, while accounting for changing preferences and dependencies across subsequences. DRIFT learns preference-aware subsequences and fuses them with recent interactions; experiments report improved accuracy and computational efficiency across three public datasets.

  • Problem

    Existing full-sequence and target-aware retrieval methods do not effectively model stage-wise user-interest evolution or dependencies among subsequences.

    Source: Introduction · pp. 1–2

  • Method

    DRIFT learns soft subsequence boundaries from multidimensional preference drift and uses cross-attention and gated fusion to combine recent interactions with subsequence context.

    Source: Introduction · p. 2

  • Results

    Across three public datasets, DRIFT improves three backbones, with up to a 21.6% gain in Recall@20 and a 97.4% reduction in training time.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    DRIFT's subsequence learning and hierarchical context fusion capture both short- and long-term user preferences with linear-complexity contextual aggregation.

    Source: Conclusions · p. 7

Abstract

from arXiv · show

Long-sequence generative recommendation methods autoregressively model the user's interaction sequence to generate the next-item representation. Existing methods generally fall into two categories: efficient full-sequence modeling and target-aware context retrieval. Our experiments reveal that as the sequence length increases, the former incurs steadily growing computational cost while its accuracy gains quickly saturate and even degrade due to noise; the latter, though shortening the input sequence, is susceptible to noise that is semantically consistent yet preference-inconsistent, as well as to incomplete contexts. Both paradigms ignore the dynamic changes of user preferences and the cross-subsequence dependencies when handling historical information, thereby limiting accuracy and efficiency. To address these issues, we propose a preference-drift-aware subsequence learning and hierarchical context fusion for long-sequence generative recommendation. Specifically, we learn differentiable soft subsequence boundaries using multidimensional preference-drift information and aggregate items within each subsequence into preference-coherent representations via linear attention with soft assignment weights, thereby circumventing the expense of full-sequence attention. A cross-attention mechanism is then employed to capture dependencies between recent interactions and relevant subsequence contexts, mitigating noise in learning recent-item representations. Finally, a gated fusion mechanism adaptively combines the recent-item representation with the global subsequence context, allowing the resulting target representation to encode both recent and long-term preferences. Extensive experiments demonstrate that our method consistently outperforms existing baselines in both recommendation accuracy and computational efficiency.

Introduction

Long-sequence generative recommendation faces an accuracy–efficiency tradeoff: full-history methods become more costly as histories grow, while retrieval can be hurt by preference-inconsistent noise and incomplete context. DRIFT addresses these challenges by segmenting histories according to preference drift and fusing recent interactions with subsequence context.

Introduction

  • Prior long-sequence generative recommendation methods either optimize full-sequence modeling or shorten inputs through target-aware retrieval.Examples include efficient interaction operators and attention mechanisms, interest compression, and semantic item retrieval.Source: Introduction · p. 1
  • Full-sequence modeling costs grow with sequence length, while accuracy gains can saturate or become negative; retrieval performance can also decline as histories grow.Retrieval may initially benefit from larger retrieval sets, but expansion beyond a point degrades performance.Source: Introduction · pp. 1–2
  • Existing methods leave user-interest evolution and dependencies among historical subsequences insufficiently modeled.Full-sequence methods treat history homogeneously, while retrieval relies on a single similarity measure.Source: Introduction · p. 2
  • DRIFT learns differentiable soft subsequence boundaries from semantic shifts, temporal intervals, and interest distributions, then aggregates items into preference-coherent representations.Linear attention with soft membership weights supports this aggregation while filtering historical noise irrelevant to each subsequence preference.Source: Introduction · p. 2
  • Cross-attention connects recent items with relevant subsequence contexts, and gating fuses local recent context with global subsequence context.The resulting target representation captures both recent and long-term preferences.Source: Introduction · p. 2
  • Across three public datasets, DRIFT improves three backbones, with up to a 21.6% gain in Recall@20 and a 97.4% reduction in training time.Source: Introduction · p. 2

Assumption Validation

The assumption-validation experiments examine how sequence length affects efficient full-sequence models and how sequence length and retrieval size affect GLASS. Full-sequence models incur higher training costs with limited accuracy gains, while GLASS performance depends on both sequence length and retrieval size.

Assumption Validation

  • When sequence length grows from 50 to 1,000, HSTU training time rises by up to 6.9× and FuXi-Linear training time by up to 33.9×, while accuracy gains are limited or negative.NDCG@5 is best or near-best with short sequences on KuaiRec and nearly unchanged across lengths on Taobao MM.Source: Table 1 · p. 2, Assumption Validation · p. 2
  • At length 100, GLASS reaches Recall@5 of 0.0831 on KuaiRec and 0.0931 on Taobao MM; at length 1,000, Recall@5 falls by 7.3% and 33.7%, respectively.Source: Figure 1 · p. 2, Assumption Validation · p. 2
  • With no retrieval, GLASS Recall@5 is 0.0761 on KuaiRec and 0.0442 on Taobao MM; retrieval peaks at 0.0777 with 30 items on KuaiRec and 0.1415 with 10 items on Taobao MM.On Taobao MM, larger retrieval sizes bring Recall@5 down to 0.1344–0.1349.Source: Figure 1 · p. 2, Assumption Validation · p. 2

Methodology

DRIFT represents a user’s history as recent interactions and historical subsequences, using multidimensional preference drift to form preference-consistent subsequences. It learns subsequence representations, fuses recent and long-term context, and trains with recommendation and subsequence-structure losses.

Preliminaries

  • The user history is a chronological interaction sequence; each item has M tokens, and the sequence is divided into recent and historical interactions.The recent window contains the last r interactions, while the historical sequence contains the preceding interactions.Source: Preliminaries · p. 3

Preference-drift-aware Subsequence Learning

  • Semantic, temporal, and categorical drift signals produce boundary scores that adapt the number and soft coordinates of a user’s subsequences.Boundary scores are aggregated into user-level statistics, and the estimated subsequence count grows with boundary density, drift-point concentration, and history length.Source: Preference-drift-aware Subsequence… · pp. 3–4
  • Soft membership weights restrict each subsequence’s linear attention to its own members, supporting preference-coherent representations while avoiding cross-subsequence interference.The method sharpens assignments with Gumbel-Softmax to mitigate over-smoothing while preserving end-to-end differentiability.Source: Preference-drift-aware Subsequence… · p. 4
  • Cross-attention lets recent items select preference-consistent subsequence contexts, then decoder-level gated fusion combines recent and global subsequence context.The gate assigns more weight to recent context at smaller values and to global subsequence context at larger values.Source: Hierarchical Context Fusion · p. 4

Model Training

  • Training combines each paradigm’s task loss with a generic subsequence-structure constraint; for GLASS, task objectives include SID generation and recent- and subsequence-recommendation losses.The structure loss encourages semantic similarity, shorter time gaps, and category similarity within subsequences.Source: Model Training · pp. 4–5

Time Complexity

  • With K,r ≪ L, complexity is approximately O(LKd), scaling linearly with historical sequence length L.The stated complexity includes terms LhKd, r^2d, and rKd.Source: Time Complexity · p. 5

Experiments

Across three datasets and multiple backbones, DRIFT improves recommendation accuracy and reduces computational costs; ablations and sensitivity analyses examine its components and settings.

Performance Comparison

  • DRIFT variants outperform their backbones across all comparisons, with gains across KuaiRec, ML-20M, and Taobao MM.On KuaiRec, DRIFT-HSTU improves NDCG@20 and Recall@20 over HSTU by 13.1% and 21.6%, respectively; on ML-20M, gains reach 74.3% over TIGER and 90.4% over GLASS.Source: Performance Comparison · p. 5, Table 2 · p. 6
  • DRIFT-GLASS supports sequences 20× longer than GLASS while training 3.9× faster on KuaiRec and 2.7× faster on Taobao MM.DRIFT-Tiger reaches up to 80.4× training and 22.4× inference speedups over Tiger on Taobao MM; DRIFT-HSTU reduces per-epoch training time by 37.6%–42.1%.Source: Performance Comparison · pp. 5–6, Table 3 · p. 6

Ablation Experiments

  • DRIFT achieves Recall@5 scores of 0.0881 on KuaiRec and 0.0577 on ML-20M, outperforming uniform and K-means segmentation variants.All subsequence-compression methods outperform retrieval-based GLASS; DRIFT learns soft boundaries from preference-drift signals.Source: Ablation Experiments · p. 6, Figure 3 · p. 6
  • Removing either recent or global cross-attention degrades performance, while naive concatenation causes Taobao MM N@5 to fall to approximately 10^-3 or lower.Gated fusion achieves the best performance on both datasets, combining recent-item and subsequence representations.Source: Ablation Experiments · pp. 6–7, Table 4 · p. 6

Sensitivity Analysis

  • DRIFT-TIGER performs best at λrec=0.1, λsub-rec≈0.3, and λsub≈0.02; DRIFT-GLASS favors approximately 0.5, 0.1, and 0.1, respectively.All three losses are beneficial; removing Lrec causes the largest performance drop, while removing either other loss also reduces Recall@5.Source: Sensitivity Analysis · p. 7, Figure 4 · p. 7
  • Both datasets peak at 20 subsequences; increasing sequence length improves KuaiRec Recall@5 while Taobao MM performance remains stable.Too few subsequences may mix distinct preferences, whereas too many may fragment coherent behaviors.Source: Sensitivity Analysis · p. 7, Figure 5 · p. 7

Related Work

Generative recommendation research has developed SID-based approaches and efficiency-oriented generation methods, while long-sequence methods balance broad history coverage against computation or information loss.

Generative Recommendation

  • SID-based recommenders evolved from content-based item tokenization toward recommendation-aware SID learning and joint optimization.TIGER uses quantized item representations; ETEGRec and UniSID jointly optimize tokenization and recommendation, while DIGER propagates recommendation gradients to the codebook.Source: Related Work · p. 7
  • SE-TRec and RPG replace ordered SIDs with unordered tokens to enable parallel prediction and constrained decoding.Despite these advances, the methods described largely rely on dense self-attention over histories.Source: Related Work · p. 7

Long-sequence Generative Recommendation

  • Long-sequence methods either retain broad behavioral coverage with redundant computation or reduce context efficiently while risking information loss.Full-sequence approaches process irrelevant or redundant events; context-reduction methods may discard infrequent decisive behaviors, and GLASS retrieval can exclude useful evidence when coarse predictions are inaccurate.Source: Related Work · p. 7

Conclusions

DRIFT combines preference-drift-aware subsequence learning with hierarchical context fusion to address the accuracy–efficiency tradeoff in long-sequence generative recommendation. It captures short- and long-term preferences and outperforms existing baselines in accuracy and computational efficiency.

Conclusions

  • DRIFT adaptively segments user histories into preference-consistent subsequences using multidimensional preference-drift information.The method supports contextual aggregation and noise filtering with linear computational complexity.Source: Conclusions · p. 7
  • Cross-attention enriches recent-interaction representations, while gated fusion combines local and global subsequence information to capture short- and long-term preferences.Source: Conclusions · p. 7
  • DRIFT outperforms existing baselines in both computational efficiency and recommendation accuracy.Source: Conclusions · p. 7

A Dataset Description

The evaluation uses three public datasets spanning different recommendation domains, scales, interaction densities, and sequence lengths. Dataset-specific preprocessing and semantic identifier construction prepare them for evaluation.

A Dataset Description

  • Taobao MM is highly sparse (99.89%), with over 870K items, an average of 905 interactions per user, and multimodal item features.Its scale and long behavior sequences provide an industrial-scale multimodal evaluation setting.Source: A Dataset Description · p. 9
  • KuaiRec is relatively dense (83.72% sparsity), with a small item catalog and 1,746 interactions per user on average.The dataset is suited to evaluating long-range sequential modeling.Source: A Dataset Description · p. 9
  • ML-20M is a medium-scale, sparse benchmark (99.46% sparsity) with moderate sequence lengths.It is a standard benchmark for sequential recommendation.Source: A Dataset Description · p. 9
  • Interactions are chronologically ordered; leave-one-out assigns the last event to testing and the penultimate event to validation.Users with fewer than five interactions are removed, histories are truncated to a predefined maximum, and Taobao MM also has a minimum history length.Source: A Dataset Description · p. 9
  • Items receive three-level hierarchical semantic identifiers through residual quantization of dense representations.KuaiRec and ML-20M use pretrained text embeddings, while Taobao MM uses provided pretrained multimodal item embeddings.Source: A Dataset Description · p. 9

B.1 Experiment Setting

The experiments configure full-sequence models on KuaiRec and Taobao MM across several input lengths and evaluate GLASS under varying history lengths and retrieval sizes.

B.1 Experiment Setting

  • HSTU and FuXi-Linear are evaluated on KuaiRec and Taobao MM at sequence lengths 100, 500, and 1,000.Both use one layer and a 96-dimensional hidden representation; HSTU uses one attention head, while FuXi-Linear combines temporal, attention, and positional channels.Source: §B.1 · p. 9
  • The full-sequence models train for up to 100 epochs with AdamW, early stopping, 500 sampled negatives, and softmax temperature 0.05.The best checkpoint is selected by validation NDCG@20; experiments use random seed 2025.Source: §B.1 · p. 9
  • GLASS is tested with history lengths 100, 500, and 1,000 and retrieval sizes 0, 10, 30, 50, 70, and 90.At retrieval size zero, the retrieval branch is disabled and GLASS is conditioned solely on user history.Source: §B.1 · p. 9
  • GLASS uses a four-layer encoder and decoder, hidden size 96, and autoregressive beam search with width 20.It trains with AdamW at learning rate 1 × 10^-4 for up to 100 epochs, with validation NDCG@20 early stopping.Source: §B.1 · p. 9

C Interest Drift Analysis in Long Behavior Sequences

Taobao MM behavior sequences show local semantic coherence that weakens over longer temporal spans, alongside differences between adjacent and non-adjacent subsequences. Target items are more aligned with recent than earlier interactions.

C Interest Drift Analysis in Long Behavior Sequences

D Model Training

Training combines a recommendation objective specific to the underlying paradigm with a subsequence-structure loss. For DRIFT-GLASS, recommendation supervision includes SID generation and two sequence-level objectives, while the structure loss encourages coherent subsequences.

D Model Training

D.1 Recommendation Loss

  • DRIFT-GLASS combines SID generation loss with recent-context and subsequence recommendation objectives for sequence-level supervision.The two auxiliary objectives use in-batch negatives; the subsequence objective encourages the subsequence branch to predict the target independently.Source: §D.1 · pp. 10–11

D.2 Subsequence Structure Loss

  • The structure loss encourages semantic, temporal, and categorical coherence within subsequences through differentiable soft assignments.Its gradients reach both the boundary predictor and the user-adaptive subsequence budget without boundary annotations.Source: §D.2 · p. 11

E.1 Implementation Details.

The implementation uses standard ranking metrics and distinct configurations for sequence-modeling and SID-based generative baselines. DRIFT-GLASS is robust to the tested temporal and category constraint weights, with the default setting near-optimal across datasets.

E.1 Implementation Details.

  • Evaluation uses Recall@M and NDCG@M for M ∈ {5, 20}, with hidden dimension d = 96 and training on four NVIDIA A100 GPUs.Source: §E.1 · p. 11
  • Sequence-modeling baselines use up to 1,000 recent interactions on KuaiRec and Taobao-MM, while ML20M uses 200.The listed baselines include GRU4Rec, SASRec, HSTU, and FuXi-Linear.Source: Sequence Modeling Baselines · p. 11
  • SID-based generative baselines use a T5-style encoder–decoder, three-level SIDs, beam size 20, and a fixed context length of 50 item tokens.Their retrieval range and recent-window size vary by dataset.Source: SID-based Generative Recommendations · p. 12
  • DRIFT hyperparameter searches cover K ∈ {10, 20, 30} and specified grids for the structure, coherence, and recommendation-loss weights.The in-batch recommendation losses share temperature τrec = 0.07.Source: Hyperparameters of Our Method · p. 12

E.2 Impact of Subsequence Constraint Weights

  • Recall@20 varies within [0.147, 0.154] on KuaiRec when tuning λt, while TaobaoMM variation is no greater than 0.003.The default λt = λc = 0.5 achieves near-optimal performance across datasets; disabling either constraint degrades performance.Source: §E.2 · p. 12
Loading 2609.12556v1…