Source-linked AI summary
HyTRec: A Hybrid Temporal-Aware Attention Architecture for Long Behavior Sequential Recommendation
Lei Xin, Yuhao Zheng, Ke Cheng, Changjiang Jiang, Zifan Zhang, Fanhu Zeng
TL;DR
Long-sequence recommendation must balance linear efficiency against softmax attention’s retrieval precision and adapt to rapid interest drift. HyTRec combines linear attention for historical behavior, softmax attention for recent interactions, and TADN for temporal intent weighting. It maintains linear inference speed and reports over 8% Hit Rate improvement for users with ultra-long histories, while the paper identifies remaining dependence on long-range context and future generalization work.
Problem
Existing long-sequence recommenders face a trade-off between softmax attention’s computational overhead and linear attention’s limited retrieval precision, while linear models also lag behind rapid interest changes.
Method
HyTRec assigns historical behavior to a linear attention branch, recent interactions to a softmax branch, and uses TADN with temporal decay to prioritize fresh signals over historical noise.
Results
Over 8% improvement in Hit Rate is reported for users with extensive interaction histories while maintaining linear inference speed.
Takeaways & Limitations
The reported results support efficient long behavior sequence modeling with separate treatment of long-term preferences and short-term intentions.
Takeaways & Limitations
The paper identifies dependence on long-range context and proposes future work to reduce it while improving cross-domain and cross-scenario generalization.
Abstract
from arXiv · showhide
Modeling long sequences of user behaviors has emerged as a critical frontier in generative recommendation. However, existing solutions face a dilemma: linear attention mechanisms achieve efficiency at the cost of retrieval precision due to limited state capacity, while softmax attention suffers from prohibitive computational overhead. To address this challenge, we propose HyTRec, a model featuring a Hybrid Attention architecture that explicitly decouples long-term stable preferences from short-term intent spikes. By assigning massive historical sequences to a linear attention branch and reserving a specialized softmax attention branch for recent interactions, our approach restores precise retrieval capabilities within industrial-scale contexts involving ten thousand interactions. To mitigate the lag in capturing rapid interest drifts within the linear layers, we furthermore design Temporal-Aware Delta Network (TADN) to dynamically upweight fresh behavioral signals while effectively suppressing historical noise. Empirical results on industrial-scale datasets confirm the superiority that our model maintains linear inference speed and outperforms strong baselines, notably delivering over 8% improvement in Hit Rate for users with ultra-long sequences with great efficiency.
1 Introduction
Long behavior sequences support dynamic preference modeling, but existing attention mechanisms struggle to balance efficiency, retrieval precision, and rapid interest drift. HyTRec addresses these challenges with hybrid attention and temporal-aware intent modeling, while reporting improved recommendation performance at linear inference speed.
- Long behavior sequences provide rich signals for modeling users’ long-term interest trajectories and complex decision-making paths.
- Softmax attention has quadratic complexity, whereas linear attention can compromise retrieval precision and struggle with fine-grained dependencies.
- HyTRec combines predominantly linear attention with a small proportion of softmax layers to preserve near-linear complexity while restoring high-fidelity retrieval.
- TADN uses exponential gating and temporal decay to upweight fresh behavioral signals, suppress historical noise, and track rapid interest shifts.
- Over 8% improvement in Hit Rate is reported for users with extensive interaction histories while maintaining linear inference speed.
2 Related Work
Sequential recommendation has progressed from short-term transition models to self-attention, generative frameworks, and architectures designed for ultra-long sequences. This evolution reflects the need to retain expressive dependency modeling while improving scalability and reducing semantic information loss.
- Early sequential recommendation methods used Markov Chains and Recurrent Neural Networks to model short-term transitions and broader sequential regularities.
- Transformer-based models such as SASRec introduced self-attention for global dependency modeling in sequential recommendation.
- Industrial ultra-long sequence systems include SIM’s two-stage search strategy and ETA’s Locality Sensitive Hashing approach.
- Generative recommendation frameworks such as P5 unify multiple recommendation tasks, while SimUSER uses LLMs to simulate user behavior for evaluation.
- Sparse, kernel-approximated, state-space, and hybrid architectures were developed to address the scalability and semantic limitations of full or pure linear attention.
3 Preliminaries
The paper studies next-item prediction from a user’s historical interaction sequence. Its preliminary formulation represents users, items, and interactions explicitly and estimates the next-item probability conditioned on the sequence.
- Sequential recommendation predicts the ID of a user’s next purchased item from the user’s historical interaction sequence.
- Users are represented by U, items by I, and a user’s sequence is S_u = [x_1, x_2, . . . , x_n], where each x_t is an interacted item.
- The model estimates the probability distribution of the next item x_n+1 conditioned on the historical sequence S_u by maximizing P(x_n+1 | S_u).
4 HyTRec
HyTRec separates long-term historical behavior from recent interactions and processes them through specialized attention branches before fusion. Its TADN module adds temporal-aware gating to prioritize fresh signals while preserving stable preferences.
- Sequence Decomposition Strategy: HyTRec decomposes each user sequence into a fixed-length recent subsequence and a complementary long-term historical subsequence.The recent sequence captures sudden consumption intents and interest drifts, while the historical sequence captures stable consumption preferences.
- Dual-Branch Data Flow: The short-term branch applies standard multi-head self-attention, while the long-term branch uses hybrid attention to compress extensive history into a compact representation.The two branches process their inputs in parallel before their outputs are fused for final prediction.
- Hybrid Attention: The hybrid attention mechanism combines linear attention for extensive history with softmax attention for recent interactions to balance efficiency and retrieval precision.The design targets the O(n^2) complexity bottleneck while retaining global context awareness.
- Temporal-Aware Delta Networks: TADN combines temporal decay with feature similarity to generate dynamic gates that amplify behaviors correlated with the recent purchase period.The temporal decay factor τt measures relevance to the current decision, with τt ∈(0, 1] characterizing correlation strength.
- Information Fusion Mechanism: Recent interactions receive higher gating weights, allowing short-term deviations to dominate immediate-intent predictions while the static gate preserves routine long-term preferences.The fused feature matrix is then integrated into the Gated DeltaNet framework through a temporal-aware state update.
5 Experiments
Experiments evaluate HyTRec against recommendation and long-text baselines, its scalability, component contributions, attention-ratio trade-offs, and performance in sparse-history scenarios. HyTRec combines strong recommendation accuracy with efficient ultra-long-sequence processing.
- Experimental Setup: Experiments compare HyTRec with sequential-recommendation and long-text baselines across three Amazon datasets using H@500, NDCG@500, AUC, and latency.The setup aligns computational budgets and evaluates models on next-item prediction.
- Performance Comparison (RQ1): HyTRec achieves H@500 of 0.6643 and AUC of 0.8655 on Beauty, while Electronics results include H@500 of 0.3272 and AUC of 0.876.On Beauty, its H@500 exceeds the listed baseline types and its AUC ranks first; on Electronics, its AUC exceeds the other baselines.
- Training Efficiency (RQ2): 65.3 K token/sec at sequence length 5k shows HyTRec retains high throughput, while HSTU loses nearly 60% of its processing efficiency.At length 12k, HSTU reaches 8.9 K token/sec, reported as 19% of HyTRec’s throughput.
- Ablation Study (RQ3): The full model reaches H@500 0.6643, NDCG@500 0.3480, and AUC 0.8655, outperforming variants with only short-term attention or only TADN.The ablation attributes complementary roles to short-term attention for immediate changes and TADN for long-term patterns.
- Efficiency Evaluation (RQ4): A 3:1 hybrid-attention ratio provides the best balance between recommendation performance and inference efficiency among the tested ratios from 2:1 to 6:1.Efficiency is defined as performance change divided by latency change.
- Challenging Scenarios (RQ5): HyTRec achieves superior performance in new-user cold-start and silent-old-user scenarios after augmenting sparse histories with behavior from similar users.These scenarios are evaluated as typical sparse-history business cases.
6 Conclusion
The paper presents HyTRec as a model for efficient extremely long behavior sequences that integrates historical interests with short-term intentions. Its hybrid attention, temporal decay, and short-term branch are reported to improve efficiency and effectiveness.
- 6 Conclusion: HyTRec integrates historical interests and short-term intentions through a hybrid attention architecture for efficient long-term behavior modeling.The method targets extremely long sequences and user short-term interest drift.
- 6 Conclusion: The proposed temporal decay mechanism and short-term attention branch account for fresh intentions while modeling long-term interests.The conclusion reports extensive experimental evidence for better efficiency and method effectiveness.
Future Analysis
Future analysis identifies cross-domain and cross-scenario generalization, robustness to noisy sequential data, performance-efficiency trade-offs, and broader application scope as directions for further work.
- Future Analysis: Future work targets stronger cross-domain and cross-scenario generalization by optimizing the long-term encoder and temporal-aware delta network.The stated goal includes reducing dependence on long-range context information and adding a more flexible adaptive mechanism.
- Future Analysis: The paper proposes noise detection, denoising, and noise-aware sequential learning mechanisms to improve robustness against noisy sequential data.
- Future Analysis: Further work will study performance-efficiency trade-offs through lightweight structures and efficient training strategies.
- Future Analysis: The authors also plan to expand the model’s application scope.
Impact Statement
The paper positions HyTRec between softmax attention’s retrieval quality and linear attention’s efficiency for long-sequence recommendation. Its hybrid design combines sparse softmax layers with predominantly linear processing while retaining recommendation-domain mechanisms.
- Attention trade-offs: Softmax attention has quadratic time and memory complexity in sequence length, limiting scalability for long behavior sequences.It computes pairwise query-key similarities and stores an n×n attention matrix.
- Attention trade-offs: Linear attention reduces complexity to O(nd^2) by avoiding explicit n×n attention matrices, but may sacrifice retrieval fidelity.Causal implementations can also require scan or cumsum-style operations that reduce parallel efficiency.
- HyTRec positioning: Hybrid attention inserts softmax-attention layers sparsely among linear layers to retain near-linear complexity while improving retrieval over purely linear backbones.This design is the stated motivation for HyTRec.
- Related approaches: HyTRec builds on sequential recommendation architectures including GRU-, self-attention-, hierarchical-transformer-, and item-aware attention-based models.The paper also compares its positioning with long-text architectures such as Transformer, GLA, and Qwen-next.
- HyTRec positioning: HyTRec assigns linear attention to historical behavior modeling and softmax attention to recent interactions, preserving semantic integrity for long-term preferences.The framework is described as combining softmax accuracy with linear-attention efficiency.
C.2 Intermediate Process Sequence Data Processing Strategy
The processing strategy consolidates and organizes heterogeneous user behavior data to construct longer, behaviorally informative sequences. It also distinguishes data conditions and evaluates attention-head settings for performance and latency.
- Data construction: Ad attribution identifiers are used to trace user states across funnel events including clicks, activation, product interactions, and payment.The resulting behaviors are ordered by correlation strength with payment behavior to form hierarchical long-cycle sequence data.
- Data construction: Community information can supplement product behavior data on community-ecommerce platforms by extracting interests from community articles.Single e-commerce platforms are restricted to product data.
- User-sequence conditions: Short-sequence samples are common in business data, whereas long-sequence samples often correspond to active or high-value active users.The paper characterizes long-sequence users as relatively easy samples and proposes imputing missing histories or empty intervals for short-sequence samples.
- User-sequence conditions: Direct long-sequence modeling is unsuitable for cold-start users, inactive long-term users, and scalper accounts.These groups lack useful behavior sequences, contain only distant interactions, or may not contribute positively to platform revenue despite many records.
- Attention-head analysis: 2 attention heads provide the overall best balance of recommendation performance and inference efficiency across the evaluated settings.Performance rises slightly and then declines as heads increase from 1 to 8, while latency decreases and then increases.
E.2 Comparison Experiments on Different Numbers of Experts
The experiments examine expert count and hybrid-layer ratios as efficiency–accuracy design choices. Four experts and a 3:1 hybrid ratio are identified as the preferred configurations under the reported metrics and latency criteria.
- Different numbers of experts: Four experts achieve the overall optimal performance among the evaluated expert counts.The comparison uses 4, 6, and 8 experts, corresponding to four heterogeneous user groups and larger alternatives.
- Different numbers of experts: Increasing experts from 4 to 8 continuously reduces recommendation performance while significantly increasing inference latency.The reported metrics include H@500, NDCG@500, and AUC.
- Hybrid structure ratio: The 3:1 hybrid ratio provides the optimal trade-off between recommendation performance and inference latency.It improves core metrics without significant extra latency relative to the 2:1 structure.
- Hybrid structure ratio: The 2:1 ratio has the lowest reported latency at 1.0802 but leaves room for improvement in evaluation metrics.Ratios of 4:1 and 5:1 increase latency, while 6:1 reaches latency 2.5296 despite peak NDCG@500 and AUC.
- Discussion: HyTRec integrates linear and softmax attention to balance efficiency and semantic integrity in long-sequence recommendation.The discussion identifies further improvements in performance and adaptability as future directions.
F.2 Integration with Expanded Memory Architectures
HyTRec’s linear attention can overwrite information in fixed-dimensional states for extremely long sequences, motivating expanded-memory designs. Its validation is currently limited to e-commerce recommendation datasets.
- Memory capacity: For extremely long sequences such as 10k+ tokens, fixed-dimensional linear-attention states can suffer memory overwriting.This limitation constrains performance for users with extensive interaction histories.
- Memory capacity: Expanded memory architectures are proposed to retain more valuable historical information while preserving linear efficiency.The proposed integration is intended to expand model memory capacity for extensive histories.
- Validation scope: HyTRec has only been validated on e-commerce recommendation datasets, restricting its demonstrated generalizability to other generative recommendation scenarios.The paper suggests extending evaluation to content and social recommendation settings.