Source-linked AI summary
LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders
Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, Xionghang Xie, Shiru Ren, Xiang Sun, Yaocheng Tan, Peng Xu, Yuchao Zheng, Di Wu
TL;DR
Industrial recommenders need to model ultra-long user behavior sequences, but existing approaches commonly use indirect strategies with computational limitations. LONGER combines global tokens, token merging with InnerTrans, hybrid attention, and GPU-oriented training and serving optimizations for end-to-end scaling. Experiments and online A/B tests across advertising and e-commerce validate its effectiveness at billion-user industrial scale.
Problem
Existing industrial approaches to ultra-long behavior sequences rely mainly on two-stage retrieval or indirect modeling, limiting end-to-end long-sequence modeling under computational constraints.
Method
LONGER uses global tokens, InnerTrans-based token merging, hybrid attention, and synchronized GPU training-serving optimizations for efficient end-to-end sequence modeling.
Results
LONGER’s industrial billion-scale experiments and online A/B tests across advertising and e-commerce validate its effectiveness and generalizability at billion-user scale.
Takeaways & Limitations
LONGER enables ultra-long sequence modeling under real-world industrial constraints and has been deployed across large-scale recommendation scenarios.
Takeaways & Limitations
Performance gains from additional computation must be balanced against real-world computational and memory constraints.
Abstract
from arXiv · showhide
Modeling ultra-long user behavior sequences is critical for capturing both long- and short-term preferences in industrial recommender systems. Existing solutions typically rely on two-stage retrieval or indirect modeling paradigms, incuring upstream-downstream inconsistency and computational inefficiency. In this paper, we present LONGER, a Long-sequence Optimized traNsformer for GPU-Efficient Recommenders. LONGER incorporates (i) a global token mechanism for stabilizing attention over long contexts, (ii) a token merge module with lightweight InnerTransformers and hybrid attention strategy to reduce quadratic complexity, and (iii) a series of engineering optimizations, including training with mixed-precision and activation recomputation, KV cache serving, and the fully synchronous model training and serving framework for unified GPU-based dense and sparse parameter updates. LONGER consistently outperforms strong baselines in both offline metrics and online A/B testing in both advertising and e-commerce services at ByteDance, validating its consistent effectiveness and industrial-level scaling laws. Currently, LONGER has been fully deployed at more than 10 influential scenarios at ByteDance, serving billion users.
1 Introduction
Ultra-long behavior sequences capture both long- and short-term preferences, but industrial systems commonly use indirect strategies because fully modeling them is computationally difficult. LONGER proposes end-to-end sequence modeling with token compression, hybrid attention, and GPU-oriented optimizations.
- Motivation: Ultra-long sequences capture long- and short-term preferences, while fully modeling lengths above 10^3 can improve recommendation accuracy and diversity.The introduction also connects this modeling goal with mitigating information cocoon effects.
- Existing approaches: Industrial practice commonly relies on two-stage retrieval, pre-trained user embeddings, or memory-augmented models for long-sequence modeling.These approaches select relevant items, transfer condensed representations, or store sequence information in memory structures.
- Motivation: Recent recommender architectures and improved GPU infrastructure motivate scaling end-to-end modeling to ultra-long sequences.The introduction cites HSTU and Wukong as examples of scaling-oriented recommendation research.
- LONGER: LONGER combines global tokens, InnerTransformer-based token merging, hybrid attention, and system optimizations for industrial deployment.Its engineering stack includes mixed precision, activation recomputation, synchronous training and serving, and KV-cache serving.
- Contributions: 10,000 is the sequence length LONGER scales to end-to-end in industry, while token merge and hybrid attention reduce ~50% FLOPs with nearly lossless performance.The framework is presented as GPU-efficient and validated through industrial experiments and online testing.
- Evaluation: Offline billion-scale experiments and online A/B tests in two Douyin business scenarios validate LONGER’s performance, with development extending to dozens of ByteDance scenarios.The paper reports deployment affecting billions of users.
2 Related Work
Related work spans short-sequence interaction models, long-sequence strategies based on retrieval, transferred embeddings, or memory, and newer efforts toward direct long-sequence modeling. LONGER is positioned within this shift toward scalable direct modeling.
- Traditional modeling: Industrial recommendation systems commonly combine sequence modeling with feature interaction, with influential sequence models including DIN, DIEN, and CAN.Multi-domain, multi-interest, and sequence-denoising methods are also identified as related directions.
- Research direction: Sophisticated traditional structures were largely developed for short sequences, while long-sequence modeling later attracted increasing research attention.This transition motivates direct approaches such as LONGER.
- Long-sequence modeling: Long-sequence methods are categorized into two-stage retrieval, pre-trained user embeddings, and memory-augmented models.Retrieval and pre-trained approaches are described as two-stage strategies, while memory methods accumulate information in memory slots over training.
3.1 Problem Statement
The recommendation task uses a user’s historical behavior, user features, and a target item to predict whether an interaction occurs. The model learns this prediction from historical interaction data using binary cross-entropy.
- Task definition: Historical interaction data are represented as tuples containing the behavior sequence, user features, target item, and interaction label.The model learns the mapping from these tuples to the prediction.
- Prediction: The recommendation model produces the predicted probability as ŷ = f_θ(S_u, v).This prediction is used for the binary interaction outcome.
3.2 Overall Framework
LONGER combines global anchor tokens, token merging, hybrid attention, and system-level optimizations to model long and complex behavior sequences efficiently. The resulting system targets scalable deployment in large-scale recommendation scenarios.
- Overall framework: LONGER integrates input generation, token merge, hybrid attention, and training-serving optimizations into one long-sequence architecture.The framework is designed to maintain training and inference efficiency at industrial scale.
- Global Tokens: Global Tokens aggregate anchor representations such as target-item and user-ID embeddings to fuse global information and stabilize attention distributions.They are introduced before sequence compression and attention processing.
- Token Merge and attention: Token Merge compresses long behavior sequences, while InnerTrans preserves intra-group dependencies within merged token segments.The core architecture uses hybrid attention combining cross-causal and other attention components.
- System optimization: Synchronous training and serving unify dense and sparse parameter storage on GPU clusters, while mixed precision and recomputation improve memory and compute efficiency.These optimizations support large-scale production training and deployment.
- Deployment: LONGER supports expressive and efficient long-sequence modeling that can be deployed in large-scale real-world recommendation scenarios.This conclusion summarizes the intended system-level outcome of the integrated design.
3.3 Global Tokens
LONGER appends global tokens that aggregate information across the full behavior sequence and anchor interactions among history, context, and candidate items. These tokens also stabilize attention dynamics in long sequences, especially with sparse attention.
- Global token design: Global Tokens are auxiliary representations appended to the input sequence to extract and anchor global information.They may include target item representation tokens, learnable CLS tokens, UID embeddings, and compressed user–item interaction features.
- Global token design: Global tokens have a full attention receptive field, aggregating signals from the entire sequence while influencing all other sequence tokens.
- Global token functions: Global tokens centralize interactions among user history, contextual attributes, and candidate items.
- Global token functions: Global tokens stabilize attention dynamics in long sequences, particularly under sparse attention configurations.The passage connects this role to alleviating the attention sink effect in deeper attention layers.
3.4 Token Merge
Token Merge compresses long behavior sequences by grouping adjacent tokens, reducing quadratic attention cost while preserving local semantics. Lightweight InnerTrans blocks add within-group interactions to retain fine-grained information.
- Motivation: For industrial sequences with L typically at least 2000, vanilla Transformer attention costs O(L^2d), making long-context processing prohibitive.The passage gives L = 2000 and d = 32 as typical industrial recommender values.
- Token Merge: Token Merge groups adjacent tokens and compresses them into shorter sequences, reducing sequence length by a factor of K.This spatial compression trades model efficiency against representational fidelity.
- Computational complexity: With K = 4, merging reduces FLOPs from approximately 587M for a vanilla Transformer to 336M, a 42.8% reduction.
- Computational complexity: Token merging shortens sequences while increasing merge parameters, improving efficiency and model expressiveness.
- InnerTrans: InnerTrans applies a lightweight transformer within each token group so local interactions are captured without the information loss of direct concatenation.Its computation budget remains limited because both group dimension and sequence length are very small.
3.5 LONGER Model Structure
LONGER combines global and sampled sequence tokens with hybrid causal attention: cross-attention first compresses long inputs, followed by stacked self-attention for higher-order interactions. Sampling recent sequence tokens retains most performance improvement while reducing computation.
- Input generation: The model concatenates global tokens with sequence tokens, using global representations and behavior representations as its two input components.The input representations are partitioned into G for global tokens and H for sequence tokens.
- Input generation: The query matrix O combines all m global tokens with k sampled sequence tokens selected from the full sequence.Experiments compare recent, uniformly sampled, and learnable tokens; recent k performs best.
- Input generation: Sampling 40% of the full sequence retains over 95% of the performance improvement while reducing approximately 50% of FLOPs.
- Cross-causal attention: The first layer uses cross-causal attention over the compressed query and full input representations, followed by an FFN.Its causal mask preserves temporal relevance and supports KV Cache Serving by preventing sequence information from being visible to the candidate item.
- Self-causal attention: Subsequent self-causal attention blocks model internal dependencies among sampled behavior tokens and are stacked N times with FFNs.The resulting compressed output is used for downstream prediction.
- Model structure: The architecture combines cross-attention to compress long sequences with subsequent self-attention to learn high-order interactions.
3.6 Training and Deployment Optimization
LONGER uses synchronized GPU training, memory-efficient optimization, and KV caching to support industrial-scale training and serving. KV caching reuses user-sequence projections across candidates, reducing throughput degradation from −40% to −6.8%.
- Training framework: The training framework synchronously updates dense and sparse parameters across GPU runners for distributed large-scale model training.It uses batch or streaming ingestion, Fountain preprocessing, and coordinated updates across devices and nodes.
- Training framework: Dense and sparse parameters share unified GPU storage and updates, eliminating external Parameter Server components.A hierarchical memory system places sparse embeddings across GPU HBM, CPU memory, and other memory tiers according to feature frequency.
- Training optimization: Mixed precision and activation recomputation reduce training resource demands, with mixed precision delivering +18% throughput, −16% training time, and −18% memory usage on average.Dense layers achieve up to −28% memory reduction.
- KV Cache Serving: KV Cache Serving precomputes and caches user-sequence key and value projections, then computes only candidate-specific attention for each global token.This creates a two-stage inference process that reuses the same user-sequence representation across candidates.
- KV Cache Serving: Throughput degradation falls from as high as −40% to only −6.8% with KV Cache Serving.The optimization avoids redundant computation when scoring multiple candidates.
4 Experiments
Experiments evaluate LONGER on billion-scale industrial recommendation data through offline comparisons, ablations, scaling analyses, and online A/B tests. LONGER improves recommendation metrics while reducing computational cost, with gains across advertising and e-commerce scenarios.
- Experimental setup: The Douyin Ads CVR dataset contains 5.2 billion samples collected over 130 consecutive days, and comparisons use a 48×A100 GPU cluster.Baselines include short-sequence methods using 50 interactions and long-sequence methods with scalability challenges.
- Offline comparison: AUC 0.85290 and LogLoss 0.47103 outperform all baselines, including a 1.57% relative AUC improvement over the base model and 0.21% over Transformer.The 0.21% improvement is reported against the most competitive model, Transformer.
- Ablation study: TokenMerge reduces FLOPs from 3.73×10^9 to 3.03×10^9 while improving AUC by 1.58% and reducing LogLoss by 3.48%; adding InnerTrans yields 0.47052 LogLoss and 1.63% AUC improvement.The ablation attributes additional gains to combining token merging with the lightweight InnerTrans component.
- Ablation study: Using 100 queries achieves AUC 0.85290 and LogLoss 0.47103, close to all 250 queries while using only 54% of the FLOPs.The results identify a practical accuracy–computation trade-off for deployment.
- Scaling analysis: Increasing sequence length consistently improves AUC and reduces LogLoss, while deeper models benefit more but eventually show diminishing returns.AUC also follows power-law trends when scaling model width and FLOPs, with R^2 = 0.987 for parameter count and R^2 = 0.967 for FLOPs.
- Online A/B tests: Online A/B tests report improvements in both Douyin Ads and Douyin E-Commerce, including Ads gains of 1.063% ADSS and 1.168% ADVV for Live Streaming.E-commerce results include 7.9222% higher Order/U and 6.5404% higher GMV/U for Live Streaming.
5 Conclusions
LONGER enables efficient, scalable end-to-end modeling of ultra-long user behavior sequences under industrial constraints. Experiments and online A/B tests validate its robustness and generalizability at billion-user scale.
- LONGER combines global tokens, token merge with InnerTrans, hybrid causal attention, and system-level GPU optimizations for ultra-long sequence modeling.The system optimizations include mixed-precision and recomputation training, KV cache serving, and a GPU-synchronous framework.
- Extensive experiments on billion-scale industrial datasets and online A/B tests across advertising and e-commerce validate LONGER’s robustness and generalizability.
- LONGER achieves competitive accuracy while significantly reducing computational overhead for latency-sensitive production deployment.
- Future work targets more efficient sequence modeling techniques and improved cross-domain behavior modeling in industry.