Source-linked AI summary
From Language to Behavior: Scaling Sequence Transformers for Industrial Recommendation Ranking with Rec-Native Designs
Jie Chen, Xiangqian Yu, Yanchao Lian, Tan Lu, Run Yang, Zhengchun Shang, Xing Wang, Cheng Chen, Ke Hu, Qiang Li, Tianjiu Yin, Xiaobing Liu
TL;DR
Production recommendation ranking lacks the signal quality and computation structure that make Transformer scaling effective in language modeling. ReST addresses these differences with rec-native sequence modeling, asymmetric encoder–decoder computation, and shared-prefix reuse. It improves scaling and online outcomes, including a 1.31% online AUC gain and an 11.93% core revenue-metric lift within a 50 ms P99 budget.
Problem
Directly scaling Transformer sequence models for production ranking is challenging because behavior data are noisy, temporally irregular, and sparsely supervised, while each request scores many candidates against one shared history.
Method
ReST combines a recommendation-native sequence encoder with an asymmetric reusable encoder and lightweight cross decoder, supported by auxiliary supervision and shared-prefix training and serving.
Results
Across industrial and public benchmarks, ReST achieves better accuracy–efficiency trade-offs and scales along sequence length, depth, and width where LLM-style blocks saturate; online AUC improves by 1.31%.
Takeaways & Limitations
Behavior-sequence scaling remains an under-exploited axis for industrial ranking and can deliver substantial online gains under practical latency budgets.
Takeaways & Limitations
The study scales the sequence-modeling component while keeping the non-sequential ranking stack and candidate-scoring protocol intact.
Abstract
from arXiv · showhide
Scaling Transformers has driven large gains in language modeling, but transplanting this to behavior-sequence modeling in production ranking is challenging: recommendation differs in signal quality, where behavior sequences are noisy, temporally irregular, and sparsely supervised, and in computation asymmetry, where each request scores many candidates against one shared user history under tight latency budgets. We propose ReST, a recommendation-native Transformer scaling framework. For signal quality, it introduces a sequence encoder with dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives. For computation asymmetry, it factorizes ranking into a heavy reusable encoder and a lightweight cross decoder with projection-free KV attention and token-specific parameterization, coupling user-level shared-prefix training with shared-prefix serving for compute-once, decode-many-times ranking. Across industrial and public benchmarks, ReST achieves higher accuracy and scales more consistently along sequence length, depth, and width, where LLM-style Transformer blocks saturate. A one-week online A/B test on a production advertising platform improves online AUC by 1.31% and lifts a core revenue metric by 11.93% within a 50 ms P99 budget; ReST has since been fully deployed in production, showing that behavior-sequence scaling remains a promising, under-exploited axis for production ranking.
1 INTRODUCTION
ReST redesigns Transformer scaling for production recommendation ranking by addressing noisy, irregular, sparsely supervised behavior sequences and one-history-to-many-candidates computation. Its rec-native encoder, asymmetric ranking architecture, and shared-prefix reuse improve scaling and online performance under latency constraints.
- Motivation: Recommendation scaling differs from language modeling because behavior sequences are noisy, temporally irregular, and sparsely supervised.Clicks and views can reflect accidental exposure or exploratory browsing; ordinal distance does not capture irregular recency; ranking labels are sparse and delayed.
- Motivation: One user history is paired with N candidate targets, making shared sequence computation amortizable but candidate-conditioned computation a repeated latency bottleneck.This asymmetry motivates allocating more capacity to reusable sequence processing while keeping per-candidate interaction lightweight.
- ReST design: ReST combines dual-gated attention, rotary positional and temporal embedding, stabilized residual normalization, and training-only auxiliary objectives for behavior-sequence modeling.The components target unreliable behavior evidence, irregular temporal structure, depth stability, and weak sequence-side supervision.
- ReST design: ReST factorizes ranking into a reusable compute-heavy sequence encoder and a lightweight cross decoder with projection-free key/value attention and token-specific parameterization.The encoder contextualizes each history once, while the decoder scores many candidates using the resulting reusable memory.
- System co-design: User-level shared-prefix training and shared-prefix serving translate the reusable encoder boundary into computation reuse during training and ranking.These mechanisms support compute-once, decode-many-times execution across samples or candidates sharing a user history.
- Empirical validation: Across industrial and public benchmarks, ReST improves accuracy–efficiency trade-offs and continues scaling with sequence length, depth, and width where LLM-style blocks saturate.The reported online deployment improves online AUC by 1.31% and a core revenue metric by 11.93% within a 50 ms P99 latency budget.
2 METHOD
ReST redesigns behavior-sequence ranking around recommendation-specific signal challenges and one-history-to-many-candidates computation. Its encoder addresses noisy, irregular, weakly supervised sequences, while auxiliary objectives and asymmetric computation support scalable ranking.
- 2.1 Problem Formulation: ReST studies CVR prediction where chronological behavior sequences are combined with non-sequential user and context features for candidate-ad ranking.
- 2.3 Rec-Native Sequence Encoder (𝑇): ReST factorizes ranking into a compute-heavy sequence encoder that builds reusable user memory and a lightweight cross decoder for candidate-specific scoring.
- 2.3 Rec-Native Sequence Encoder (𝑇): Dual-Gated Attention filters unreliable behavior values before aggregation and modulates the resulting context before updating token representations.
- 2.3 Rec-Native Sequence Encoder (𝑇): RoPE and RoTE jointly encode ordinal position and irregular physical time using attention heads with complementary temporal granularities.
- 2.3 Rec-Native Sequence Encoder (𝑇): Stabilized Residual Normalization regulates layer normalization placement and residual updates to improve depth scaling under sparse, shortcut-prone supervision.
- 2.5 Auxiliary Supervision for Sequence Scaling: Training-only auxiliary sequence CVR supervision addresses sequence starvation, increasing the sequence encoder’s average gradient norm by more than 3× with λseq = 0.1.
- 2.5 Auxiliary Supervision for Sequence Scaling: The alignment objective regularizes sequence and non-sequential user representations using training-only projection heads and a sigmoid contrastive loss.
3 EXPERIMENTS
Experiments evaluate ReST against production and public baselines, scaling behavior, component contributions, computation reuse, and online deployment. ReST consistently improves accuracy and scaling efficiency while meeting production latency constraints.
- Experimental Setup: ReST is evaluated on an anonymized industrial TikTok Shop Ads dataset and three public benchmarks, with sequence-modeling components compared under controlled pipelines.The industrial evaluation keeps the ranking stack, behavior sequence, and query tokens fixed while replacing only the sequence-modeling component.
- Main Comparisons: At comparable FLOPs, ReST achieves the best industrial AUC lift across Base and Large budgets, reaching +0.92% in the Large setting.An offline AUC Δ of 0.05% is considered practically meaningful in this production ranking setting.
- Main Comparisons: ReST achieves the best AUC and log loss across MovieLens-1M, MovieLens-20M, and Amazon-Books while maintaining comparable FLOPs.The results indicate that the observed benefits transfer beyond the large-scale industrial dataset.
- Scaling Analysis: ReST has a more favorable empirical compute-performance trend than LLaMA and HSTU, with fitted scaling exponents of 0.101 versus 0.090 and 0.088, respectively.Across the tested compute regime, ReST attains consistently higher AUC gains; axis-wise analyses also report earlier diminishing returns for LLM-style models.
- Scaling Analysis: ReST continues improving with longer contexts and deeper models, reaching approximately 0.4% additional AUC lift through 16k tokens while generic baselines show diminishing returns.LLaMA and HSTU exhibit diminishing returns beyond 8 and 16 layers, respectively, whereas ReST benefits more consistently from depth.
- Ablation Studies: Ablations show that candidate-conditioned decoding, dual-gated attention, stabilized residual learning, temporal encoding, and auxiliary objectives each improve the accuracy–efficiency trade-off.Adding the cross decoder improves AUC by +0.03% with +0.2% extra FLOPs; token-specific parameterization raises the gain to +0.05%, while output-plus-value gating reaches +0.06% at +7.5% FLOPs.
- System Efficiency: Shared-prefix reuse improves training throughput by 5.8× and reduces sequence-modeling inference cost by up to 20× by computing the encoder once per request.These gains support deployment of larger sequence models under strict production latency budgets.
- Online Validation: +1.31% online AUC and +11.93% Advertiser Value were achieved in a one-week A/B test within the 50 ms P99 latency budget.The treatment received 20% of production traffic, satisfied monitored guardrails, and was fully deployed in production.
4 RELATED WORK
Industrial ranking builds on the DLRM paradigm, combining large embedding tables with lightweight interaction modules. Prior work refines this paradigm through feature crossing, target-aware attention, and long-history behavior modeling.
- DLRMs combine large embedding tables with relatively lightweight interaction modules for industrial ranking.
- DCN-V2 strengthens explicit feature crossing within the DLRM paradigm.
- DIN introduces target-aware attention for user-behavior modeling.
- SIM and SDIM extend behavior modeling to much longer histories through retrieval-based approaches.
5 CONCLUSION
ReST is a recommendation-native framework for scaling behavior-sequence Transformers under industrial latency constraints. Its rec-native modeling, asymmetric architecture, and shared-prefix reuse support more effective scaling and full production deployment.
- ReST addresses behavior-sequence scaling through rec-native modeling, asymmetric architecture, and shared-prefix computation reuse.
- ReST scales more effectively than LLM-style blocks along behavior sequence length, depth, and width.
- ReST enables larger sequence models to achieve substantial business gains under practical latency budgets.
- The framework has been fully deployed in production.
A.1 Denoising Analysis of Dual-Gated Attention
A controlled noise-injection study evaluates whether Dual-Gated Attention remains robust when behavior evidence becomes unreliable. DGA is consistently more robust as token noise increases because it gates both value and output streams.
- The study injects Gaussian noise into behavior tokens and compares standard, output-gated, and dual-gated attention under matched protocols.
- DGA is consistently more robust as token noise increases.
- Standard attention degrades most rapidly because noisy features enter the value stream and are aggregated into the contextual representation.
- DGA suppresses corrupted behavior evidence by gating the value stream before aggregation and the output stream afterward.
A.2 Relative-Time Property of RoTE
RoTE makes temporal attention depend on relative timestamp buckets rather than their absolute shifted values. Its multi-head temporal granularities capture recency at complementary resolutions, while periodic rotary features impose important interpretive limits.
- RoTE rotates query and key vectors according to timestamp buckets so their dot product depends on relative rotation.
- The temporal head depends on the relative bucket difference b_m−b_n, not on shifted buckets separately.
- Subtracting sequence start time cancels in pairwise differences and preserves the elapsed time represented between tokens.
- RoTE provides complementary temporal resolutions through head-specific granularities ranging from seconds to years.
- RoTE does not map elapsed time monotonically or one-to-one to attention scores because rotary features are periodic and content-dependent.
A.3 Gradient Analysis for Auxiliary Sequence CVR Supervision
The analysis examines whether auxiliary sequence CVR supervision strengthens optimization of the Transformer sequence encoder, whose gradients can be weakened by the non-sequential branch.
- The main CVR objective can be partially optimized through strong non-sequential DLRM features, leaving the sequence encoder with weak gradients.
- Figure 2 measures average gradients of Q, K, and V projection matrices across all sequence-encoder attention layers during training.
- More than 3×: auxiliary sequence CVR supervision increases average gradient norm from 2.35 to 8.50 with λ_seq = 0.1.
- The same trend appears for FFN parameters, indicating stronger gradient flow throughout the sequence encoder beyond attention projections.
A.4 Implementation Details
The public-benchmark implementation uses filtered interaction datasets, chronological binary-prediction splits, and fixed Transformer and prediction-head configurations.
- Public benchmarks are used only for offline academic benchmarking, with models implemented in RecBole and trained using Adam.
- Datasets undergo 5-core filtering, retaining users and items with at least five interactions.
- Ratings of 4 or above are positive, lower ratings are negative, and chronological interactions are split 80%/10%/10% into train, validation, and test sets.
- Evaluation reports AUC and binary log loss without additional negative sampling, using observed low-rating interactions as negatives.
- Transformer models use maximum sequence length 200, embedding dimension 32, two sequence layers, two attention heads, and feed-forward dimension 128.
- The prediction head is a three-layer MLP with hidden sizes 256, 128, and 64, while dropout is 0.25 across embeddings, hidden layers, and attention weights.