Source-linked AI summary
SORT: A Systematically Optimized Ranking Transformer for Industrial-scale Recommenders
Chunqi Wang, Bingchao Wu, Taotian Pang, Jiahao Wang, Jie Yang, Jia Liu, Hao Zhang, Hai Zhu, Lei Shen, Shizhun Wang, Bing Wang, Xiaoyi Zeng
TL;DR
Industrial-scale ranking Transformers face high feature sparsity and low label density, challenging robust learning. SORT combines request-centric data, attention and architecture optimizations, and generative pre-training. It reports business gains and improved serving efficiency in AliExpress A/B testing, alongside scalability across data, model, and sequence sizes.
Problem
Industrial-scale ranking models have not widely adopted Transformers, and high feature sparsity with low label density makes robust representation learning difficult.
Source: §1 · p. 2Method
Results
+7.47% in orders, +6.67% in buyers, and +8.65% in GMV, with latency reduced by 62% and throughput boosted by 589% in AliExpress testing.
Source: §1 · p. 2, §5 · p. 7, §4.6 · p. 7Takeaways & Limitations
SORT is fully deployed on AliExpress and reports scalability across data size, model size, and sequence length.
Source: §5 · p. 7
Abstract
from arXiv · showhide
While Transformers have achieved remarkable success in LLMs through superior scalability, their application in industrial-scale ranking models remains nascent, hindered by the challenges of high feature sparsity and low label density. In this paper, we propose SORT (Systematically Optimized Ranking Transformer), a scalable model designed to bridge the gap between Transformers and industrial-scale ranking models. We address the high feature sparsity and low label density challenges through a series of optimizations, including request-centric sample organization, local attention, query pruning, and generative pre-training. Furthermore, we introduce a suite of refinements to the tokenization, multi-head attention (MHA), and feed-forward network (FFN) modules, which collectively stabilize the training process and enlarge the model capacity. To maximize hardware efficiency, we optimize our training system to elevate the model FLOPs utilization (MFU) to 45%. Extensive experiments demonstrate that SORT outperforms strong baselines and exhibits excellent scalability across data size, model size, and sequence length, while remaining flexible at integrating diverse features. Finally, online A/B testing in large-scale e-commerce scenarios confirms that SORT achieves significant gains in key business metrics, including orders (+7.47%), buyers (+6.67%), and GMV (+8.65%), while simultaneously cutting latency by 62% and boosting throughput nearly sevenfold (+589%). SORT has been fully deployed in production, serving all users on AliExpress.
1 Introduction
Industrial ranking has not yet fully adopted Transformers, in part because sparse features and labels make robust learning difficult. SORT combines data, attention, pre-training, and architectural optimizations; experiments report strong baseline performance and substantial AliExpress business gains alongside faster serving.
1 Introduction
- High feature sparsity and low label density challenge Transformers in ranking by making robust representation learning difficult and contributing to overfitting and wasted computation.Ranking models face billion-scale item vocabularies, unsupervised historical sequences, and sparse binary labels.Source: §1 · p. 2
- SORT addresses these challenges with request-centric organization, local attention, query pruning, generative pre-training, and refinements to tokenization, attention, and feed-forward layers.Generative training adds next-item supervision, while architectural refinements include special tokens, QKNorm, attention gating, and sparse MoE.Source: §1 · p. 2
- Experiments report gains over strong baselines and evaluate SORT’s scalability across data size, model size, and sequence length.The supplied passage says SORT exhibits excellent scalability across these axes.Source: §1 · p. 2
- Across multiple AliExpress online A/B-testing scenarios, SORT yielded +7.47% in orders, +6.67% in buyers, and +8.65% in GMV, while cutting latency by 62% and boosting throughput by 589%.Source: §1 · p. 2
2 Related Work
Prior recommendation models use specialized architectures for sequence or heterogeneous-feature interactions, while newer work explores scalable Transformer-style ranking. SORT positions itself closer to unified, scalable architectures and uses data-driven rather than rule-based routing.
2 Related Work
- Traditional DLRMs use specialized feature engineering and sparse-embedding memorization, while SORT shifts toward the unified, scalable architecture end of the ranking-model spectrum.Prior work reports that massive embedding tables can impair generalization and lead to overfitting [20, 29].Source: §2 · p. 2
- HSTU[25] treats ranking as next-token prediction over a unified sequence, while Wukong[26], RankMixer[31], and OneTrans[28] address scalable or heterogeneous-feature modeling in different ways.SORT avoids OneTrans’s rule-based routing MoE in favor of data-driven routing.Source: §2 · p. 2
- GPSD[20] combines generative pre-training with sparse-embedding freezing to address sparse item-ID features and one-epoch overfitting in Transformer ranking.SORT incorporates GPSD and reports testing its effectiveness.Source: §2 · p. 2
3.1 Preliminaries
The paper frames ranking as multi-objective prediction of user behaviors. Its request-centric data representation groups candidates with the history and profile shared across a request.
3.1.1 Task.
- Ranking is formulated as multi-objective binary classification of click, cart, and purchase probabilities, with AUC used to measure performance.The model predicts these objectives for the candidate items.Source: §3.1.1 · p. 2
3.1.2 Data.
- A request-centric sample groups a candidate set with request-invariant user history and profile features, rather than repeating those features in one sample per candidate.The history is chronological, and candidate items include metadata, statistical features, and user-item cross features.Source: §3.1.2 · p. 3
3.2 Model
SORT adapts a Transformer for industrial ranking through tokenization, attention, and feed-forward refinements, alongside a multi-objective ranking head and sparse-embedding strategies. The model also uses pre-training and embedding choices to address sparse recommendation features.
3.2 Model
- SORT builds on the Transformer architecture and incorporates practices including RMSNorm pre-normalization, RoPE, and SwishGLU activations.The paper describes the model as systematically optimized for industrial ranking.Source: §3.2 · p. 3
3.2.1 Tokenization.
- Each history and candidate item becomes one token, while the user profile maps to multiple tokens; BOS and SEP tokens mark sequence boundaries and act as attention sinks.The authors report that adding the special tokens significantly enhances performance.Source: §3.2.1 · pp. 3–4
3.2.2 Multi-head Attention.
- Attention uses RoPE, candidate-isolating masks, QKNorm, and a gate; sparse local attention and progressive query pruning reduce computation on long histories.Local attention reduces complexity from O(L^2) to O(W·L); pruning keeps at most 128 non-candidate tokens in the final layer and almost halves computational cost.Source: §3.2.2 · p. 4
3.2.3 Feed-Forward Network.
- SORT replaces the standard FFN with sparse MoE, and adopts DeepSeek-style MoE over Switch-style MoE for slightly better performance without tuning an auxiliary load-balancing loss.Sparse MoE activates selected experts to increase parameter capacity without additional computational overhead.Source: §3.2.3 · p. 4
3.2.4 Ranking Head and Loss Function.
- The ranking head applies a ReLU hidden layer and sigmoid output to candidate-token representations, with binary cross-entropy losses summed across objectives using weights.Source: §3.2.4 · p. 4
3.2.5 Sparse Embedding Strategy.
- To address sparse-embedding overfitting, SORT omits explicit user IDs and uses user histories and profiles; item embeddings receive generative pre-training through next-item supervision.The paper investigates both fine-tuning transferred embeddings and transferring then freezing them during ranking training.Source: §3.2.5 · p. 4
3.3 Infrastructure
SORT's infrastructure targets both training and inference bottlenecks with specialized sparse and dense computation optimizations. These changes improve hardware utilization and serving efficiency.
3.3 Infrastructure
- MFU rises from 13% to 22% for base-scale models and reaches 45% for the large model.Training optimizations include multi-process-group communication for sparse embeddings and block-wise sparse attention with mask preloading for dense computation.Source: §3.3 · p. 4
- Table 2 defines effective intermediate size for SORT as activated experts per token multiplied by each expert's intermediate size.HSTU's intermediate size is omitted because it has no FFN; HSTU is widened to match other models' parameter counts and FLOPs.Source: Table 2 · p. 5
- Inference optimizations increase throughput by 29.4% and reduce latency by 29.3% over the unoptimized baseline.They include sparse attention kernels, operator fusion, half precision, KV caching, and multi-context/multi-stream execution.Source: §3.3 · pp. 4–5
4 Experiments
Experiments evaluate SORT using industrial interaction logs and strong baselines, then test generative pre-training and key hyperparameters. Sparse transfer with frozen embeddings improves performance, while local-attention and MoE settings are explored for their effects.
4.1.1 Dataset.
- The training set comes from recent user logs of an industrial recommender system, with the subsequent day reserved for evaluation.Dataset statistics are listed in Table 1.Source: §4.1.1 · p. 5
4.1.2 Baseline.
- Comparisons use Transformer, HSTU, and OneTrans baselines, with model sizes set for fair comparison.The standard Transformer baseline lacks SORT's local attention, query pruning, MoE, attention gate, and QKNorm.Source: §4.1.2 · p. 5
4.1.3 Training Settings.
- Training uses AdamW, batch size 12K, one epoch and a 1K maximum history by default, with BF16 except FP32 ranking heads and MoE routers.Models are trained on 4 to 32 A100 GPUs for 1 to 10 days, depending on model size.Source: §4.1.3 · p. 5
4.2 Effectiveness of Generative Pre-training
- Sparse transfer alone yields no improvement and slightly degrades performance, whereas adding sparse freeze significantly boosts performance and supports smooth multi-epoch CTR AUC growth.The training and validation AUC curves both grow smoothly across multiple epochs; the authors adopt sparse transfer plus sparse freeze as the default strategy.Source: Table 3 · p. 5, §4.2 · p. 5, Figure 2 · p. 5
4.3 Hyperparameter Exploration
- The hyperparameter exploration examines local-attention window size and MoE sparsity ratio.Figure 3 presents the impacts of the explored options.Source: §4.3 · p. 5, Figure 3 · p. 5
4.3.1 Local Attention Window Size.
- A local-attention window of 256 performs best among 64, 128, and 256, even surpassing vanilla causal attention.The authors suggest constrained windows filter distant noise and focus attention on semantically relevant local context.Source: §4.3.1 · p. 6
4.3 Hyperparameter Exploration
- DeepSeek-style MoE is more effective than Switch-style MoE, avoids hyperparameter tuning, and performs best at a sparsity ratio of 1/8.The sparsity ratio controls MoE sparsity; a dense model without MoE corresponds to a ratio of 1.Source: §4.3.1 · p. 6
4.4 Architectural Ablation and Comparison
SORT’s architectural optimizations improve ranking performance over Transformer and specialized-model baselines, while its performance also scales with data, model size, and sequence length. The experiments identify data scaling as the strongest improvement under a fixed compute budget.
4.4 Architectural Ablation and Comparison
- Special tokens and query pruning improve base-scale CTR-AUC by +0.33pt and +0.26pt, while QKNorm stabilizes attention weights and local attention matters more at longer sequences.At sequence length 1K, local attention reduces FLOPs from 43G to 40G.Source: §4.4 · p. 6, Figure 4 · p. 6
- Across small, base, and large scales, SORT consistently maintains a significant advantage over Transformer; at large scale it improves CTR-AUC by +0.51pt with 188G versus 322G FLOPs.Across all scales, SORT’s training AUC curves stay above Transformer’s.Source: §4.4 · p. 6
4.5 Data, Model and Sequence Length Scaling
- The scaling experiments vary training epochs and scenario data, model size, and sequence length; generative pre-training and frozen embeddings help avoid overfitting across epochs.The tested sequence lengths are 256, 512, 1K, 2K, and 4K.Source: §4.5 · p. 6
- Increasing data size, model size, or sequence length consistently improves performance, with data scaling producing the steepest gains and receiving highest priority under a fixed compute budget.The recommended priority is data scaling, then sequence-length scaling, then model scaling.Source: §4.5 · p. 6
4.6 Online A/B Testing
Online tests across four AliExpress recommendation scenarios report gains in orders, buyers, and GMV alongside lower latency and higher throughput. The deployed SORT model uses a base-scale configuration trained on three months of recent data, compared with a production baseline trained on over a year of data.
4.6 Online A/B Testing
- +7.47% orders, +6.67% buyers, and +8.65% GMV on average across the online evaluation.The tests cover HomePage, DetailPage, ShoppingCart, and AfterPayment; Table 5 reports results over one month.Source: §4.6 · pp. 6–7, Table 5 · p. 7
- Latency falls by 62% and throughput rises by 589% on average across four scenarios.The paper attributes the efficiency gains to request-centric reuse, a unified Transformer architecture, and an optimized inference system.Source: Table 5 · p. 7, §4.6 · p. 7
- The production baseline uses over a year of data, whereas base-scale SORT is trained on only the most recent three months.Source: §4.6 · p. 7
5 Conclusion
SORT combines ranking-focused optimizations in a hardware-efficient Transformer and is reported to outperform strong baselines while scaling across data, model size, and sequence length. AliExpress deployment delivers business gains alongside improved serving efficiency.
5 Conclusion
- SORT combines request-centric sample organization, local attention with query pruning, generative pre-training, and architectural refinements for high feature sparsity and low label density.Source: §5 · p. 7
- SORT scales across data, model size, and sequence length, with data scaling yielding the steepest gains under a fixed compute budget.Source: §5 · p. 7
- In AliExpress production, SORT delivers +8.65% GMV while cutting latency by 62% and boosting throughput nearly sevenfold.Source: §5 · p. 7
6 GenAI Usage Disclosure
The authors disclose using generative AI tools for grammar editing and polishing only.
6 GenAI Usage Disclosure
- Qwen and Gemini were used only for grammatical editing and polishing; the authors attribute all content, ideas, and data analysis to their original work.Source: §6 · p. 8