Source-linked AI summary
Towards Large-scale Generative Ranking
Yanhua Huang, Yuqi Chen, Xiong Cao, Rui Yang, Mingliang Qi, Yinghao Zhu, Qingchang Han, Yaowei Liu, Zhaoyu Liu, Xuefeng Yao, Yuting Jia, Leilei Ma, Yinqi Zhang, Taoyu Zhu, Liujie Zhang, Lei Chen, Weihang Chen, Min Zhu, Ruiwen Xu, Lei Zhang
TL;DR
Generative ranking remains understudied in large-scale industrial settings. The paper analyzes its effectiveness and introduces GenRank, an architecture for efficient ranking. Experiments report improved efficiency and effectiveness, including a P99 response-time advantage over the production model with comparable overall resources.
Problem
Method
The paper analyzes generative recommendation’s sources of effectiveness and proposes GenRank, with action-oriented organization and position and time biases.
Source: §1 · pp. 1–2, §4.1 · p. 5Results
GenRank improves P99 response time by over 25% versus the production ranking model while requiring comparable overall resources.
Source: §6 · p. 7Takeaways & Limitations
Large-scale offline and online experiments demonstrate GenRank’s effectiveness and efficiency for industrial ranking.
Source: §7 · p. 7In large-scale distributed training, processing samples out of temporal order may risk information leakage; grouped samples are proposed to mitigate this risk.
Source: §3.1 · p. 3
Abstract
from arXiv · showhide
Generative recommendation has recently emerged as a promising paradigm in information retrieval. However, generative ranking systems are still understudied, particularly with respect to their effectiveness and feasibility in large-scale industrial settings. This paper investigates this topic at the ranking stage of Xiaohongshu's Explore Feed, a recommender system that serves hundreds of millions of users. Specifically, we first examine how generative ranking outperforms current industrial recommenders. Through theoretical and empirical analyses, we find that the primary improvement in effectiveness stems from the generative architecture, rather than the training paradigm. To facilitate efficient deployment of generative ranking, we introduce GenRank, a novel generative architecture for ranking. We validate the effectiveness and efficiency of our solution through online A/B experiments. The results show that GenRank achieves significant improvements in user satisfaction with nearly equivalent computational resources compared to the existing production system.
1 Introduction
Generative ranking remains understudied in large-scale industrial settings. This paper analyzes its sources of effectiveness and introduces GenRank, which online experiments show is effective and feasible.
1 Introduction
- Generative ranking remains understudied, particularly in large-scale industrial contexts.Source: §1 · p. 1
- Experiments identify the generative architecture, rather than the training paradigm, as the primary source of strong performance.Source: §1 · pp. 1–2
- Online A/B testing on Xiaohongshu’s Explore Feed demonstrates the effectiveness and feasibility of generative ranking in an industrial recommender.Source: §1 · p. 2
- GenRank combines action-oriented sequence organization with position and time bias strategies designed for industrial ranking.Source: §1 · p. 2
2 Problem Setup
The paper frames ranking as making predictions for predefined tasks on candidate items and constructs offline data from Xiaohongshu Explore Feed exposure logs.
2 Problem Setup
- Offline experiments use hundreds of billions of Explore Feed exposure logs collected over 15 days.Source: §2 · p. 2
- Inputs comprise categorical features, numerical features, and frozen embeddings from pretrained models.Source: §2 · p. 2
- An absolute increase of 0.0010 in AUC is considered significant and typically yields a 0.5% improvement in topline metrics.Source: §2 · pp. 2–3
3 Source of Effectiveness in Generative Recommendation
The section tests which parts of generative recommendation explain its effectiveness and how common ranking modules and feature engineering behave under the generative paradigm.
3 Source of Effectiveness in Generative Recommendation
- The experiments examine generative mechanisms and compare modules that are established in industrial ranking systems.Source: §3 · p. 3
3.1 Key Mechanisms in Generative Paradigm
- Including historical positions in the loss decreases AUC by more than 0.0100, while replacing the causal mask causes a drop exceeding 0.0015.The mask-related decline grows with larger model sizes; the authors conclude autoregressive interaction is critical to effectiveness.Source: §3.1 · p. 3
- Training grouped samples in point-wise order causes only a slight AUC decrease, supporting architecture—not sample organization—as the primary source of effectiveness.The proposed benefits of grouping include more stable gradient estimates and reduced risk of processing later samples before earlier ones.Source: §3.1 · pp. 3–4
3.2 Module Performance Comparison across Paradigms
- SIM, PPNet, and PLE yield comparable gains across paradigms, while content embeddings provide over twice the AUC improvement under the generative paradigm.The authors attribute the content-embedding result to alignment between generative training and downstream use.Source: §3.2 · p. 4
- Most features provide negligible benefits to generative architectures, but real-time statistical features, especially window-based ones, remain effective.Minimal feature-engineering needs and KV caching are described as supporting inference scalability as candidate sets grow.Source: §3.2 · p. 4
4 Efficient Generative Ranking in Industrial Scenarios
GenRank targets efficient large-scale ranking through an action-oriented sequence design and revised position and time biases. Its training results show a large speed-up with slightly improved test AUC.
4 Efficient Generative Ranking in Industrial Scenarios
- 94.8% training speed-up over HSTU, with a slight improvement in test-set AUC.The action-oriented organization yields 78.7% speed-up and the proposed position and time biases yield 25.0%.Source: Table 1 · p. 4, §4 · p. 4
4.1 Item-Action Organization
- Treating items as positional information and iteratively predicting their actions halves attention input length, cutting attention costs by 75% and linear projection costs by 50%.This action-oriented organization contrasts with HSTU’s interleaved item-action sequence, which doubles sequence length for ranking.Source: §4.1 · p. 5, Figure 2 · p. 5
- Historical tokens sum item and action embeddings, while candidate tokens use a masked action embedding and a candidate mask to prevent information leakage.Source: §4.1 · p. 5, Figure 3 · p. 6
4.2 Position & Time Biases
- GenRank’s position and time embedding design requires only linear I/O operations while preserving positional and temporal information.The embeddings include position, request index, and bucketed time since the previous request.Source: §4.2 · p. 6
- Parameter-free ALiBi penalizes attention between more distant action tokens and avoids quadratic memory access and gradient-backpropagation overhead.The authors fuse it into flash attention at minimal computational cost.Source: §4.2 · p. 6
5 Related Work
Generative recommendation frames recommendation as sequence generation, while large-scale generative ranking remains underexplored. GenRank treats items as positional indicators, and the paper examines scaling and efficiency for real-world ranking.
5.1 Generative Recommendation
- TIGER quantizes item semantic embeddings into hierarchical IDs for next-item prediction; ColaRec and LETTER enhance collaborative signals, while COBRA uses coarse-to-fine generation.Generative recommendation formulates recommendation from historical user behaviors as sequence generation.Source: §5.1 · p. 6
- GenRank treats items as positional indicators and reformulates recommendation as action-oriented generation, addressing an underexplored large-scale ranking setting.Prior work includes HSTU, which predicts actions through an interleaved organization that treats user actions as a new modality.Source: §5.1 · p. 7
5.2 Scaling Law in Recommendation System
- Recommendation scaling behaviors have been observed in retrieval and ranking, while large-scale deployment of generative models requires efficiency consideration.The paper introduces a ranking architecture intended to maintain overhead comparable to the current industrial recommender.Source: §5.2 · p. 7
6 Online Experiments
In Xiaohongshu’s Explore Feed online experiment, GenRank outperformed production ranking across the reported online metrics. It used comparable overall resources, with higher training but lower inference and storage costs.
6 Online Experiments
- The test assigned non-overlapping random 10% user groups to production ranking and GenRank, with each group containing tens of millions of users.All models traced back more than three months of data and were trained online.Source: §6 · p. 7
- GenRank outperformed production ranking in all four online metrics: time spent, reads, engagements, and LT7.The reported result averages a 15-day A/B test; the table is identified as the Explore Feed online A/B test result.Source: Table 2 · p. 7, §6 · p. 7
- Over 25% better P99 response time accompanied comparable overall resources, with GenRank requiring higher training but lower inference and storage costs.The paper notes potential for further optimization in test-time scaling.Source: §6 · p. 7
7 Conclusion
The paper finds generative architecture to be the primary source of effectiveness in generative recommendation and presents GenRank as an efficient ranking architecture. Large-scale offline and online experiments demonstrate its effectiveness and efficiency.
7 Conclusion
- GenRank treats items as positional information and iteratively predicts user behaviors to address inefficiencies in existing approaches.The paper’s theoretical analysis and empirical results identify generative architecture, rather than training paradigm, as the primary source of effectiveness.Source: §7 · p. 7