Source-linked AI summary
GR2: Generative Reasoning Re-ranker
Mingfu Liang, Yufei Li, Jay Xu, Kavosh Asadi, Xi Liu, Shuo Gu, Kaushik Rangadurai, Frank Shyu, Shuaiwen Wang, Song Yang, Zhijing Li, Jiang Liu, Mengying Sun, Fei Tian, Xiaohan Wei, Chonglin Sun, Jacob Tao, Shike Mei, Wenlin Chen, Santanu Kolay, Sandeep Pandey, Hamed Firooz, Luke Simon
TL;DR
LLM-based recommendation research leaves reranking, reasoning supervision, and scalable item representations insufficiently addressed. GR2 combines semantic-ID mid-training, curated reasoning-trace fine-tuning, and reranking-specific DAPO reinforcement learning, and it outperforms OneRec-Think across Recall@K and NDCG@K on two real-world datasets.
Problem
Existing LLM recommendation approaches often neglect reranking, underuse RL-enhanced reasoning, and rely on non-semantic item identifiers that create scalability challenges.
Method
GR2 uses three stages: semantic-ID mid-training, rejection-sampled reasoning traces for supervised fine-tuning, and DAPO with a reranking-specific reward function.
Results
GR2 consistently surpasses OneRec-Think across Recall@K and NDCG@K on two real-world datasets, while ablations validate reasoning and RL contributions.
Takeaways & Limitations
Semantic representations, structured reasoning, and reward-driven optimization together support LLM reranking tailored to large-scale recommendation systems.
Abstract
from arXiv · showhide
Recent studies increasingly explore Large Language Models (LLMs) as a new paradigm for recommendation systems due to their scalability and world knowledge. However, existing work has three key limitations: (1) most efforts focus on retrieval and ranking, while the reranking phase, critical for refining final recommendations, is largely overlooked; (2) LLMs are typically used in zero-shot or supervised fine-tuning settings, leaving their reasoning abilities, especially those enhanced through reinforcement learning (RL) and high-quality reasoning data, underexploited; (3) items are commonly represented by non-semantic IDs, creating major scalability challenges in industrial systems with billions of identifiers. To address these gaps, we propose the Generative Reasoning Reranker (GR2), an end-to-end framework with a three-stage training pipeline tailored for reranking. First, a pretrained LLM is mid-trained on semantic IDs encoded from non-semantic IDs via a tokenizer achieving $\ge$99% uniqueness. Next, a stronger larger-scale LLM generates high-quality reasoning traces through carefully designed prompting and rejection sampling, which are used for supervised fine-tuning to impart foundational reasoning skills. Finally, we apply Decoupled Clip and Dynamic sAmpling Policy Optimization (DAPO), enabling scalable RL supervision with verifiable rewards designed specifically for reranking. Experiments on two real-world datasets demonstrate GR2's effectiveness: it surpasses the state-of-the-art OneRec-Think by 2.4% in Recall@5 and 1.3% in NDCG@5. Ablations confirm that advanced reasoning traces yield substantial gains across metrics. We further find that RL reward design is crucial in reranking: LLMs tend to exploit reward hacking by preserving item order, motivating conditional verifiable rewards to mitigate this behavior and optimize reranking performance.
1 Introduction
GR2 targets overlooked LLM-based recommendation reranking with semantic item representations, structured reasoning data, and reward-driven reinforcement learning. Its three-stage pipeline combines semantic-ID mid-training, rejection-sampled reasoning traces, and customized DAPO supervision.
- Motivation: Recent LLM recommendation work often neglects reranking, underuses RL-enhanced reasoning, and relies on non-semantic item IDs that challenge industrial scalability.These limitations concern both the reasoning supervision available to LLMs and the vocabulary expansion caused by large item-ID inventories.
- Motivation: GR2 investigates design principles for applying LLMs specifically to the reranking stage rather than primarily to retrieval or late-stage ranking.The framework is tailored to refining candidate lists in large-scale recommendation systems.
- Three-stage pipeline: The first stage mid-trains a student LLM on semantic IDs produced from non-semantic IDs with ≥99% uniqueness.This representation is intended to connect item semantics with world knowledge while preserving item distinguishability.
- Three-stage pipeline: The second stage uses reranking-focused prompts and rejection sampling to create high-quality hierarchical reasoning traces for supervised fine-tuning.The curated traces provide foundational reasoning skills for connecting item semantics with user preferences.
- Three-stage pipeline: The final stage adapts DAPO with a reranking-specific reward function, providing scalable reward-driven supervision to refine reasoning and reranking performance.The pipeline overview describes student mid-training, teacher-generated reasoning data, and student reasoning enablement through SFT and RL.
- Evaluation: GR2 surpasses OneRec-Think across Recall@K and NDCG@K on two real-world datasets, while ablations validate contributions from reasoning and RL.The evaluation also reports additional design insights from the ablation studies.
2 Tokenized Mid-Training
This section presents semantic-ID tokenization and mid-training methods for scalable recommendation, emphasizing codebook utilization, uniqueness, and alignment with language-model knowledge.
- Tokenization: Semantic IDs represent items as compact discrete sequences derived from textual features, helping mitigate scalability issues from massive item vocabularies.The tokenizer maps item text to codebook indices, with RQ-VAE providing the core representation mechanism.
- Mid-Training: Contrastive loss encodes co-engagement history by increasing similarity between co-engaged items and distinguishing randomly sampled non-co-engaged items.Similarity is defined using temperature-scaled cosine similarity between item embeddings.
- Balanced Codebook Utilization: Codebook collapse can cause poor reconstruction and high collision rates, motivating complementary techniques for balanced codebook utilization.The paper uses k-means++ initialization, EMA updates, diversity regularization, and dead-code resetting.
- Uniqueness Enhancement: Randomizing the last M codebook levels trades some reconstruction fidelity for guaranteed uniqueness while preserving coarse semantic information in early levels.Early levels encode coarse categories, whereas later levels capture finer details.
- Mid-Training: Mid-training interleaves semantic IDs with natural-language tokens so the LLM can align recommendation knowledge with linguistic representations and world knowledge.This follows the item-alignment strategy associated with OneRec-Think.
3 Reasoning Data Generation
This section constructs chat-format training data that combines semantic-ID-grounded context, structured reasoning, and ranked outputs, then generates traces through targeted and rejection sampling.
- Chat-Format Training Data: Training examples present purchase history and candidate items with semantic IDs, titles, and categories so the model can compare the full candidate set.The shared SID-plus-metadata format supports consistent representation across history and candidates.
- Chat-Format Training Data: Assistant outputs contain stepwise reasoning and a ranked recommendation list in structured JSON format.The trace summarizes history, identifies category patterns, explains the preference, and produces an ordered list.
- Chat-Format Training Data: Structured reasoning traces provide the training signal for post-training models to develop re-ranking reasoning capabilities.The paper identifies the assistant-side trace as central to enabling reasoning for re-ranking.
- Prompt Design: Prompt design adds domain knowledge, explicit SID citation constraints, and hierarchical reasoning steps to make traces grounded and verifiable.The steps progress from pattern recognition to complementary product types and candidate matching.
- Reasoning Trace Generation: Targeted sampling supplies the target item and recent history to a larger LLM, producing a rationale for the target interaction.Because the ground truth is included, this method can generate rationales that may lack genuine belief in the result.
- Reasoning Trace Generation: Rejection sampling withholds the ground truth and repeatedly queries predictions until the predicted candidate matches the actual target.The retained trace and prediction are conditioned on agreement with the target item.
4 Reasoning Enablement for Re-Ranking
GR2 frames re-ranking as promoting the ground-truth next item within a fixed candidate set, using supervised reasoning and reinforcement learning to improve ranking quality. Its RL stage combines ranking and format rewards while addressing reward hacking through conditional reward design and DAPO optimization.
- GR2 re-ranks a retriever-produced candidate list so the ground-truth next item moves higher.
- Supervised fine-tuning: The model generates both reasoning traces and ranked outputs, using supervised training to acquire reasoning capabilities for re-ranking.The reasoning traces connect user history with candidate comparisons.
- Supervised fine-tuning: Separate loss weights for reasoning and ranking tokens preserve ranking performance while enabling coherent reasoning.The weighting condition λr < λo balances reasoning fluency and ranking accuracy.
- Reinforcement learning: RL further refines reasoning and ranking quality by rewarding promotion of the target item in the pre-ranked list.The ranking reward compares the target item's pre-ranked and re-ranked positions.
- Reinforcement learning: Combining ranking and format rewards naively can cause reward hacking, with the model preserving the original ranking to obtain format reward.The format reward checks whether reasoning and ranking outputs can be reliably extracted.
- Reinforcement learning: DAPO optimizes the policy with group-normalized advantages, asymmetric clipping, and filtering of prompts whose accuracy is already 0 or 1.These procedures are described as improving sample efficiency and avoiding zero policy gradients.
5 Experiments
Experiments evaluate semantic-ID quality, tokenized mid-training, and reasoning-aware re-ranking on Amazon Beauty and Sports datasets. Results show that EMA updates and random last-level assignment support high SID uniqueness, while tokenization, mid-training, reasoning traces, and RL improve recommendation and re-ranking performance.
- High SID Uniqueness: EMA updates increase average SID uniqueness from 31.32% to 97.50% (+66.2%), making them the most impactful codebook-balancing technique.The study attributes this improvement to preventing codebook collapse.
- High SID Uniqueness: Random last-level assignment raises average SID uniqueness from 54.06% to 82.76% (+28.7%), and every configuration achieving ≥99% uniqueness uses it.The technique randomly assigns the final quantization level during inference.
- High SID Uniqueness: ≥99% unique SID rate requires EMA updates and random last-level assignment, whereas diversity loss has minimal impact and contrastive loss reduces uniqueness by 13.2%.Dead code reset also has a slight negative effect of −4.0%.
- Tokenized Mid-Training: Tokenization enhancement improves ORT Base by 6.7% in Recall@5, 6.3% in Recall@10, 4.5% in NDCG@5, and 4.2% in NDCG@10 on Amazon Beauty.Combining tokenization with mid-training substantially outperforms ORT Base+IA, including an 18.7% NDCG@5 gain for single-task learning.
- Re-Ranking: SFT does not consistently improve re-ranking, whereas RL is needed to optimize reasoning toward ranking-aware rewards.The reported results describe a mismatch between coherent reasoning traces and ranking optimality, with SFT sometimes degrading Recall@1.
6 Related Works
Prior work applies LLMs to generative recommendation and reasoning-enhanced ranking, while GR2 emphasizes semantic IDs, high-quality reasoning traces, and re-ranking-specific optimization.
- Semantic IDs reformulate recommendation as autoregressive generation over discrete item representations, avoiding large embedding tables.
- GR2 differs by combining higher-uniqueness semantic IDs, rejection-sampled reasoning traces, and DAPO-based rewards tailored to recommendation re-ranking.
- Reasoning methods use chain-of-thought extensions and verifiable-reward reinforcement learning to improve reasoning quality and coherence.
- Document-ranking systems demonstrate reasoning-aware re-ranking across text, multi-view, and multimodal settings, while recommendation work explores iterative reflection and candidate-list randomization.
7 Conclusion
GR2 is a three-stage framework that combines semantic IDs, supervised reasoning traces, and re-ranking-specific reinforcement learning. Experiments report gains over strong baselines and show the importance of reasoning and RL objectives.
- GR2 uses mid-training on highly unique semantic IDs, supervised fine-tuning on reasoning traces, and reinforcement learning with a re-ranking-specific reward function.
- GR2 experiments on real-world datasets consistently surpass strong baselines on recall and ranking metrics.
- Ablation studies highlight the importance of advanced reasoning and reinforcement-learning objectives.
- The framework integrates semantic representations, structured reasoning, and reward-driven optimization for interpretable and effective large-scale re-ranking.
A Semantic ID (SID)
The semantic-ID component encodes item representations as sequences of discrete quantization indices. Residual quantization progressively approximates item embeddings, and inverse mapping supports retrieval at inference.
- A textual encoder converts an item feature x into a continuous embedding used by the semantic-ID pipeline.
- Residual quantization sequentially applies K codebooks, updating the residual after each nearest-neighbor quantization step.
- The reconstructed representation is formed by summing the quantized vectors across levels.
- An item’s semantic ID is the sequence of discrete indices (z1, z2, ..., zK), with each index drawn from its codebook.
- At inference, the generated semantic ID is inverse-mapped through the tokenizer to retrieve candidate items.
- The RQ-VAE training procedure computes embeddings and residuals, generates level-wise IDs, and updates codebooks across training samples.
B Examples of Item Alignment Tasks
The item-alignment examples train models to connect semantic item tokens with descriptions, user histories, preferences, and category hierarchies. Reasoning traces progressively narrow broad interests to a specific candidate category.
- Item-alignment tasks include persona grounding, sequential preference modeling, itemic dense captioning, and general language modeling.
- Itemic dense captioning prompts the model to generate a concise textual description from an itemic token.
- Purchase-history examples pair semantic IDs with item titles and categories so the model can analyze user preferences.
- Category-aware prompts organize products hierarchically from broad domains to specific types such as flattening irons.
- The reasoning format requires SID citations and category lines that progress from broad to specific and final categories.
- The generated trace selects a ceramic hair straightener because it aligns with the user’s smoothing and frizz-control focus.
D.1 Reasoning Trace Generation with the Context of Category Hierarchy
The reasoning-trace generation setup uses category hierarchies, semantic-ID citations, and staged reasoning to predict the next purchase from a user’s history. The example narrows from broad hair-care preferences to a specific category and candidate prediction.
- Category Hierarchy: The category hierarchy organizes products from broad roots such as Beauty and Hair Care to increasingly specific product types and variants.The hierarchy uses levels such as Conditioners, Shampoos, Hair & Scalp Treatments, and product-specific attributes.
- Prompt Design: The prompt asks the model to predict the candidate most likely to be purchased next from the user’s purchase history.The task is framed as e-commerce preference prediction over a list of candidates.
- Semantic-ID Reasoning: The prompt requires purchase-history items to be cited with semantic IDs so reasoning can interleave language with item identifiers.The instructions explicitly connect SID citations with natural-language reasoning.
- Staged Reasoning: The example proceeds through three reasoning steps, moving from broad hair-care interest to moisture-focused conditioners before predicting Candidate 3.Each step includes a reasoning line and a category line, ending with a specific prediction.
- Generated Trace: A generated trace instead identifies argan-oil and treatment patterns and concludes that Candidate 1 best matches the predicted category.The trace compares candidate categories with the user’s prior purchases before selecting Candidate 1.
E Methodology Details
The reranking parser recovers reasoning and rankings from structured or malformed language-model outputs, then validates the result into a complete candidate permutation.
- Parser Overview: The parser accepts raw LLM output and returns an optional reasoning trace together with a validated ranking.If no valid ranking is recovered, it returns None values.
- Structured Parsing: JSON-based parsing searches JSON-like substrings in reverse order and extracts reasoning and ranking fields from the first usable structure.It recognizes explanation or reasoning fields and recommendations or ranking fields.
- Fallback Parsing: When structured parsing fails, regex fallback extracts numeric candidate indices from ranking patterns.The fallback is designed to tolerate outputs that do not form valid JSON.
- Validation: Ranking validation keeps valid integers within the candidate range, removes duplicates while preserving order, and converts indices to zero-based form.Invalid entries are filtered before downstream use.
- Canonicalization: Missing candidates are appended to create a full permutation, guaranteeing a valid ranking for reward computation.The resulting permutation has length |D| when valid ranking entries remain.
F Case Study
The case study applies targeted reasoning to hair-care purchase histories, where prior shampoo-and-conditioner purchases guide reranking toward a matching product set. The example selects Candidate 2 as the best category and purchase-pattern match.
- Case Setup: The case study asks the model to rerank pre-ranked candidates using purchase history and predict the item most likely to be purchased next.The setup uses reasoning with semantic identifiers.
- Candidate Matching: Candidate 2 is a Pureology Hydrate shampoo-and-conditioner duo that matches the user’s category and set-oriented purchase pattern.Its category is Beauty > Hair Care > Shampoo & Conditioner Sets.
- Candidate Pool: The candidate pool also includes shampoos, conditioners, styling tools, skin-care moisturizers, and broader hair-care products.These alternatives provide different category matches for the reranking decision.
- Purchase History: The user history consists of Pureology Hydrate shampoo and conditioner products, indicating interest in hydrating hair-care products and complete sets.The example describes both individual products and a shampoo-and-conditioner duo.
- Prediction: The generated reasoning trace concludes that Candidate 2 best matches the predicted category and complements the user’s recent purchases.The conclusion is stated directly after the category-pattern analysis.
G Evaluation Metrics
The evaluation defines ranking quality with Recall@K and NDCG@K, while also measuring semantic-ID quality through uniqueness and collision rate. These metrics assess both recommendation accuracy and identifier discrimination.
- Recommendation Metrics: Recall@K measures whether the ground-truth next item appears among the model’s top-K ranked items.The metric uses the user set, the top-K item set, and an indicator function.
- Recommendation Metrics: NDCG@K computes discounted gain according to the position of the ground-truth next item in the ranking.The ideal gain places the ground-truth item first under the leave-one-out protocol.
- Semantic-ID Metrics: SID collision rate measures how often multiple items receive the same semantic ID, with lower rates indicating stronger discriminative power.The metric is used to evaluate the quality and fidelity of semantic item encoding.
- Semantic-ID Metrics: The collision set contains items whose SID is shared with at least one distinct item.This definition identifies the items contributing to collisions.