Learning Better Reasoning for Generative Recommendation with Semantic IDs

1 Introduction

• Semantic IDs represent items as discrete code sequences whose shared prefixes let related items share codes, unlike independent item embeddings that scale with the catalog and hinder cold-item placement.Generative recommendation frames next-item prediction as autoregressive sequence generation.Source: §1 · p. 1
• Reasoning traces can be plausible yet misleading, so indiscriminate training may add noisy supervision and reinforce spurious patterns that misguide later reinforcement learning.Different traces for the same history may summarize different preferences, infer different transitions, or conflict in their recommendations.Source: §1 · p. 2
• Exact-match feedback gives limited ranking guidance: a trace moving the target from rank 100 to rank 2 can receive the same top-1 reward as one leaving it unretrieved.Jointly optimizing reasoning and item generation also entangles reasoning quality with stochasticity in SID generation.Source: §1 · p. 2
• Evo-Rec aligns SIDs with item semantics and behavior, selects positive-utility reasoning traces for SFT, then optimizes the reasoning policy through catalog-constrained, ranking-aware RL.The RL reward uses the ground-truth SID's rank in trie-constrained beam search.Source: §1 · p. 2
• Across three Amazon Review benchmarks, Evo-Rec beats discriminative, generative, and reasoning-enhanced baselines on every metric; Games Recall@5 rises from 0.0710 to 0.0847 and NDCG@10 from 0.0563 to 0.0746 over the strongest baseline.The gains correspond to relative improvements of 19.3% and 32.5%, respectively.Source: §1 · p. 2

2.1 From Discriminative to Generative Recommendation

• Discriminative recommenders encode histories and score independent item embeddings, so embedding tables grow with catalogs and sparse or unseen items are difficult to place.These systems do not share representations between semantically related items, limiting long-tail generalization.Source: §2.1 · pp. 2–3
• Generative methods replace item tables with sequence generation; Semantic IDs encode items as discrete codes, letting related items share prefixes and enabling generalization to unseen items.Later work improves tokenizers with collaborative signals, aligns large language models to SID vocabularies, and studies end-to-end scaling.Source: §2.1 · p. 3

2.2 Reasoning-Enhanced Recommendation

• Reasoning recommenders span latent and natural-language approaches, but prior methods do not identify which reasoning trajectories benefit downstream recommendation; Evo-Rec selects useful traces and applies ranking-aware RL.Latent reasoning states are not inspectable or verifiable, limiting direct measurement and supervision of reasoning quality.Source: §2.2 · p. 3

3.1 Problem Formulation

• The model predicts the next item's fixed-length Semantic ID by generating an intermediate reasoning sequence conditioned on the interaction history.Source: §3.1 · p. 3

3.2 Aligning Language and Semantic IDs in LLMs

• SID–language alignment grounds newly introduced item tokens in item semantics and user behavior through complementary prediction, translation, and interleaved-text tasks.The unified autoregressive objective establishes a representation foundation before the later stages learn reasoning trajectories.Source: §3.2 · pp. 3–4

3.3 Best-of-𝑁Rejection Sampling Supervised Fine-Tuning

• Best-of-N Rejection Sampling SFT selects the highest-utility candidate CoT only when it improves target-SID prediction over history-only prediction, then fine-tunes on accepted pairs.Utility compares target-SID likelihood with and without each candidate rationale under the same interaction context.Source: §3.3 · p. 4

3.4 Reasoning Optimization with Constrained SID Ranking

• Ranking-aware RL samples reasoning trajectories, uses trie-constrained beam search to rank valid catalog items, and rewards trajectories according to the target item's rank.NDCG@10 gives progressively smaller credit to lower target ranks, and GRPO optimizes rewards across sampled trajectories.Source: §3.4 · p. 5

4.1 Experimental Setup

• Games, Office, and Industrial are 5-core filtered, chronologically ordered, and split into earliest 80% training, next 10% validation, and latest 10% testing interactions.Histories are truncated to a maximum length of 10; dataset statistics after preprocessing are summarized in Table 1.Source: §4.1.1 · p. 5, Table 1 · p. 5

4.1.1 Datasets and Evaluation.

• Evaluation ranks the target against the full item catalog using Recall@K and NDCG@K at K=5 and 10.Source: §4.1.1 · p. 5

4.1.2 Baselines.

• The comparison covers traditional discriminative, classic generative, and reasoning-enhanced recommenders, with baseline results quoted from [3].Evo-Rec uses the same data split and evaluation protocol as the baselines.Source: §4.1.2 · p. 5

4.1.3 Implementation Details.

• A Qwen3-1.7B model is separately trained per dataset; Stage 1 uses up to five epochs, and RL uses GRPO with trie-constrained beam search.Each item has three semantic tokens; training uses full fine-tuning, while inference decodes reasoning before full-catalog trie-constrained beam search.Source: §4.1.3 · p. 5

4.2 Main Results

• On Games, Evo-Rec improves N@5 by 41.1% and N@10 by 32.5%, versus R@5 by 19.3% and R@10 by 11.4% over the strongest baseline.Evo-Rec consistently outperforms all baselines across every metric and dataset.Source: §4.2 · p. 5, Table 2 · p. 6
• On Games, R@5 rises from 0.0710 to 0.0847 and N@10 from 0.0563 to 0.0746, the largest reported gains.The authors attribute the gains to Stage 2's stronger reasoning initialization and Stage 3's alignment with the downstream ranking objective.Source: §4.2 · p. 6
• Games has lower thinking-baseline performance than Office and Industrial, suggesting a more challenging setting with greater RL optimization headroom.Improvements on Office and Industrial also suggest the framework is not specific to one domain.Source: §4.2 · p. 6

4.3 Ablation Study

• Best-of-N rejection sampling improves N@10 by 27.1% over random selection and 37.1% over rejection sampling, with an 83.8% relative gain in R@1.The variants use the same candidate CoTs at N=5 and the same training pipeline, differing only in the retained candidate.Source: §4.3 · p. 6, Table 3 · p. 6
• Because 94.0% of candidates already have positive utility, accepting the first positive trace is close to a random draw; ranking candidates by utility matters more.Random selection performs on par with training on a single trace despite using the full sampling budget.Source: §4.3 · p. 7
• On Games, N@10 increases from 0.0571 at N=1 to 0.0614 at N=3 and 0.0746 at N=5.Performance increases monotonically with the sampling budget across every metric.Source: Table 4 · p. 6, §4.3 · p. 7

4.3.2 Sampling Strategy.

• On Games, constrained beam search beats constrained sampling by 30.9% on N@10 and 56.5% on R@1.Both decoders produce catalog-valid candidates, but independent draws leave candidate order undefined and reduce reward to a hit indicator.Source: §4.3.2 · p. 7, Table 5 · p. 7

4.3.3 Reward Design.

• The pure NDCG reward performs best across all evaluation metrics, while exact match does not distinguish the relative ordering of retrieved items.The authors report that adding Recall@10 overemphasizes retrieval and weakens the relative ranking signal.Source: §4.3.3 · p. 7, Table 6 · p. 7

4.4 Discussion

• Across all three domains, mean reasoning length decreases and converges during Stage-3 training while recommendation performance continues improving, without a length penalty.The reported shortening emerges from ranking-aware optimization.Source: §4.4.1 · p. 7, Figure 2 · p. 8

4.4.1 Reasoning length.

• Stage-3 RL achieves stronger recommendation performance with shorter, more targeted reasoning, suggesting that useful ranking information matters more than chain length.The model progressively discards redundant or less relevant reasoning that does not contribute to target ranking.Source: §4.4.1 · p. 7

4.4.2 Evolution of Thinking Performance.

• On Games, R@1 increased from 0.0103 to 0.0446 (333.0% relative improvement), while Stage-3 RL improved all ranking metrics across three domains.Table 7 compares thinking performance before and after Stage-3 RL.Source: §4.4.2 · pp. 7–8, Table 7 · p. 8
• Stage 2 serves as a thinking-mode warm-up with higher-quality trajectories, while Stage 3 reinforces reasoning that places the ground-truth SID higher in the beam.The Stage-2 objective is not to directly optimize final ranking performance.Source: §4.4.2 · p. 8

• Requiring at least one positive-utility candidate rejects only 2.4% of Games, 3.2% of Office, and 2.4% of Industrial instances; rejected cases are mostly near the decision boundary.The binary Δ > 0 criterion is too weak to substantially reshape the training distribution through rejection alone.Source: §5 · p. 8, Figure 3 · p. 9
• The retained arg max trace raises predictive utility over a uniform candidate pick by 0.56 on Games, 0.43 on Office, and 0.49 on Industrial.Mean within-instance utility ranges are 1.22, 0.94, and 1.06, respectively; larger ranges tend to offer more benefit.Source: §5 · p. 8
• History-only saturation rates are 1.9% on Games, 11.5% on Office, and 6.7% on Industrial, so small utility gains need not indicate poor reasoning.Office has the highest saturation and smallest within-instance range (0.94), whereas Games has the lowest saturation and largest range (1.22).Source: §5 · pp. 8–9
• Best-of-N gains arise primarily from comparing candidates within each history, not rejection or sampling-slot bias; the five slots are selected at approximately uniform rates.Negative-utility rates are also largely stable across slots, and increasing N from 1 to 5 monotonically improves results.Source: §5 · p. 9, Figure 4 · p. 9

• Evo-Rec grounds Semantic IDs in language and behavior, selects higher-utility reasoning with Best-of-N rejection-sampling SFT, and refines the policy through catalog-constrained, ranking-aware RL.The framework is evaluated on three Amazon Review benchmarks and consistently improves over discriminative, generative, and reasoning-enhanced recommenders.Source: §6 · p. 9
• Best-of-N sampling primarily helps by distinguishing candidate reasoning traces with different predictive utility within the same interaction history, rather than through rejection alone.NDCG-based RL provides a ranking signal that reflects the ranking induced by each reasoning trajectory and produces particularly pronounced gains at the top, especially on Recall@1.Source: §6 · p. 9
• The optimized policy tends to produce shorter reasoning while achieving stronger recommendation performance, suggesting useful information matters more than reasoning length.Source: §6 · p. 10

• The public Amazon Review benchmarks contain no directly identifying information, and users appear as opaque pseudonymous identifiers.Source: Ethical Considerations · p. 10
• The evaluated setting is offline and does not deploy Evo-Rec on private user data.Source: Ethical Considerations · p. 10
• Future real-world use involving private interaction histories should consider privacy protection, bias auditing, and safeguards against over-personalization.Source: Ethical Considerations · p. 10