Source-linked AI summary
Difficulty-Aware Semantic-ID Optimization for Generative Recommendation
Xin Yu, Stephen Li, Sina Aghaei, Zifan Zhu, Jiamu Bai, Guanjie Huang, Bo Peng, Yiyao Liu, Lingzhou Xue
TL;DR
Target-missing semantic-ID rollout groups can deprive vanilla GRPO of useful reward variation, especially when candidates partially follow the target path. DASO profiles each group’s prefix depth, reallocates a bounded subset to bottleneck-guided completions while retaining raw contrast, and adds prefix credit plus an SFT anchor. It improves over MiniOneRec-style GRPO on 11 of 12 metrics, leads on 9 of 12, and improves most level-wise recall metrics on the internal task.
Problem
Target-missing rollout groups can yield weak or degenerate GRPO reward variation even when candidates match part of the target SID path.
Method
DASO profiles rollout groups by prefix-match depth, reallocates bounded guided completions to bottleneck SID levels, retains raw rollouts, adds SID-prefix credit, and uses an auxiliary SFT anchor.
Results
DASO improves over MiniOneRec-style GRPO on 11 of 12 metrics, achieves the highest value on 9 of 12, and improves most level-wise recall metrics on the internal four-level SID task.
Takeaways & Limitations
Prefix-depth-aware rollout construction improves aggregate recommendation quality across public Amazon benchmarks and an internal industrial dataset.
Abstract
from arXiv · showhide
Semantic-ID-based generative recommendation casts retrieval and ranking as autoregressive generation over hierarchical item identifiers. A common recipe is SFT followed by GRPO, yet vanilla GRPO is poorly matched to this tree-structured task. Under the frozen SFT checkpoint, the exact target is absent from the first 16 candidates of the 50-beam constrained ranking for many prompts, and in harder cases none of these candidates enters the target SID branch. This prompt-level diagnostic motivates a training concern: when on-policy GRPO groups are similarly target-missing, item-level rewards may produce weak or degenerate reward variation even if some candidates follow part of the target path. We propose Difficulty-Aware Semantic-ID Optimization (DASO), a tree-aware post-training method that addresses this failure mode as an online rollout-allocation problem. Instead of using fixed difficulty buckets or uniformly injecting ground-truth completions, DASO profiles each current rollout group by prefix-match depth, locates the bottleneck SID levels where candidates leave the target path, and reallocates a bounded portion of the group to prefix-guided completions while retaining raw rollouts for contrast. A SID-prefix reward provides graded credit, while an auxiliary SFT anchor mitigates regression on examples already solved by the SFT checkpoint. On the public benchmarks, DASO improves over MiniOneRec-style GRPO on 11 of 12 metrics and achieves the best result on 9 of 12 metrics; it also improves most level-wise recall metrics on the internal recommendation task.
Introduction
Semantic-ID recommendation makes autoregressive generation practical by representing items as discrete token sequences, but target-missing rollout groups can weaken GRPO’s relative learning signal. DASO addresses this mismatch through adaptive, tree-aware rollout allocation and auxiliary supervision.
- Semantic-ID generation represents each item as a short sequence of discrete semantic tokens, turning recommendation into autoregressive item-ID generation.
- 52.5%–63.9% of public test prompts had no first-token match among the first 16 candidates in 50-beam constrained rankings.Under MiniOneRec rewards, such groups can collapse to identical zero rewards even when candidates partially match the target path.
- DASO profiles rollout groups by prefix depth and reallocates a bounded subset to completions guided at bottleneck SID depths while retaining raw rollouts for contrast.The method combines online profiling, bounded allocation, SID-prefix credit, and an auxiliary SFT anchor.
- DASO’s auxiliary SFT anchor improves target-missing cases while reducing regression on examples already solved by the SFT checkpoint.
- DASO provides empirical evidence of improved aggregate recommendation quality across two Amazon categories, two Qwen backbones, and an internal four-level SID task.Diagnostic improvements are largest for initially partial-prefix and no-prefix prompts, while ablations support the roles of its main components.
Related Work
Prior work spans generative recommendation with semantic identifiers, partial-guidance reinforcement learning, structured sampling, and supervised regularization. DASO is positioned as a tree-specific method for choosing both the amount and depth of rollout guidance.
- Semantic-ID recommenders generate valid hierarchical item identifiers under constrained decoding, linking recommendation with entity and document generation.TIGER and OneRec-style systems study semantic identifiers and end-to-end generative recommendation in different settings.
- LUFFY, BREAD, and Prefix-RFT use expert-guided traces or prefixes to improve reinforcement learning when successful trajectories are rarely sampled.These methods suggest that partial guidance can densify sparse rewards and improve within-group comparisons.
- PPO and GRPO provide the policy-gradient backbone for semantic-ID recommendation, alongside rank-level credit, value-guided sampling, and Sibling-GRPO.DASO instead focuses on selecting how much and where to guide a current rollout group relative to the target SID.
- Supervised or pretraining-style gradients and related regularization methods are used to reduce policy regressions during reinforcement-learning optimization.The cited work includes reference-model KL constraints, supervised gradients during PPO, and recommendation-specific NLL regularization.
Preliminaries
Semantic IDs organize items in a coarse-to-fine tree, making prefix correctness meaningful for generation and reward design. GRPO compares within-prompt rollouts, while DASO modifies rollout construction to restore informative contrast when rewards collapse.
- Semantic-ID Tokenization and Generation: Each item is represented by an M-level semantic-ID sequence whose codebooks have sizes Cℓ, with the tokenizer fixed before recommender post-training.The resulting identifiers form a coarse-to-fine semantic tree in which related items ideally share high-level prefixes.
- Semantic-ID Tokenization and Generation: A valid-prefix trie constrains decoding to catalog SIDs, and the generated SID is resolved to an item through a fixed SID-to-item index.
- Semantic-ID Tokenization and Generation: The prefix-match-depth score measures the longest prefix shared by a generated SID and the target SID, so partial correctness receives structured credit.
- Semantic-ID Tokenization and Generation: An early mismatch at a high-level SID token routes decoding into an incorrect subtree, making later recovery of the target item unlikely.This structure motivates DASO’s prefix-depth profile and SID-prefix reward.
- Group Relative Policy Optimization: GRPO samples multiple completions for each prompt and uses within-group reward statistics to compute critic-free relative advantages.Identical or near-identical rewards make all advantages approximately zero, producing a degenerate policy-gradient signal.
- DASO Rollout Construction: DASO samples raw completions, identifies bottleneck depths from prefix-profile drops, replaces selected weak completions with guided suffix samples, and updates normalized rewards with GRPO plus an SFT loss.
Methodology
DASO reallocates a bounded portion of current GRPO rollout groups using target-path progress in the SID tree, while retaining raw completions for contrast. It combines bottleneck-aware prefix guidance, graded SID-prefix rewards, and an SFT anchor to recover sparse training signal without discarding solved examples.
- Online Prefix-Depth Profiling: DASO profiles each raw rollout group by prefix-match depth and uses the current target-path profile rather than fixed difficulty buckets or external guidance.The profile is recomputed online under the current policy π_t.
- Bottleneck-Aware Rollout Allocation: DASO assigns a bounded guided budget across bottleneck depths while preserving raw-policy completions, keeping the final group size G and GRPO’s within-group contrast.Guided slots replace the weakest raw rollouts, and the forced target prefix conditions only the decoded suffix for policy likelihood terms.
- Bottleneck-Aware Rollout Allocation: Bottleneck depths are identified by drops in cumulative prefix coverage, with larger drops marking more rollouts that leave the target path at that SID level.If the raw group contains the exact target SID, rollout construction remains unchanged.
- Bottleneck-Aware Rollout Allocation: The guided budget increases as average prefix-match depth decreases, giving more intervention to no-prefix groups and less to partial-prefix groups.This deterministic group-local rule uses no additional learned value model or search module.
- Difficulty-Aware Reward Design: DASO adds intermediate SID-prefix credit to the MiniOneRec recommendation reward so partially correct candidates receive graded signal when exact-item rewards are sparse.The ranking-reward weight remains λ_rank = 0.3, while the normalized SID-prefix term has unit coefficient in reported runs.
- SFT-Anchored Policy Stabilization: An auxiliary SFT anchor stabilizes the policy by limiting drift and mitigating regression on prompts whose target SIDs the SFT checkpoint already solves.This regularization offsets the rollout distribution’s deliberate shift toward target-prefixed trajectories.
Experiments
DASO is evaluated on public Amazon Reviews 2018 categories and an internal four-level SID task using shared constrained-decoding and GRPO settings. It outperforms strong comparison frameworks, especially on difficult prompts, while ablations support online profiling, prefix credit, bounded guidance, and SFT anchoring.
- Datasets: DASO is evaluated on two Amazon Reviews 2018 categories and a proprietary internal user-to-item recommendation dataset.The internal dataset contains 1.36M main recommendation examples, a 1.2M-example SFT pool, 100K GRPO prompts, and about 36K evaluation prompts.
- Semantic-ID construction: Amazon experiments use fixed three-level SIDs, while the internal task uses four-level SIDs with 2048 possible codes per level.The public setting follows MiniOneRec’s released SID pipeline; the internal setting tests a deeper and wider SID tree.
- Protocol and baselines: DASO is compared with MiniOneRec SFT, MiniOneRec-style GRPO, uniform ground-truth injection, and Sibling-GRPO under shared constrained decoding.The GRPO configuration uses group size G = 16, β = 10^-3, learning rate 10^-5, and 600 optimizer updates.
- Amazon results: DASO achieves the highest value on 9 of 12 Amazon metrics and improves over MiniOneRec-style GRPO on 11 of 12 metrics.The largest reported HR@5 gains occur on Office Products: 0.1420 to 0.1639 for 1.5B and 0.1313 to 0.1683 for 3B.
- Amazon results: DASO outperforms MiniOneRec + GT on 11 of 12 metrics and Sibling-GRPO on 10 of 12 metrics, with gains in both HR and NDCG.These are framework-level comparisons rather than isolated attribution tests for individual components.
- Internal results: On the internal task, Top-20 lv0 recall rises from 47.21% to 54.23% and final-level lv3 recall from 4.23% to 4.67%, while Top-1 recall improves at all four levels.Top-20 lv2 is the only reported exception, with the GRPO baseline slightly higher.
- Difficulty analysis: The largest bucket-level gains occur on initially Medium and Hard prompts, including Office Hard HR@5 improvements from 0.0331 to 0.0624 for 1.5B and 0.0252 to 0.0521 for 3B.These buckets correspond to partial-prefix and no-prefix prompts under the frozen SFT checkpoint.
- Ablations: Ablations show that online profiling, SID-prefix credit, a maximum guided budget of Bmax = 9, and SFT anchoring contribute to performance.Removing SFT anchoring lowers aggregate HR@5 in all four reported Amazon settings.
Conclusion
The paper identifies target-path coverage gaps as a source of weak or degenerate GRPO reward variation and proposes DASO to recover signal through prefix-aware rollout allocation. Across public and internal tasks, the method improves recommendation quality, with the largest bucket-level gains on target-missing examples.
- Conclusion: DASO profiles current rollout groups, allocates prefix-guided completions to bottleneck SID depths, and adds an auxiliary SFT anchor for stability.The method is designed for target-missing regimes in which group-relative rewards may be weak or degenerate.
- Conclusion: DASO improves aggregate recommendation quality on public Amazon benchmarks and an internal industrial dataset, with largest bucket-level gains on initially target-missing examples.The conclusion emphasizes partial-prefix and no-prefix regimes as the main diagnostic beneficiaries.
Organization
The appendix expands experimental and diagnostic details that cannot fit in the main paper while preserving the main method and its evaluation focus.
- Organization: The appendix documents the evaluation protocol, semantic-ID setting, ablation plan, and bucket-level analysis without changing the main method.It also explains reproducibility information and describes the public and internal recommendation tasks.
Scope of the Appendix
The appendix supplies lower-level information needed to interpret the paper’s motivation and aggregate results.
- Scope of the Appendix: The appendix records fixed SID assignments, training stages, decoding constraints, guided-rollout budgets, diagnostic bucket construction, controlled ablations, and bucket-level Amazon tables.These materials provide the experimental detail underlying interpretation of the main results.
Run Protocol
Public experiments use fixed chronological splits, fixed semantic-ID assignments, and a shared SFT checkpoint before method-specific GRPO training. Evaluation uses a common valid-SID trie, fixed random seeds, and single runs without multi-seed significance tests.
- All methods start from the same SFT checkpoint, so post-training comparisons isolate differences among the GRPO-stage recipes.
- The experiments use fixed chronological data splits and fixed semantic-ID assignments across datasets, backbones, and methods.
- A shared valid-SID trie constrains decoding during both rollout collection and evaluation, excluding syntactically invalid item identifiers.
- Each dataset, backbone, and method setting is evaluated from one algorithm run with fixed random seeds.
- The study does not report multi-seed significance tests because GRPO rollout collection is computationally expensive.
Data and Code Availability
The public benchmark uses reproducible Amazon data and fixed semantic-ID assignments, while the internal industrial task is proprietary and reported only through aggregate statistics. Diagnostic and ablation analyses examine how DASO performs across difficulty regimes and training variants.
- Public experiments use Amazon Reviews 2018 categories with released MiniOneRec-style semantic-ID assignments and reproducible preprocessing and evaluation code.
- The internal industrial dataset is proprietary, so the paper reports aggregate task, user, prompt, and SID-space statistics without releasing raw logs, catalog metadata, or access code.
- Public evaluation uses Office Products and Industrial and Scientific under a 5-core chronological next-item prediction protocol with constrained semantic-ID decoding.
- The public methods share fixed semantic-ID mappings and SFT initialization, isolating post-training differences among GRPO variants and DASO.
- Diagnostic buckets reveal that many prompts are initially difficult, motivating analysis beyond aggregate HR and NDCG metrics.
- DASO’s largest bucket-level gains generally occur on Medium and Hard prompts, including 1.5B Office Products Hard HR@5 increasing from 0.0331 to 0.0624 over MiniOneRec.