Difficulty-Aware Semantic-ID Optimization for Generative Recommendation

Introduction

• Semantic-ID generation represents items as short sequences of discrete semantic tokens, enabling autoregressive recommendation directly from user context rather than ranking a fixed candidate set.The introduction situates this approach in systems including TIGER-style identifiers, OneRec, OneRec-V2, and MiniOneRec.Source: Introduction · p. 1
• 52.5%–63.9% of public test prompts have no first-token match among the first 16 candidates of the frozen-SFT 50-beam ranking, exposing target-missing regimes for GRPO.Under the MiniOneRec reward, groups may have identical zero rewards despite partial target-path matches, weakening or degenerating the group-relative signal.Source: Introduction · pp. 1–2
• DASO profiles prefix-match depth online, reallocates a bounded subset to bottleneck-guided completions, retains strong raw rollouts for contrast, and adds SID-prefix credit plus an SFT anchor.The anchor limits regression on examples already solved by the SFT checkpoint.Source: Introduction · p. 2
• DASO improves aggregate recommendation quality across two Amazon categories, two Qwen backbones, and an internal four-level SID task, with largest bucket-level gains on partial-prefix and no-prefix prompts.The introduction also says ablations are consistent with the roles of online profiling, SID-prefix credit, guided-budget choice, and the SFT anchor.Source: Introduction · p. 2

Semantic-ID Based Generative Recommendation

• DASO targets informative RL when semantic-ID rollout groups miss the target item or correct SID branch, unlike prior work on generation, industrial alignment, and SID grounding.The related work includes TIGER’s hierarchical semantic IDs, OneRec’s industrial generative recommendation, MiniOneRec’s open-source SFT and GRPO pipeline, and SID-reasoning systems.Source: Semantic-ID Based Generative… · p. 3

Guided Rollouts and GRPO for Recommendation

• DASO selects both how much guidance to add and which SID depth to guide within current rollout groups, preserving GRPO optimization and exact-item prediction as the primary objective.Earlier guided-rollout methods use expert traces, partial expert anchors, or demonstration prefixes; semantic-ID work also studies PPO and GRPO, Rank-GRPO, and value-guided structured sampling with Sibling-GRPO (Yan et al., 2025; Zhang et al., 2025; Huang et al., 2026; Schulman et al., 2017; Shao et al., 2024; Zhu et al., 2025; Jiang et al., 2026).Source: Guided Rollouts and GRPO for… · p. 3

Supervised Regularization for Stable Policy Optimization

• DASO uses prefix-guided rollouts to recover signal for groups without exact hits and supervised anchoring to reduce regression on examples already solved by SFT.This shares the stabilization motivation of SFT initialization, reference-model KL constraints, supervised gradients during PPO, reward-regularized self-distillation, and GenRec’s NLL regularization (Christiano et al., 2017; Ziegler et al., 2019; Ouyang et al., 2022; Yu et al., 2026a; Bai et al., 2026; Zou et al., 2026).Source: Supervised Regularization for Stable… · p. 3

Semantic-ID Tokenization and Generation

• Fixed semantic-ID tokenizers map items to discrete hierarchical sequences, and a prefix trie constrains generation to valid catalog identifiers.Related items ideally share high-level prefixes, forming a coarse-to-fine semantic tree; the tokenizer is trained before recommender post-training and then kept fixed.Source: Semantic-ID Tokenization and Generation · p. 4
• An early high-level SID mismatch sends decoding into the wrong subtree, so prefix-match depth captures meaningful partial correctness and motivates DASO’s prefix-depth profile and reward.The prefix-match-depth score measures the longest matched prefix between a generated SID and its target.Source: Semantic-ID Tokenization and Generation · p. 4

Group Relative Policy Optimization

• GRPO compares multiple completions sampled for one prompt to estimate advantages without a critic; near-identical group rewards can make advantages and policy-gradient signals degenerate.DASO retains the clipped GRPO/PPO update but profiles prefix-depth bottlenecks, allocates guided completions to those depths, and combines the GRPO loss with an SFT anchor.Source: Group Relative Policy Optimization · pp. 4–5

Methodology

• DASO deterministically reallocates a bounded portion of each current rollout group using its target-path profile, retains raw completions, leaves GRPO’s update form unchanged, and adds an SFT anchor.Guidance comes from the prompt’s current profile rather than external traces, learned value estimates, or fixed difficulty buckets.Source: Methodology · p. 5

Online Prefix-Depth Profiling

• DASO profiles G valid current-policy SIDs by matched target-prefix depth and summarizes the group with the fraction of rollouts reaching at least each depth.The profile is recomputed during training; SID depth M is 3 in Amazon experiments and 4 in the internal industrial experiment.Source: Online Prefix-Depth Profiling · pp. 5–6

Bottleneck-Aware Rollout Allocation

• DASO leaves groups containing an exact target SID unchanged; otherwise, it allocates a bounded guided budget by bottleneck depth, fixes each guided prefix, and retains raw-policy contrast.The budget increases as average prefix-match depth falls, and guided completions decode the suffix with the current policy; GRPO likelihood-ratio terms apply only to suffix tokens.Source: Bottleneck-Aware Rollout Allocation · p. 6

Difficulty-Aware Reward Design

• DASO retains the MiniOneRec ranking reward (Kong et al., 2025) and adds normalized SID-prefix credit with unit weight, while λrank = 0.3 remains unchanged.The auxiliary prefix term distinguishes partial matches when exact-item rewards are sparse, while accuracy and ranking keep item-level recommendation fidelity central.Source: Difficulty-Aware Reward Design · pp. 6–7

SFT-Anchored Policy Stabilization

• Because prefix guidance shifts trajectories toward target-prefixed completions, DASO adds a token-level SFT anchor to limit policy drift and regression on ground-truth SID generation.The anchor complements guidance that supplies plasticity for groups without exact hits.Source: SFT-Anchored Policy Stabilization · p. 7

Experimental Setup

• Evaluation spans two Amazon Reviews 2018 categories and a proprietary internal recommendation dataset, using three-level public and four-level internal SIDs.The internal task includes 1.36M main examples from 319K users; fixed SFT-derived Easy/Medium/Hard buckets are diagnostic only, not DASO training controls.Source: Experimental Setup · pp. 7–8

Public Amazon Results

• DASO improves over MiniOneRec-style GRPO on 11 of 12 reported metrics and leads on 9 of 12 across Amazon datasets and model sizes.Office Products HR@5 rises from 0.1420 to 0.1639 with the 1.5B backbone and from 0.1313 to 0.1683 with the 3B backbone.Source: Public Amazon Results · p. 8

Internal Industrial Results

• Top-20 lv0 recall increases from 47.21% to 54.23%, and Top-20 lv3 from 4.23% to 4.67%; Top-1 improves at all levels, while Top-20 lv2 is the exception.Source: Table 3 · p. 9, Internal Industrial Results · p. 9

Diagnostic Difficulty-Bucket Analysis

• Hard prompts comprise 63.7% and 63.9% of Office test sets for 1.5B and 3B, where Hard HR@5 rises from 0.0331 to 0.0624 and from 0.0252 to 0.0521.The Easy/Medium/Hard buckets are diagnostic only; the results support gains on initially Medium and Hard prompts while the SFT anchor helps limit regression on SFT exact-hit cases.Source: Diagnostic Difficulty-Bucket Analysis · p. 9

Ablation Study

• Static DASO trails Full DASO on aggregate HR@5/NDCG@5, while removing SFT anchoring lowers aggregate HR@5 across all four settings.Static DASO uses frozen SFT difficulty buckets; removing SID-prefix credit lowers aggregate Amazon performance, and Bmax = 9 performs best among {5, 9, 13} in the Office 1.5B sensitivity study.Source: Ablation Study · pp. 9–10

• DASO improves aggregate recommendation quality on public Amazon benchmarks and an internal industrial dataset, with its largest bucket-level improvements on initially target-missing examples.Source: Conclusion · p. 10
• DASO profiles each rollout group, allocates prefix-guided completions to bottleneck SID depths, and adds an auxiliary SFT anchor for stability.Source: Conclusion · p. 10
• Under the frozen SFT checkpoint, the exact target is absent from the first 16 of 50 constrained-ranking candidates for many prompts, and more than half of public test prompts miss the first target SID token.Such target-missing patterns in on-policy groups can lead to weak or degenerate group-relative rewards during GRPO-style post-training.Source: Conclusion · p. 10

Reproducibility Overview

• The reproducibility overview distinguishes what is fixed across runs from what is included in the public release.Source: Appendix Organization · p. 10

Dataset and Semantic-ID Details

• Dataset and Semantic-ID Details describe the public and internal tasks and how fixed SID trees define the generation space.Source: Appendix Organization · p. 10

Training and Evaluation Protocol

• Training and Evaluation Protocol records SFT initialization, GRPO hyperparameters, DASO rollout construction, baselines, and metrics.Source: Appendix Organization · p. 10

Ablation Study Details

• Ablation Study Details report the controlled ablations.Source: Appendix Organization · p. 10

Diagnostic Bucket Construction

• Diagnostic Bucket Construction defines the Easy, Medium, and Hard buckets used only for analysis.Source: Appendix Organization · p. 10

Full Diagnostic Bucket Results

• Full Diagnostic Bucket Results provide complete bucket-level Amazon results and explain how they support the main-paper observations.Source: Appendix Organization · p. 10

Scope of the Appendix

• The appendix supplies lower-level experimental information to interpret the main paper’s motivation and aggregate results.Source: Scope of the Appendix · p. 10
• The appendix records fixed SID assignments and training stages for interpreting the reported experiments.Source: Scope of the Appendix · p. 11
• It documents decoding constraints and the guided-rollout budget.Source: Scope of the Appendix · p. 11
• It describes how diagnostic buckets are constructed.Source: Scope of the Appendix · p. 11
• It specifies the controlled ablation variants.Source: Scope of the Appendix · p. 11
• It includes all bucket-level Amazon tables.Source: Scope of the Appendix · p. 11

Run Protocol

• All methods start from the same SFT checkpoint, with fixed chronological splits and SID assignments, isolating differences to post-training recipes.Reported results are single runs per dataset, backbone, and method setting.Source: Run Protocol · p. 11
• Fixed seeds and a shared valid-SID trie constrain rollout and evaluation to ranking within the valid catalog, not syntactic validity.Because rollout collection is computationally expensive, the paper reports no multi-seed significance tests and instead provides protocol and diagnostic breakdowns.Source: Run Protocol · p. 11

• Public experiments use Amazon Reviews 2018 categories and released MiniOneRec-style SID assignments, with code for permitted preprocessing, SID loading, rollout construction, training, and evaluation.The public data and SID mappings are sufficient to reproduce the open-source benchmark setting.Source: Data and Code Availability · p. 11
• The proprietary industrial dataset and data-access code are withheld, while aggregate task, user, prompt, and SID-space statistics are reported without identifying information or raw logs.These statistics describe the industrial task’s depth, width, sparsity, and evaluation scale.Source: Data and Code Availability · p. 11

Dataset and Semantic-ID Details; Public Amazon Benchmarks; Internal Industrial Benchmark; Why SID Structure Matters

• Public evaluation uses 5-core chronological next-item prediction with fixed three-level SIDs; the internal task tests four-level SIDs with 2048 codes per level.Both settings use constrained SID decoding, and DASO relies on item identifiers forming paths in a fixed semantic tree rather than dataset-specific difficulty labels. The public methods share the same SID-to-item mapping and trie constraints; internal statistics are reported in privacy-preserving aggregate form.Source: Public Amazon Benchmarks · p. 11, Internal Industrial Benchmark · p. 12, Why SID Structure Matters · p. 12

Model Initialization and SFT; Evaluation Metrics; Compute Resources

• Experiments evaluate Qwen2.5-1.5B-Instruct and Qwen2.5-3B-Instruct using full-parameter SFT (cutoff 512, batch 128, bfloat16, cosine schedule, validation early stopping) on 8 NVIDIA A100 80GB GPUs.Internal evaluation reports level-wise Top-20 and Top-1 recall, measuring whether generated candidates match the target SID prefix at each depth.Source: Model Initialization and SFT · pp. 12–13, Evaluation Metrics · p. 14, Compute Resources · p. 19

Training and Evaluation Protocol; GRPO-Stage Training

• DASO samples a raw group, then, unless it already contains the full target SID, profiles prefix depth and replaces weakest completions with guided replacements at bottleneck depths (Bmax = 9), preserving group size.It adds SID-prefix reward m(predicted SID, target SID)/M with unit weight before group-wise reward normalization and uses an SFT anchor λsft = 0.1.Source: GRPO-Stage Training · p. 13

Baselines

• DASO’s comparisons include SFT-only, MiniOneRec-style GRPO, uniform ground-truth injection, and Sibling-GRPO, with framework-level comparisons not isolating prompt-specific profiling alone.MiniOneRec + GT tests uniform positive injection, while Sibling-GRPO uses sibling relations; DASO allocates guidance by each prompt group’s current prefix-depth profile.Source: Baselines · p. 14

Ablation Study Details; Static Difficulty Variant; Removing the SID-Prefix Reward

• Static DASO falls below Full DASO on all reported aggregate HR@5 and NDCG@5 metrics, and removing SID-prefix reward consistently worsens aggregate Amazon metrics in these runs.The static variant uses frozen SFT buckets and fixed intervention recipes; the reward ablation removes Rsid while retaining online allocation, Bmax = 9, λrank = 0.3, and λsft = 0.1.Source: Static Difficulty Variant · p. 15, Table 6 · p. 15, Table 7 · p. 15

Guided-Budget Sensitivity; Removing the SFT Anchor; Interpretation of Easy-Bucket Results

• Bmax = 9 gives the strongest aggregate result in the tested Office Products 1.5B budget sweep, while removing the SFT anchor lowers aggregate HR@5.Easy-bucket effects without the anchor are mixed, so the paper treats that diagnostic as inconclusive by itself. Easy prompts were already exact-hit cases under frozen SFT; bucket analysis indicates DASO’s largest gains usually occur on Medium and Hard prompts, while the buckets remain an evaluation lens rather than a training controller.Source: Guided-Budget Sensitivity · p. 16, Table 9 · p. 16, Removing the SFT Anchor · p. 16

Diagnostic Bucket Construction

• More than 63% of Office Products test prompts are initially Hard for both backbones; because buckets derive from the frozen-SFT ranked list, SFT-only HR@5 and NDCG@5 are zero on Medium and Hard subsets by construction.Buckets use the first 16 candidates from 50-beam constrained decoding under the frozen SFT checkpoint and are fixed before methods are compared; they are used only for evaluation analysis.Source: Diagnostic Bucket Construction · p. 17, Table 11 · p. 17

Full Diagnostic Bucket Results

• On 1.5B, DASO raises HR@5 on Office Products Medium from 0.1224 to 0.1693 and Hard from 0.0331 to 0.0624, and on Industrial and Scientific Hard from 0.0305 to 0.0490.On Industrial and Scientific Medium, HR@5 rises from 0.1223 to 0.1320. For 3B, DASO raises Industrial and Scientific Hard HR@5 from 0.0294 to 0.0577; MiniOneRec is slightly higher on Medium NDCG@5, while DASO is higher on Medium HR@5.Source: Full Diagnostic Bucket Results · p. 18, Table 13 · p. 18, Table 14 · p. 19