Source-linked AI summary
Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation
Jie Jiang, Yangru Huang, Zeyu Wang, Changping Wang, Yuling Xiong, Jun Zhang, Huan Yu
TL;DR
Generative recommendation faces a probability–reward mismatch because likelihood-driven decoding can prune high-reward branches and compress within-group RL signals. V-STAR combines value-guided candidate construction with sibling-relative policy optimization, and experiments report consistent improvements in reachability, diversity, and performance under strict token budgets.
Problem
Likelihood-driven decoding can remove low-probability prefixes leading to high-reward items and produce candidate groups with weak reward contrast for reinforcement learning.
Method
V-STAR combines Value-Guided Efficient Decoding for budgeted expansion of decisive prefixes with Sibling-GRPO for tree-structured policy optimization.
Results
V-STAR consistently improves over strong generative baselines under strict token budgets, with analyses showing better value–reward alignment and more effective use of extra decoding compute.
Takeaways & Limitations
V-STAR links candidate construction and policy optimization so that targeted exploration improves reachability and sibling-relative signals restore learning contrast under prefix coupling.
Abstract
from arXiv · showhide
Generative recommendation via autoregressive models has unified retrieval and ranking into a single conditional generation framework. However, fine-tuning these models with Reinforcement Learning (RL) often suffers from a fundamental probability-reward mismatch. Conventional likelihood-dominated decoding (e.g., beam search) exhibits a myopic bias toward locally probable prefixes, which causes two critical failures: (1) insufficient exploration, where high-reward items in low-probability branches are prematurely pruned and rarely sampled, and (2) advantage compression, where trajectories sharing high-probability prefixes receive highly correlated rewards with low within-group variance, yielding a weak comparative signal for RL. To address these challenges, we propose V-STAR, a Value-guided Sampling and Tree-structured Advantage Reinforcement framework. V-STAR forms a self-evolving loop via two synergistic components. First, a Value-Guided Efficient Decoding (VED) is developed to identify decisive nodes and selectively deepen high-potential prefixes. This improves exploration efficiency without exhaustive tree search. Second, we propose Sibling-GRPO, which exploits the induced tree topology to compute sibling-relative advantages and concentrates learning signals on decisive branching decisions. Extensive experiments on both offline and online datasets demonstrate that V-STAR outperforms state-of-the-art baselines, delivering superior accuracy and candidate-set diversity under strict latency constraints.
1 Introduction
Generative recommendation uses autoregressive models to unify candidate generation and ranking, but likelihood-driven decoding can misalign with reward. V-STAR targets this mismatch through value-guided exploration and tree-structured reinforcement learning.
- Generative recommendation reformulates item identifiers as hierarchical Semantic IDs, making autoregressive candidate generation computationally practical.
- Likelihood-based pruning can discard low-probability prefixes leading to high-reward items, causing insufficient exploration and advantage compression.These failures reduce both candidate reachability and the reward contrast available for RL.
- VED allocates decoding compute to high-value, high-ambiguity prefixes rather than expanding the entire search tree.It uses value and uncertainty signals to identify decisive branching points under a fixed budget.
- Sibling-GRPO uses sibling groups to mitigate advantage compression among candidates sharing correlated prefixes.
- Experiments on offline datasets and online settings show consistent gains over strong generative baselines.
2 Related Work
Related work develops generative recommendation with Semantic IDs, stochastic decoding, tree search, and reinforcement learning. These approaches improve particular aspects of recommendation but leave trade-offs in diversity, reward reliability, computational cost, or candidate dependence.
- Generative Recommender: Semantic IDs represent items as short hierarchical token sequences, enabling Transformer models to generate recommendations token by token.
- Decoding Strategies: Likelihood-driven decoding can prune low-probability branches early, making high-reward niche items effectively unreachable.
- Decoding Strategies: Heuristic sampling improves diversity through stochasticity but lacks an explicit reward guarantee and can produce irrelevant recommendations.
- Decoding Strategies: Tree-search methods provide lookahead and backtracking, but their computational cost limits broad industrial recommendation use.
- Reinforcement Learning: Generative RL methods often treat autoregressive candidates as independent despite shared prefixes that create correlated features and rewards.
3 Preliminaries
The paper formulates Semantic-ID recommendation as conditional autoregressive sequence generation and applies GRPO to decoded candidate sets. It identifies probability-driven pruning as a source of both insufficient exploration and compressed learning signals.
- Problem Formulation: Each item is represented by a fixed-length sequence of discrete Semantic-ID tokens generated conditionally on user context.
- Problem Formulation: The decoding operator maps a policy and user context to a candidate set, while ideal decoding would maximize ground-truth reward rather than model probability.
- Policy Refinement: Standard GRPO compares rewards within the decoded candidate set using normalized advantages based on the group mean and standard deviation.
- Probability-Driven Decoding Bias: Likelihood-only pruning irreversibly removes branches, including low-probability prefixes that may lead to higher-reward items.
- Probability-Driven Decoding Bias: Candidate collapse narrows reward dispersion, making within-group advantages nearly indistinguishable and producing advantage compression.
4 Methodology
V-STAR unifies value-guided candidate construction and tree-structured policy optimization to improve reachability and learning signals under constrained decoding compute. Its value model estimates downstream returns, VED allocates expansion to decisive prefixes, and Sibling-GRPO compares sibling alternatives.
- Framework: V-STAR combines Value-Guided Efficient Decoding with Sibling-GRPO in a unified decoding-and-learning framework.VED addresses reachability, while Sibling-GRPO restores within-group learning signal under prefix coupling.
- Semantic-aware Value Model Learning: The prefix value function estimates discounted downstream return from each user-context and SID-prefix state.A lightweight value head enables frequent value queries during decoding.
- Semantic-aware Value Model Learning: Semantic-aware dense supervision uses candidate-item embeddings and cosine similarity to provide informative feedback for mismatched prefixes.The method combines sampled prefix buckets with ground-truth item embeddings, then trains the value function using one-step temporal-difference learning.
- Value-Guided Efficient Decoding: VED formulates candidate construction as budgeted set optimization and allocates expansion to decisive prefixes using value, entropy, and gated tree search.The procedure initializes a shallow likelihood-guided tree, selects paths with a UCB-style score, expands only sufficiently prioritized nodes, and backpropagates information.
- Sibling-GRPO: Sibling-GRPO compares sibling alternatives under shared parent prefixes to concentrate policy updates on decisive branching actions.VED produces candidate groups with improved reachability and reward contrast, while Sibling-GRPO uses their structure for stable updates in a self-evolving loop.
5 Experiments
Experiments on offline and online recommendation data show that V-STAR improves accuracy, commercial metrics, diversity, value–reward alignment, and budget efficiency through value-guided decoding and tree-structured advantage optimization.
- Offline Dataset: V-STAR achieves 4.0% higher HR@3 and 4.3% higher NDCG@10 than MiniOneRec on Industrial, plus 10.4% higher HR@3 on Office.The comparison uses recent state-of-the-art baselines under the same setting.
- Online Performance: V-STAR improves GMV by 1.23% and GMV-Normal by 1.87% relative to BeamSearch+GRPO in a five-day WeChat Channels A/B test.The test covered 5% of live request traffic.
- Decoding Strategy: VED consistently achieves the best performance across datasets and metrics among beam search, top-K sampling, and VED candidate construction.The ablation fixes the trained model and varies only inference-time candidate construction.
- VED Expansion Rules: The joint VED expansion score combining value and entropy is consistently best, using uncertainty as an expansion gate rather than the sole priority.Value-only can miss low-prior high-reward branches, while entropy-only can explore high-uncertainty low-reward regions.
- Training Objective: The combined GRPO+Sibling-GRPO objective achieves the highest NDCG and HR, while Sibling-GRPO focuses gradients on decisive shared-prefix branching nodes.Standard GRPO performs worst because global normalization is dominated by inter-cluster variance and compresses intra-group advantages.
- Further Analysis: VED improves value–reward alignment, candidate diversity, and best-in-set reward, while outperforming beam search across matched decoding budgets.Its diversity gains indicate reduced SID redundancy and broader coverage without sacrificing candidate quality.
6 Conclusion
V-STAR addresses probability-driven candidate collapse in Semantic-ID generative recommendation by coupling budgeted value-guided exploration with sibling-relative policy optimization. Experiments report stronger performance, value–reward alignment, and compute use under strict token budgets.
- Conclusion: V-STAR combines VED for fixed-budget candidate construction with Sibling-GRPO for structure-aware policy optimization.VED targets high-potential branching points, while Sibling-GRPO uses prefix-tree structure for sibling-relative learning signals.
- Conclusion: The framework improves candidate reachability and diversity without exhaustive tree search, while Sibling-GRPO stabilizes optimization under prefix coupling.The conclusion reports consistent improvements over strong generative baselines under strict token budgets.