PaperlayerView exact revision on arXiv

Source-linked AI summary

Spend Search Where It Pays: Value-Guided Structured Sampling and Optimization for Generative Recommendation

Jie Jiang, Yangru Huang, Zeyu Wang, Changping Wang, Yuling Xiong, Jun Zhang, Huan Yu

arXiv:2602.10699v2cs.AIcs.LG

TL;DR

Generative recommendation faces a probability–reward mismatch because likelihood-driven decoding can prune high-reward branches and compress within-group RL signals. V-STAR combines value-guided candidate construction with sibling-relative policy optimization, and experiments report consistent improvements in reachability, diversity, and performance under strict token budgets.

  • Problem

    Likelihood-driven decoding can remove low-probability prefixes leading to high-reward items and produce candidate groups with weak reward contrast for reinforcement learning.

    Source: §1 · p. 1, §3.2 · p. 3

  • Method

    V-STAR combines Value-Guided Efficient Decoding for budgeted expansion of decisive prefixes with Sibling-GRPO for tree-structured policy optimization.

    Source: §1 · p. 2, §4 · p. 3

  • Results

    V-STAR consistently improves over strong generative baselines under strict token budgets, with analyses showing better value–reward alignment and more effective use of extra decoding compute.

    Source: §6 · pp. 8–9

  • Takeaways & Limitations

    V-STAR links candidate construction and policy optimization so that targeted exploration improves reachability and sibling-relative signals restore learning contrast under prefix coupling.

    Source: §1 · p. 2, §4 · p. 3, §6 · p. 9

Abstract

from arXiv · show

Generative recommendation via autoregressive models has unified retrieval and ranking into a single conditional generation framework. However, fine-tuning these models with Reinforcement Learning (RL) often suffers from a fundamental probability-reward mismatch. Conventional likelihood-dominated decoding (e.g., beam search) exhibits a myopic bias toward locally probable prefixes, which causes two critical failures: (1) insufficient exploration, where high-reward items in low-probability branches are prematurely pruned and rarely sampled, and (2) advantage compression, where trajectories sharing high-probability prefixes receive highly correlated rewards with low within-group variance, yielding a weak comparative signal for RL. To address these challenges, we propose V-STAR, a Value-guided Sampling and Tree-structured Advantage Reinforcement framework. V-STAR forms a self-evolving loop via two synergistic components. First, a Value-Guided Efficient Decoding (VED) is developed to identify decisive nodes and selectively deepen high-potential prefixes. This improves exploration efficiency without exhaustive tree search. Second, we propose Sibling-GRPO, which exploits the induced tree topology to compute sibling-relative advantages and concentrates learning signals on decisive branching decisions. Extensive experiments on both offline and online datasets demonstrate that V-STAR outperforms state-of-the-art baselines, delivering superior accuracy and candidate-set diversity under strict latency constraints.

1 Introduction

Generative recommendation uses autoregressive models to unify candidate generation and ranking, but likelihood-driven decoding can misalign with reward. V-STAR targets this mismatch through value-guided exploration and tree-structured reinforcement learning.

  • Generative recommendation reformulates item identifiers as hierarchical Semantic IDs, making autoregressive candidate generation computationally practical.Source: §1 · p. 1
  • Likelihood-based pruning can discard low-probability prefixes leading to high-reward items, causing insufficient exploration and advantage compression.These failures reduce both candidate reachability and the reward contrast available for RL.Source: §1 · pp. 1–2
  • VED allocates decoding compute to high-value, high-ambiguity prefixes rather than expanding the entire search tree.It uses value and uncertainty signals to identify decisive branching points under a fixed budget.Source: §1 · p. 2
  • Sibling-GRPO uses sibling groups to mitigate advantage compression among candidates sharing correlated prefixes.Source: §1 · p. 2
  • Experiments on offline datasets and online settings show consistent gains over strong generative baselines.Source: §1 · p. 2

2 Related Work

Related work develops generative recommendation with Semantic IDs, stochastic decoding, tree search, and reinforcement learning. These approaches improve particular aspects of recommendation but leave trade-offs in diversity, reward reliability, computational cost, or candidate dependence.

Generative Recommender

  • Semantic IDs represent items as short hierarchical token sequences, enabling Transformer models to generate recommendations token by token.Source: §2.1 · p. 2

Decoding Strategies

  • Likelihood-driven decoding can prune low-probability branches early, making high-reward niche items effectively unreachable.Source: §2.1 · p. 2
  • Heuristic sampling improves diversity through stochasticity but lacks an explicit reward guarantee and can produce irrelevant recommendations.Source: §2.2 · p. 2
  • Tree-search methods provide lookahead and backtracking, but their computational cost limits broad industrial recommendation use.Source: §2.2 · p. 2

Reinforcement Learning

  • Generative RL methods often treat autoregressive candidates as independent despite shared prefixes that create correlated features and rewards.Source: §2.3 · pp. 2–3

3 Preliminaries

The paper formulates Semantic-ID recommendation as conditional autoregressive sequence generation and applies GRPO to decoded candidate sets. It identifies probability-driven pruning as a source of both insufficient exploration and compressed learning signals.

Problem Formulation

  • Each item is represented by a fixed-length sequence of discrete Semantic-ID tokens generated conditionally on user context.Source: §3.1 · p. 3
  • The decoding operator maps a policy and user context to a candidate set, while ideal decoding would maximize ground-truth reward rather than model probability.Source: §3.1 · p. 3

Policy Refinement

  • Standard GRPO compares rewards within the decoded candidate set using normalized advantages based on the group mean and standard deviation.Source: §3.1 · p. 3

Probability-Driven Decoding Bias

  • Likelihood-only pruning irreversibly removes branches, including low-probability prefixes that may lead to higher-reward items.Source: §3.2 · p. 3
  • Candidate collapse narrows reward dispersion, making within-group advantages nearly indistinguishable and producing advantage compression.Source: §3.2 · p. 3

4 Methodology

V-STAR unifies value-guided candidate construction and tree-structured policy optimization to improve reachability and learning signals under constrained decoding compute. Its value model estimates downstream returns, VED allocates expansion to decisive prefixes, and Sibling-GRPO compares sibling alternatives.

Framework

  • V-STAR combines Value-Guided Efficient Decoding with Sibling-GRPO in a unified decoding-and-learning framework.VED addresses reachability, while Sibling-GRPO restores within-group learning signal under prefix coupling.Source: §4 · p. 3

Semantic-aware Value Model Learning

  • The prefix value function estimates discounted downstream return from each user-context and SID-prefix state.A lightweight value head enables frequent value queries during decoding.Source: §4.1 · pp. 3–4
  • Semantic-aware dense supervision uses candidate-item embeddings and cosine similarity to provide informative feedback for mismatched prefixes.The method combines sampled prefix buckets with ground-truth item embeddings, then trains the value function using one-step temporal-difference learning.Source: §4.1 · p. 4

Value-Guided Efficient Decoding

  • VED formulates candidate construction as budgeted set optimization and allocates expansion to decisive prefixes using value, entropy, and gated tree search.The procedure initializes a shallow likelihood-guided tree, selects paths with a UCB-style score, expands only sufficiently prioritized nodes, and backpropagates information.Source: §4.2 · pp. 4–5

Sibling-GRPO

  • Sibling-GRPO compares sibling alternatives under shared parent prefixes to concentrate policy updates on decisive branching actions.VED produces candidate groups with improved reachability and reward contrast, while Sibling-GRPO uses their structure for stable updates in a self-evolving loop.Source: §4.3 · p. 5, §4.4 · p. 6

5 Experiments

Experiments on offline and online recommendation data show that V-STAR improves accuracy, commercial metrics, diversity, value–reward alignment, and budget efficiency through value-guided decoding and tree-structured advantage optimization.

Offline Dataset

  • V-STAR achieves 4.0% higher HR@3 and 4.3% higher NDCG@10 than MiniOneRec on Industrial, plus 10.4% higher HR@3 on Office.The comparison uses recent state-of-the-art baselines under the same setting.Source: §5.2 · p. 6

Online Performance

  • V-STAR improves GMV by 1.23% and GMV-Normal by 1.87% relative to BeamSearch+GRPO in a five-day WeChat Channels A/B test.The test covered 5% of live request traffic.Source: §5.3 · pp. 6–7

Decoding Strategy

  • VED consistently achieves the best performance across datasets and metrics among beam search, top-K sampling, and VED candidate construction.The ablation fixes the trained model and varies only inference-time candidate construction.Source: §5.4 · p. 7

VED Expansion Rules

  • The joint VED expansion score combining value and entropy is consistently best, using uncertainty as an expansion gate rather than the sole priority.Value-only can miss low-prior high-reward branches, while entropy-only can explore high-uncertainty low-reward regions.Source: §5.4 · p. 7

Training Objective

  • The combined GRPO+Sibling-GRPO objective achieves the highest NDCG and HR, while Sibling-GRPO focuses gradients on decisive shared-prefix branching nodes.Standard GRPO performs worst because global normalization is dominated by inter-cluster variance and compresses intra-group advantages.Source: §5.4 · p. 7

Further Analysis

  • VED improves value–reward alignment, candidate diversity, and best-in-set reward, while outperforming beam search across matched decoding budgets.Its diversity gains indicate reduced SID redundancy and broader coverage without sacrificing candidate quality.Source: §5.5 · pp. 7–8

6 Conclusion

V-STAR addresses probability-driven candidate collapse in Semantic-ID generative recommendation by coupling budgeted value-guided exploration with sibling-relative policy optimization. Experiments report stronger performance, value–reward alignment, and compute use under strict token budgets.

Conclusion

  • V-STAR combines VED for fixed-budget candidate construction with Sibling-GRPO for structure-aware policy optimization.VED targets high-potential branching points, while Sibling-GRPO uses prefix-tree structure for sibling-relative learning signals.Source: §6 · p. 8
  • The framework improves candidate reachability and diversity without exhaustive tree search, while Sibling-GRPO stabilizes optimization under prefix coupling.The conclusion reports consistent improvements over strong generative baselines under strict token budgets.Source: §6 · pp. 8–9

Related papers

Loading 2602.10699v2…