Source-linked AI summary
PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations
Chengcheng Guo, Kuo Cai, Yu Zhou, Qiang Luo, Ruiming Tang, Han Li, Kun Gai, Guorui Zhou
TL;DR
Generative recommendation can suffer from Semantic Drift because exposure-biased generation accumulates early errors in hierarchical Semantic IDs. Promise adds a path-level PRM and PRM-guided Beam Search, with experiments and online A/B tests showing improved recommendation performance and test-time scaling in which smaller models can outperform larger ones.
Problem
Exposure bias leaves generative recommenders unprepared for erroneous intermediate states, allowing early Semantic ID errors to accumulate and divert generation.
Method
Promise combines a lightweight path-level Process Reward Model with PRM-guided Beam Search to evaluate intermediate steps and prune low-quality branches.
Results
Promise improves recommendation precision and enables smaller models to outperform larger ones as inference search width increases, with effectiveness confirmed on public benchmarks and industrial A/B tests.
Takeaways & Limitations
Process supervision provides an efficient test-time scaling direction for industrial-scale recommender systems, including lower serving cost than scaling model parameters.
Takeaways & Limitations
Gains on public benchmarks may not fully reflect practical value because industrial environments have larger data scales, longer user sequences, and more severe exposure bias.
Abstract
from arXiv · showhide
Generative Recommendation has emerged as a promising paradigm, reformulating recommendation as a sequence-to-sequence generation task over hierarchical Semantic IDs. However, existing methods suffer from a critical issue we term Semantic Drift, where errors in early, high-level tokens irreversibly divert the generation trajectory into irrelevant semantic subspaces. Inspired by Process Reward Models (PRMs) that enhance reasoning in Large Language Models, we propose Promise, a novel framework that integrates dense, step-by-step verification into generative models. Promise features a lightweight PRM to assess the quality of intermediate inference steps, coupled with a PRM-guided Beam Search strategy that leverages dense feedback to dynamically prune erroneous branches. Crucially, our approach unlocks Test-Time Scaling Laws for recommender systems: by increasing inference compute, smaller models can match or surpass larger models. Extensive offline experiments and online A/B tests on a large-scale platform demonstrate that Promise effectively mitigates Semantic Drift, significantly improving recommendation accuracy while enabling efficient deployment.
1 Introduction
Generative recommendation uses hierarchical Semantic IDs but suffers from Semantic Drift when early token errors divert generation into irrelevant semantic subspaces. Promise addresses this with process-level verification and PRM-guided search, enabling inference-compute scaling that can let smaller models outperform larger ones.
- Generative recommendation reformulates recommendation as sequence-to-sequence generation over item identifiers, supporting end-to-end preference modeling and scaling.
- Semantic Drift occurs when accumulated prediction errors, especially in high-level tokens, irreversibly move generation into irrelevant semantic subspaces.Semantic IDs use coarse-to-fine structure, so early deviations can misroute all later predictions.
- Unlike reasoning systems with process verification, SID-based generative models lack intermediate-step evaluation to detect and stop Semantic Drift before it becomes irreversible.
- Promise integrates a lightweight path-level PRM with PRM-guided Beam Search to supervise intermediate steps and prune low-quality branches.The PRM supplies dense feedback while search explores higher-potential semantic subspaces.
- Increasing search width enables smaller models to outperform larger models, creating a latency–quality trade-off for efficient recommendation deployment.
- The framework is supported by extensive offline experiments and online A/B tests reporting improvements in core business metrics.
2 Preliminary
The preliminary formulation represents items with hierarchical Semantic IDs and generates them autoregressively from user history and context. Exposure bias causes early errors to accumulate, while coarse-to-fine quantization makes these errors capable of producing irreversible semantic drift.
- Semantic IDs: Each item embedding is quantized into a d-layer Semantic ID with codebook size M.
- Recommendation objective: The recommendation objective maximizes the conditional probability of the next interacted item given user history, user identity, and context.
- Inference: Beam search retains the top-K most probable paths at each generation step.
- Generative Backbone: The decoder predicts each next Semantic ID token using prefix self-attention, encoder cross-attention, feed-forward layers, and codebook matching.
- Next-token Prediction: Next-token prediction trains on ground-truth histories but infers autoregressively from generated prefixes, creating exposure bias.
- Semantic Drift: Semantic Drift is error propagation in hierarchical IDs: deviations in coarse early tokens can irreversibly divert later fine-grained generation into irrelevant subspaces.
3 Methodology
Promise integrates a path-level Process Reward Model with generative recommendation to evaluate intermediate Semantic ID paths and guide test-time beam search. The method uses positive and negative paths, joint optimization, and lightweight path scoring to reduce semantic drift while expanding inference search.
- Framework: Promise combines an encoder-decoder generator with a path-level PRM to address error propagation in Semantic ID generation.The encoder-decoder generates Semantic ID paths, while the PRM evaluates intermediate paths for relevance.
- Path Construction: Positive Semantic ID paths are defined as prefixes of the ground-truth item path at each depth.A path prefix is positive because retrieving the complete path requires retrieving its corresponding prefix.
- Inference Strategy: PRM-guided beam search expands the candidate set to K+ paths, scores them, and retains a smaller top-K′ subset for subsequent decoding.When K′ equals the original beam size K, the method increases candidate exploration without passing all expanded paths forward.
- Negative Sampling: The PRM learns to distinguish relevant and irrelevant paths by training on positive paths alongside sampled negative paths.Negative sampling exposes the model to paths that do not correspond to the relevant user-item combination.
- Reward Model: The PRM maps a user, context, and candidate Semantic ID sequence to a relevance score, independently of the architecture implementing the mapping.The scoring function is defined for intermediate paths of varying depth and is architecture-agnostic.
- PRM Architecture: The lightweight PRM reuses encoder features and applies cross-attention to score many candidate paths efficiently.Candidate path representations interact with the final encoder output before an MLP produces a scalar reward score.
4 Experiments
Promise is evaluated on public benchmarks and industrial-scale data using standard sequential and generative recommendation baselines. The reported results show stronger performance for Semantic-ID-based generative methods and state-of-the-art results for Promise, while industrial evaluation addresses the limited realism of public datasets.
- Experimental Setup: Experiments evaluate Promise against traditional sequential and generative recommendation baselines on public benchmarks and industrial-scale data.The study also examines semantic drift, PRM contributions, test-time scaling, and real-world user feedback.
- Datasets: The public-dataset experiments use Beauty and Sports and Outdoors subsets of Amazon Reviews with the preprocessing protocol of TIGER.Users with fewer than five reviews are filtered out.
- Metrics: Promise is evaluated with Recall@k and NDCG@k at k=5 and k=10.These metrics follow the evaluation setup used by prior generative recommendation work.
- Public Results: Semantic-ID-based generative approaches outperform traditional sequential methods in the reported public-dataset results.The paper attributes this pattern to their coarse-to-fine generation scheme.
- Industrial Results: Promise significantly outperforms all baselines on the industrial-scale dataset according to a paired t-test with p<0.05.Table 2 identifies the best and second-best results using bold and underlining, respectively.
- Public Results: Promise achieves superior results across datasets and metrics against the best baseline, ActionPiece.The paper attributes the improvement to PRM-guided mitigation of semantic drift and reduced error accumulation during search.
4.2 Industrial-scale Experiments (RQ1)
On industrial-scale data, Promise is evaluated against deployed recommendation baselines and consistently improves retrieval quality, while the authors note that public-benchmark gains may not fully reflect industrial value.
- Public-dataset improvements may not fully reflect practical value because industrial settings have larger data scales, longer user sequences, and more severe exposure bias.The authors therefore conduct additional evaluations on Kuaishou.
- The industrial evaluation uses real user logs from Kuaishou, where deployed recommendation models are compared under large-scale production conditions.The broader setup includes six publicly published models deployed in industrial environments.
- Promise consistently outperforms all baselines, with Recall@100 relative improvements of 47.92% at K+ = 4000 and 59.31% at K+ = 6000.These improvements are reported against the strongest baseline.
- At NDCG@100, Promise improves over the strongest baseline by as much as 40.47%.The larger gain at smaller retrieval size indicates stronger ranking of relevant items near the top of the result set.
4.3 Online A/B Test Result (RQ2)
Online A/B tests compare traditional beam search with PRM-guided search on two short-video applications. Promise significantly increases app usage time, while ablations show that applying the PRM across reasoning steps improves recommendation performance and mitigates error accumulation.
- Online A/B Test Result (RQ2): In a seven-day A/B test, Promise significantly increases video-watching time compared with a generative recommender using traditional beam search.Each application assigns about 5% of users to the experimental group and about 5% to control.
- Online A/B Test Result (RQ2): On Kuaishou Lite, total app usage time rises by 0.131% and app usage time per user rises by 0.160%.The differences are reported as significant at the 95% confidence level.
- Ablation Studies (RQ3 & RQ4): Applying the PRM at every Semantic ID generation step improves performance under a fixed global beam size of K = 1000.The ablation study compares no discrimination, one-step, two-step, and three-step PRM application.
- Ablation Studies (RQ3 & RQ4): Activating the PRM at more steps produces further gains, indicating that it mitigates error accumulation during Semantic ID generation.The all-three-step configuration improves over configurations using one or two active steps.
4.5 Validation of Test-time Scaling Laws (RQ5)
Promise validates test-time scaling by enlarging the candidate-path set K+ while keeping the decoder beam size fixed. The resulting gains come from lightweight PRM computation rather than additional decoder computation.
- With K+ = 6000, HRecall@3@1000 reaches 36.37%, compared with 22.98% for the traditional beam-search baseline.The global beam size remains fixed at K = 1000, so the added computation is in the lightweight path-level PRM.
- Increasing K+ improves HRecall@b@1000 for b ∈ {1, 2, 3}, with especially significant gains at the second and third generation steps.The PRM selects more optimal paths as the candidate set expands.
- Brute-force increases to the global beam size K provide only marginal metric improvements while substantially increasing decoder self-attention and cross-attention computation.Promise instead scales inference through the lightweight path-level PRM.
- The path-level PRM therefore validates test-time scaling laws for generative recommendation without increasing decoder computation.The scaling variable is the expanded candidate size K+ under a fixed decoder beam.
4.6 Analysis of Inference Efficiency (RQ6)
Promise’s test-time scaling is compared with parameter scaling using inference FLOPs and HRecall@3@1000. Test-time scaling is more efficient, while performance also increases with model size and longer user sequences.
- Test-time Scaling vs. Parameter Scaling: Test-time scaling achieves better performance at equal inference FLOPs or reaches the same performance with fewer FLOPs than parameter scaling.The comparison keeps the base architecture and K = 1000 fixed while varying K+ or model size.
- Model Parameter Size: Parameter scaling consistently improves HRecall@b@1000 for b ∈ {2, 3} as model size increases.This result is measured on the industrial dataset.
- User Sequence Length: Model performance improves steadily as user sequence length increases.The authors attribute this to encoder representations being used by both the generative task and the reward task.
5 System Deployment
Promise was deployed online with a lightweight PRM configuration and a larger inference candidate set to support PRM-guided selection.
- Online deployment: K+ = 4000 was deployed online, compared with the original target size K = 1000.The larger candidate set increases GPU Top-K computation during inference.
- Online deployment: The deployed system uses one PRM block to keep the process-reward component lightweight.
- Online deployment: Queries are processed in parallel during cross-attention, while PRM attention heads are reduced to one-quarter of the main generation module.These choices target GPU utilization and deployment efficiency.
6 Related Work
Related work spans architectures, Semantic ID tokenization, and reward mechanisms, while Promise addresses neglected exposure bias through process rewards and test-time scaling.
- Research directions: Generative recommendation research covers model architecture, item tokenization, and reward mechanisms.
- Item tokenization: Existing item-tokenization methods include RQ-VAE, Residual K-means, PQ, and FSQ.
- Reward mechanisms: Prior reward mechanisms target personalized ranking, GMV estimation, or optimization from fixed recommender feedback.
- Process rewards: Promise is presented as the first process reward mechanism for generative recommendation, addressing exposure bias in NTP optimization and enabling test-time scaling.
7 Conclusion
Promise uses process supervision to mitigate Semantic Drift in hierarchical Semantic ID generation and improve recommendation efficiency. Experiments and online A/B tests support its effectiveness in public and industrial settings.
- Conclusion: Promise combines a Path-level PRM with PRM-guided Beam Search to provide dense feedback during hierarchical Semantic ID generation.The framework targets error accumulation and exposure bias while improving recommendation precision.
- Conclusion: Test-Time Scaling Laws allow smaller models to outperform larger ones at a fraction of serving cost.
- Conclusion: Public-benchmark experiments, real-world industrial evaluations, and online A/B tests confirm Promise's effectiveness and practical value.