PaperlayerView exact revision on arXiv

Source-linked AI summary

ProRL: Effective Reinforcement Learning for Proactive Recommendation via Rectified Policy Gradient Estimation

Hongru Hou, Tiehua Mei, Denghui Geng, Jinhui Huang, Ao Xu, Hengrui Chen, Jiaqing Liang, Deqing Yang

arXiv:2605.28293v2cs.LGcs.AI

TL;DR

Proactive recommendation must guide users toward unfamiliar target items while preserving acceptance along intermediate paths, but standard policy gradients favor path extension and exhibit high variance. ProRL rectifies these estimation problems with stepwise reward centering and position-specific advantage estimation, and experiments on three real-world datasets report significant improvements over state-of-the-art methods.

  • Problem

    Proactive recommendation lacks effective policy-gradient estimation because positive-mean decomposed rewards create a length shortcut and whole-path weighting causes high gradient variance.

  • Method

    ProRL combines Stepwise Reward Centering to remove length-dependent bias with Position-Specific Advantage Estimation to reduce variance using reward decomposition.

  • Results

    ProRL significantly outperforms state-of-the-art proactive recommendation methods on three real-world datasets.

  • Takeaways & Limitations

    The rectified gradients target path quality, supporting joint optimization of path feasibility and guidance effectiveness.

  • Takeaways & Limitations

    The framework estimates acceptance probabilities with a recommender-model user simulator trained on historical interactions, and its optimization can otherwise converge to lengthy low-quality paths.

Abstract

from arXiv · show

Proactive Recommender Systems (PRSs) aim to guide user preference shift toward target items by generating paths of intermediate recommendations. Reinforcement learning (RL) provides a principled framework for optimizing such sequential decision tasks, as path rewards can naturally capture both short-term acceptance and long-term guidance effectiveness. However, naively applying policy gradients to PRS results in deficient gradient estimation. We identify two deficiencies: (1) path-level rewards decompose into step-level rewards with positive mean, creating a length-dependent bias that causes gradients to favor path extension over meaningful exploration; (2) weighting each step by the entire path-level reward ignores the decomposition structure, leading to high gradient variance. To rectify these two deficiencies, we propose an effective RL framework ProRL with two novel mechanisms for proactive recommendation. First, Stepwise Reward Centering subtracts expected rewards to neutralize length-dependent bias, ensuring that path extension yields zero expected gradient signal. Second, Position-Specific Advantage Estimation leverages the reward decomposition structure to compute step-dependent baselines, reducing gradient variance. Together, these mechanisms yield policy gradients that precisely target path quality. Our experiments on three real-world datasets demonstrate that ProRL significantly outperforms state-of-the-art PRSs. Our code is available at https://github.com/hongruhou89/ProRL.

1. Introduction

Proactive recommendation seeks to guide users from familiar preferences toward target items through acceptable intermediate paths. The paper identifies two policy-gradient deficiencies in this setting and proposes ProRL to correct them.

  • PRSs construct paths of intermediate items that bridge users’ current preferences to platform-specified target items.
  • Effective paths must jointly maintain intermediate-item acceptance and increase eventual acceptance of the target item.
  • Standard policy gradients can degenerate into nearly identical overlong paths because positive-mean step rewards make path extension dominate exploration.
  • A second deficiency arises because weighting every step by the entire path reward ignores that each step affects only future rewards, producing high gradient variance.
  • ProRL combines Stepwise Reward Centering, which removes the length shortcut, with Position-Specific Advantage Estimation, which reduces variance using reward decomposition.
  • Experiments on three real-world datasets report that ProRL significantly outperforms state-of-the-art methods, with ablations and cross-evaluator analysis supporting component contributions and policy generalizability.

2. Preliminaries

The paper formulates proactive recommendation as reward-guided path generation using simulator-estimated acceptance probabilities. Its preliminary analysis shows that standard policy gradients systematically favor maximum-length paths because step rewards remain positive.

  • The task generates a bounded path of intermediate items from a user’s interaction history toward a target item.
  • A user simulator trained on historical interactions estimates item-acceptance probabilities, enabling reward computation without online feedback.
  • Path quality combines Guidance Effectiveness and Path Feasibility through IoI, IoR, and CTR, with reward Rpath = α · IoI + β · IoR + γ · CTR.
  • Standard policy-gradient optimization fails systematically across datasets and reward designs, despite providing a framework for reward-driven exploration.
  • Across reward configurations, path length rapidly reaches the maximum while path diversity collapses toward nearly zero, producing nearly identical paths.
  • Positive expected step rewards make expected path reward depend on length, creating an incentive for longer paths; this expectation is conditioned on reaching step t.
  • The length-collapse analysis predicts stopping probability p(s) decreases at O(1/s), with expected path length converging to Lmax.
  • The proposed rectification principle is that path extension should yield zero expected gain, so gradients instead reflect path quality.

3. Methodology

The methodology section introduces the policy-gradient deficiencies that motivate rectification. It points to the length shortcut and high variance as the central estimation problems.

  • The standard estimator’s length shortcut causes path length to dominate the gradient signal because decomposed step rewards have positive mean.
  • The reward decomposition also exposes high gradient variance as a second deficiency in standard policy-gradient estimation.
  • Policy Gradient Estimation Deficiencies is the section’s organizing focus.

1. Stepwise Reward Centering !!" = !! −!%

ProRL rectifies policy-gradient estimation for proactive recommendation by removing length-driven incentives and using position-adapted baselines to reduce variance. Its centered and decomposed reward treatment redirects optimization toward path quality and supports effective reinforcement learning for PRS.

  • Combined Framework: Together, the two mechanisms rectify policy-gradient estimation for proactive recommendation and target effective path exploration.The framework combines length-shortcut elimination with variance reduction rather than optimizing path length alone.
  • Stepwise Reward Centering: Stepwise Reward Centering subtracts the expected per-step reward so path extension yields zero expected gain.The method uses a global expected step-reward statistic estimated during a warm-up epoch and then frozen.
  • Motivation: Path-level rewards decompose into positive-mean step rewards, coupling expected return with path length and creating a length shortcut.This coupling biases gradients toward extending paths rather than exploring diverse alternatives.
  • Position-Specific Advantage Estimation: Position-Specific Advantage Estimation replaces total path reward with reward-to-go, excluding earlier rewards unaffected by the current action.This exploits the decomposition structure of path rewards to remove irrelevant noise from each step’s gradient signal.
  • Position-Specific Advantage Estimation: Position-specific baselines center reward-to-go around expected future return at each position, providing lower-variance estimates without a learned critic.Unlike a shared path-level baseline, each step receives its own reference point.

4. Experiments

Experiments across three real-world datasets evaluate ProRL against diverse baselines, its rectification modules, reward design, gradient estimators, and training stages. ProRL achieves strong guidance and feasibility, with stable path lengths, low gradient variance, cross-evaluator gains, and evidence that RL unlocks pretrained path quality.

  • Overall Performance: ProRL achieves the highest guidance effectiveness and path feasibility across all evaluated datasets and compared methods.It leads on IoI, IoR, CTR, and Coherence while optimizing cumulative multi-objective rewards over complete paths.
  • Overall Performance: ProRL also improves the unrewarded Coherence metric, indicating quality beyond direct optimization of the training reward.Coherence is not included in the reward function, yet ProRL substantially outperforms all baselines on it.
  • Cross-Evaluator Analysis: Cross-evaluator gains on IoI and IoR persist under an unseen GRU4Rec evaluator, supporting generalization beyond the SASRec reward model.The analysis also evaluates GRU4Rec, LightSANs, and BERT4Rec as unseen evaluators.
  • Ablation on Rectification Modules: Removing SRC produces unusually high CTR but severe IoI and IoR declines because positive-mean click rewards dominate optimization.SRC restores zero expected gain from path extension and balances optimization across objectives.
  • Ablation on Multi-Reward Design: The full multi-reward ProRL model performs best, while removing individual reward terms can degrade multiple metrics, indicating mutually reinforcing objectives.The broader ablation validates the necessity of the multi-objective design.
  • Ablation on Gradient Estimators: ProRL converges to stable 3-to-4-step paths and achieves the lowest variance, approximately 5% of RF at Epoch 1.RF collapses to length 1.5, GRPO saturates at Lmax = 10, and A2C shows unstable growth; A2C variance increases from 0.09× to 0.17×.
  • Quantitative Analysis of Training Stages: RL raises guidance effectiveness while maintaining pretrained path feasibility, and Rollout@K reveals latent pretrained potential that often matches RL performance.The analysis characterizes RL as redistributing probability mass toward high-quality paths in the low-probability tail.

5. Related Work

Related work covers sequential recommendation backbones and proactive recommendation methods based on supervision, heuristics, and large language models. These approaches respectively face limits in exploration, global optimality, or deployment cost.

  • Sequential Recommendation: Sequential recommendation progresses from recurrent and self-attention models to architectures emphasizing contextual understanding, scalability, and feature enhancement.The cited models include GRU4Rec, SASRec, BERT4Rec, LightSANs, and FEARec.
  • Proactive Recommendation: Proactive recommendation methods aim to shift preferences toward target items using supervised, heuristic, or LLM-based strategies.IRN models receptiveness, IPG jointly evaluates local feasibility and guidance effectiveness, and ITMPRec adds intention-level features.
  • Proactive Recommendation: Supervised methods cannot explore beyond historical data, heuristic methods may yield suboptimal paths, and LLM-based methods incur prohibitive deployment costs.These limitations motivate reinforcement-learning approaches that directly optimize proactive recommendation paths.

6. Conclusion

The conclusion identifies biased and high-variance policy-gradient estimation as obstacles for proactive recommendation and presents ProRL’s two rectifications. Across three datasets, ProRL outperforms state-of-the-art methods and generalizes beyond its training reward model.

  • Conclusion: Standard policy gradients in PRS suffer from a length shortcut and high gradient variance, misaligning updates with target path quality.The length shortcut favors path extension, while whole-path weighting ignores reward decomposition.
  • Conclusion: Stepwise Reward Centering removes expected gain from path extension, while Position-Specific Advantage Estimation uses reward decomposition to reduce variance.Together, the mechanisms rectify policy gradients for joint feasibility and effectiveness optimization.
  • Conclusion: Experiments on three real-world datasets show significant outperformance, and cross-evaluator analysis supports generalization beyond the training reward model.The learned guidance strategy is evaluated beyond the original reward model.

Impact Statement

The impact statement frames ProRL as an RL-based framework for proactive guidance while acknowledging that user utility and ethical standards remain important considerations.

  • Impact Statement: ProRL targets proactive assistance for exploring new interests or achieving specific goals through reinforcement-learning-based guidance.The stated focus is guidance efficiency.
  • Impact Statement: The paper acknowledges that proactive strategies should align with user utility and ethical standards to support a positive user experience.This consideration accompanies the method’s focus on guidance efficiency.

A.1. Length Collapse: Formal Statement and Proof of Theorem 2.1

The appendix formalizes length collapse under positive conditional step rewards: standard policy-gradient dynamics drive stopping probability toward zero, making paths approach the maximum length. The proof derives monotonic decay, convergence, and an O(1/s) rate, motivating reward centering.

  • Setup: The analysis simplifies recommendation to repeated continue-or-stop decisions with sigmoid stopping probability p = σ(θ), bounded stopping time τ, and fixed conditional reward means.The total return is J(θ) = Eπθ[G], with E[rt | τ ≥t] treated as parameter-independent.
  • Step 1: Express J as a function of p: The event Et = {τ ≥t} represents survival through step t−1, with probability P(Et) = (1−p)^(t−1) under the homogeneous stopping policy.Conditional step means are denoted µt = E[rt | Et] and satisfy µt ≥ µmin > 0.
  • Step 2: Show dJ/dθ < 0: Positive step-level reward means make the objective gradient with respect to stopping probability negative, so θ(s) and p(s) decrease strictly during gradient flow.The appendix establishes dJ/dθ < 0 through differentiation with respect to p.
  • Step 3: Prove p(s) →0: The decreasing stopping probability has a limit, but assuming a positive limit yields a contradiction; therefore p(s) → 0 as training time grows.The proof uses the limiting dynamics to show θ(s) → −∞ and hence σ(θ(s)) → 0.
  • Step 4: Establish the O(1/s) convergence rate: After applying the chain rule and integrating from S0, the proof obtains p(s) ≤ K/s for all sufficiently large s.The bound follows after choosing S = S0 + 1 and K = 4S/µmin.
  • Implication: The O(1/s) decay is a structural consequence of positive step-reward means, not a tuning artifact, and makes expected path length converge to Lmax.The appendix connects this result to Stepwise Reward Centering, which enforces zero-mean stepwise gains.

B.1. Implementation via Semantic IDs

ProRL implements item-level policy gradients with semantic IDs, representing each item as a token sequence while preserving item-level rewards and advantages. The appendix also describes coherent data construction, feasibility filtering, and user-centric dataset preparation.

  • Semantic ID Representation: Semantic IDs represent each item as K discrete tokens; with K = 4, the policy generates K·L + 1 tokens that decode into a path of L items.When K > 1, an item is treated as a composite action.
  • Theoretical Compatibility: The semantic-ID formulation is theoretically compatible with the item-level framework because item rewards and advantages are computed after decoding.For K = 1 the formulations are identical, while for K > 1 the item-level advantage is distributed across all K tokens.
  • Benefits of Semantic IDs: Semantic IDs reduce the action vocabulary from |I| items to |C| codebook entries and let semantically similar items share token prefixes.The representation also fits standard encoder-decoder Transformers.
  • Dataset Statistics and Splits: The experiments use MovieLens-1M, Steam, and Amazon-Book, with k-core filtering and user-centric training, validation, and test partitions.The datasets differ substantially in interaction counts, average user-sequence lengths, and item counts.
  • Goal-oriented Trajectory Mining: A feasibility oracle labels semantically coherent item transitions, and Algorithm 1 mines smooth expert trajectories by archiving paths when coherence breaks.The oracle can use structured attribute sharing or LLM-based judgments, while the mined data supplies feasible plans before optimization.
  • Item Profile Generation: Item profiles are generated with GPT-4 to normalize heterogeneous metadata and provide structured semantic descriptions for representation learning.The MovieLens-1M example prompts for attributes including genre, creators, plot, mood, awards, production quality, and soundtrack.

C. Evaluation Metrics

Evaluation measures both whether users would accept intermediate recommendations and whether paths shift interest toward target items. The section defines CTR, IoI, IoR, and Coherence and outlines comparison baselines.

  • CTR: CTR measures the average predicted interaction probability for items in a guiding sequence, serving as the path-feasibility metric.Each intermediate item is evaluated as a separate single-choice task, and sequence CTR averages the resulting probabilities.
  • IoI: IoI measures the change in predicted target-item interest after appending the influence path to the user’s original history.A positive IoI indicates increased predicted target acceptance.
  • IoR: IoR measures the change in the target item’s ranking after the influence path is appended, with positive values indicating upward movement.Rank 1 is the best position under the pretrained evaluator.
  • Coherence: Coherence measures semantic consistency by averaging correlations between adjacent items in the guiding sequence.In this setting, adjacent items correlate when they share at least one common feature.
  • Baselines: The evaluation compares sequential recommenders and proactive methods, including GRU4Rec, BERT4Rec, LightSANs, FEARec, IRN, IPG, ITMPRec, LLM-IPP, and T-PRA.Sequential baselines generate candidates and greedily re-rank them by distance to the target item.

E.2. Proactive Recommendation Methods

ProRL combines semantic-prior pretraining with reinforcement learning using feasible and guidance rewards, Stepwise Reward Centering, and Position-Specific Advantage Estimation. Experiments examine training design, ablations, pretraining maturity, and robustness to target difficulty.

  • ProRL Framework: ProRL first pretrains a T5-based sequence-to-sequence policy on Smooth-Guided paths, then applies RL to optimize path feasibility and guidance effectiveness.Semantic IDs use K = 4 tokens per item, with rewards computed after decoding and gradients distributed across item tokens.
  • Reinforcement Learning: The RL reward combines feasible and guidance components, while SRC removes length bias and PSAE assigns credit using position-dependent temporal importance.These mechanisms target length manipulation, high variance, and sparse long-sequence signals.
  • Reinforcement Learning: ProRL generates trajectories that are both reachable for users and aligned with guidance objectives.The stated framework shifts training from sequence imitation toward goal-oriented trajectory optimization.
  • Pretraining Maturity: Minimal pretraining fails to produce a meaningful policy, while Figure 6 evaluates ProRL at 1%, 33%, 66%, and 100% pretraining completion.The analysis attributes cold-start difficulty to sparse successful guidance signals in a high-dimensional action space.
  • Robustness Analysis: ProRL is evaluated under Random Selection and Filtered Selection at the 20th, 40th, and 60th willingness percentiles to vary target accessibility.Figure 7 compares ProRL with FEARec, IPG, ITMPRec, and LLM-IPP across three datasets and reports consistent robustness.

F.3.1. PERFORMANCE SUPERIORITY ACROSS DIVERSE METRICS

ProRL reports strong performance across engagement, coherence, guidance, robustness, and path-length evaluations, while data-driven reward centering avoids the instability of fixed offsets.

  • ProRL maintains high engagement and coherence, reaching CTR ≈0.89 and Semantic Coherence 0.95 on MovieLens-1M, while FEARec achieves CTRs of 0.55–0.60.
  • On Steam, ProRL achieves positive IoI and IoR scores of 1300–1500, whereas most baselines have negative IoI and typically below-200 IoR.
  • ProRL remains stable as guidance ratios increase from 20% to 60%, keeping MovieLens-1M Coherence above 0.94 while ITMPRec declines more sharply.
  • Under random targets, ProRL achieves CTR 0.547 and Coherence 0.89 on MovieLens-1M, matching or exceeding its fixed-ratio scenarios.
  • Manual fixed offsets are highly sensitive: small offsets produce maximum-length paths, large offsets collapse paths, whereas ProRL yields stable lengths of approximately 3–4 without tuning.
  • ProRL consistently outperforms baselines at each evaluated path length, indicating that its path-level results are supported by effective optimization at individual positions.
Loading 2605.28293v2…