PaperlayerView exact revision on arXiv

Source-linked AI summary

Reinforced Preference Optimization for Recommendation

Junfei Tan, Yuxin Chen, An Zhang, Junguang Jiang, Bin Liu, Ziru Xu, Han Zhu, Jian Xu, Bo Zheng, Xiang Wang

arXiv:2510.12211v1cs.IR

TL;DR

Generative recommenders face limited high-quality negative modeling and rely on implicit rewards. ReRe adapts RLVR with constrained beam search and auxiliary ranking rewards; across three real-world datasets, it consistently outperforms traditional and LLM-based recommenders in ranking performance.

  • Problem

    Generative recommenders have insufficient high-quality negative modeling and rely on implicit rewards rather than explicit user-preference signals.

    Source: §1 · p. 2

  • Method

    ReRe adapts RLVR to LLM-based recommendation with constrained decoding, beam-search sampling, and ranking rewards alongside rule-based accuracy rewards.

    Source: §1 · pp. 2–3

  • Results

    ReRe consistently outperforms traditional and LLM-based recommenders on three real-world datasets, with substantial ranking-performance improvements.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The study systematically examines RLVR design choices for recommendation and reports robustness across base and SFT-initialized models, backbone families, and optimization algorithms.

    Source: §1 · p. 3, §6 · p. 10

    Experiments use at most 32 generations per step due to resource constraints, and cross-domain and cold-start recommendation are not investigated.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent breakthroughs in large language models (LLMs) have fundamentally shifted recommender systems from discriminative to generative paradigms, where user behavior modeling is achieved by generating target items conditioned on historical interactions. Yet current generative recommenders still suffer from two core limitations: the lack of high-quality negative modeling and the reliance on implicit rewards. Reinforcement learning with verifiable rewards (RLVR) offers a natural solution by enabling on-policy sampling of harder negatives and grounding optimization in explicit reward signals. However, applying RLVR to generative recommenders remains non-trivial. Its unique generation space often leads to invalid or repetitive items that undermine sampling efficiency, and ranking supervision is sparse since most items receive identical zero rewards. To address these challenges, we propose Reinforced Preference Optimization for Recommendation (ReRe), a reinforcement-based paradigm tailored to LLM-based recommenders, an important direction in generative recommendation. ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives, while augmenting rule-based accuracy rewards with auxiliary ranking rewards for finer-grained supervision. Extensive experiments on three real-world datasets demonstrate that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance. Further analysis shows that ReRe not only enhances performance across both base and SFT-initialized models but also generalizes robustly across different backbone families and scales. Beyond empirical gains, we systematically investigate the design space of RLVR in recommendation across generation, sampling strategy, reward modeling, and optimization algorithm, offering insights for future research.

1 INTRODUCTION

ReRe adapts RLVR to LLM-based recommendation to address weak negative modeling and implicit rewards, while tackling recommendation’s narrow generation space and sparse ranking supervision. It combines constrained decoding, beam-search sampling, and ranking rewards, and outperforms traditional and LLM-based recommenders in experiments on three real-world datasets.

1 INTRODUCTION

  • Existing generative recommenders lack high-quality negatives and rely on implicit rewards, which can expose models to easy negatives and allow reward scores to rise while recommendation quality deteriorates.Negative items may be omitted or sampled randomly or from frozen reference policies; implicit rewards derive from relative likelihoods rather than explicit user preferences.Source: §1 · p. 2
  • RLVR offers on-policy negative sampling and verifiable rewards, but its adaptation to recommendation remains non-trivial.The paper also identifies the RLVR design space for recommendation as underexplored and not yet systematically investigated.Source: §1 · p. 2
  • Recommendation’s narrow item space leads to invalid or repetitive generations, while sparse interactions leave most sampled items with identical zero rewards and provide weak ranking feedback.These conditions reduce sampling efficiency and limit fine-grained ranking supervision.Source: §1 · p. 2
  • ReRe uses constrained decoding to generate valid items, beam search to sample distinct candidates, and ranking rewards to penalize harder negatives more heavily.The combined approach augments rule-based accuracy rewards with ranking signals and optimizes through GRPO.Source: §1 · p. 3, Figure 2 · p. 3
  • ReRe consistently outperforms traditional and LLM-based recommenders on three real-world datasets, with substantial ranking-performance improvements.The study also reports gains across base and SFT models and generalization across backbone families and scales.Source: §1 · p. 3
  • The paper systematically studies RLVR design choices across generation, sampling, reward modeling, and optimization for generative recommendation.The authors present this study as a source of insights and references for future research.Source: §1 · p. 3

2 PRELIMINARY

The preliminary section formulates recommendation as generating a user’s target item from interaction history, then reviews SFT, preference tuning, and GRPO as post-training approaches. It motivates RLVR as a way to sample negatives from the evolving policy and use explicit rewards, while noting recommendation-specific challenges.

2 PRELIMINARY

  • The section introduces the task, post-training methods, and their issues as preliminary context for the paper’s approach.It positions the formulation and method review as setup for the subsequent discussion.Source: §2 · p. 3

2.1 TASK FORMULATION

  • LLM-based recommenders generate a target item from a prompt containing a user’s chronologically ordered interaction history, with constrained token generation used to keep outputs within valid item titles.The task is defined over a predefined item set.Source: §2.1 · p. 3

2.2 POST-TRAINING OF LLM-BASED RECOMMENDERS

  • SFT trains on prompts paired with target item titles, while direct preference alignment suppresses negatives by enlarging implicit reward gaps between positive and negative items.Preference methods may use random negatives or sample them from a frozen reference model, leaving a distribution mismatch with the updated model.Source: §2.2 · pp. 3–4

2.3 GRPO

  • GRPO generates a group of candidate item titles for one user prompt and evaluates each candidate against the target with a rule-based reward.It normalizes rewards within each group to compute token-level advantages without training a separate value model.Source: §2.3 · p. 4
  • RLVR can sample negatives directly from the evolving policy and optimize with explicit verifiable rewards, but recommendation poses challenges of inefficient sampling and sparse rewards.The paper frames adapting GRPO to recommendation as treating candidate titles as generations evaluated against the target item.Source: §2.3 · p. 4

3 METHODOLOGY

ReRe addresses low sampling efficiency and coarse binary feedback in RLVR for recommendation. It adopts beam search for distinct candidates and combines rule-based correctness with a ranking reward, while also examining denser recommendation-specific signals.

3.1 SAMPLING STRATEGY

  • Generation diversity counts unique items among G generations, and higher diversity exposes the model to richer negative signals for ranking supervision.The narrow item space makes repeated sampling prone to duplicates and lowers sampling efficiency.Source: §3.1 · p. 5

3 METHODOLOGY

  • Constrained beam search and dynamic sampling address duplicate candidates in recommendation’s narrow output space, with beam search ultimately adopted to ensure distinct sampled items.Dynamic sampling over-generates candidates and requires substantially more samples; its diversity also declines as training proceeds.Source: §3.1 · p. 5, §3 · p. 5

3.2 REWARD DESIGN

  • Rule-based GRPO rewards assign 1 to the target and 0 to every negative, treating negatives equally and providing only coarse-grained ranking information.The paper therefore introduces a ranking reward that penalizes harder, higher-probability negatives more heavily.Source: §3.2 · p. 5
  • The overall reward combines rule-based correctness with ranking signals to provide finer-grained supervision among negatives.The paper states that this combined reward enhances discrimination among negatives and improves ranking performance.Source: §3.2 · pp. 5–6
  • The paper also investigates semantic-similarity and collaborative-filtering rewards as dense signals complementary to binary correctness.These rewards are explored to assess whether richer supervision can substitute for rule-based signals in RLVR recommendation.Source: §3.2 · p. 6

4 EXPERIMENTS

Across three datasets, ReRe is evaluated against traditional and LLM-based recommenders and consistently outperforms them. Further studies favor ranking rewards and beam search, show gains across backbones, and find performance improves as G increases.

4 EXPERIMENTS

  • Evaluation covers three real-world datasets using HR@K and NDCG@K against traditional and LLM-based recommendation baselines.Source: §4 · p. 6

4.1 OVERALL PERFORMANCE (RQ1)

  • ReRe outperforms traditional and LLM-based recommenders, with relative gains of 27.13%, 12.40%, and 8.95% on Amazon Toys, Amazon Industrial, and Yelp.Source: §4.1 · p. 6, Table 1 · p. 7

4.4 ANALYSIS ON HYPERPARAMETERS (RQ4)

  • Increasing generated items G from 4 to 32 consistently improves recommendation performance; G controls candidate diversity and ranking-signal granularity.Source: Figure 4 · p. 8, §4.4 · p. 9

4.2 STUDY ON RERE (RQ2)

  • Ranking reward performs best; enriching rule-based accuracy reward with ranking information improves NDCG@K by 3.11% on Industrial and 3.95% on Toys.S-DPO reward margins show similar misalignment trends.Source: Table 2 · p. 8, §4.2 · p. 8
  • Beam search ranks best, needs fewer generations than dynamic sampling, and maintains higher diversity while dynamic and common sampling diversity declines during training.Source: Table 3 · p. 8, §4.2 · pp. 8–9

4.3 GENERALITY OF RERE (RQ3)

  • On Amazon Industrial, ReRe beats D3 by 13.52%, 18.28%, and 20.70% across Qwen2.5-1.5B, Gemma-2B, and Qwen2.5-7B, respectively.For Qwen2.5-7B, results use 8 generations because of resource constraints.Source: Table 4 · p. 9, §4.3 · p. 9

5 LIMITATION

The authors identify four limits on ReRe’s scope: generation count, dependence on backbone knowledge, untested recommendation settings, and focus on LLM-based recommenders.

5 LIMITATION

  • ReRe was explored with at most 32 generations per step because of resource constraints, leaving higher settings for future work.The authors also report that performance scales with the number of generations.Source: §5 · p. 9
  • ReRe performs suboptimally in domains underrepresented in backbone pretraining, while in-domain SFT alleviates this at extra cost.More efficient transfer remains to be explored.Source: §5 · p. 9
  • Cross-domain recommendation and cold-start scenarios were not investigated and are left for future study.Source: §5 · p. 9
  • The study focuses on LLM-based recommenders, leaving semantic ID–based generative recommenders for future exploration.Source: §5 · p. 9

6 CONCLUSION

ReRe is a reinforcement-based post-training paradigm for LLM-based recommenders that combines beam search with ranking rewards. Across three real-world datasets, it surpasses traditional and LLM-based baselines, with further analyses reporting robustness across models and algorithms.

6 CONCLUSION

  • ReRe consistently surpasses traditional and LLM-based baselines on three real-world datasets in ranking performance.Further analyses report robustness across base and SFT-initialized models, backbone families, and optimization algorithms.Source: §6 · p. 10
  • ReRe combines beam search for harder, more diverse negatives with ranking rewards for finer-grained, verifiable supervision.Source: §6 · p. 10
  • The authors present ReRe as a foundation for future research on adapting reinforcement learning to recommendation, a largely under-explored area.Source: §6 · p. 10

ETHICS STATEMENT

The section states that the authors see no ethical concerns in their work and includes appendix material on dynamic sampling and constrained beam search. The appendix describes how the method selects training items and generates valid, distinct outputs.

ETHICS STATEMENT

Appendix

  • The appendix’s dynamic sampling algorithm selects items until the set reaches the target count, then samples randomly if needed.Source: Appendix · p. 17

A.1 DYNAMIC SAMPLING

  • Dynamic sampling first generates ⌊3G/2⌋ items, then selects G for optimization, prioritizing the target item and diverse negatives.Source: §A.1 · p. 17

A.2 CONSTRAINED BEAM SEARCH

  • Constrained beam search combines constrained token generation with beam search to produce valid, distinct items during training and inference.Source: §A.2 · p. 17, §A.2.2 · p. 18

A.2.1 CONSTRAINED TOKEN GENERATION

  • A hash map built from item-title sequences identifies valid next tokens, and masking invalid-token logits restricts generation to valid items.The map represents tokens that can continue a title prefix; invalid tokens receive a large negative logit so their probabilities approach zero after softmax.Source: §A.2.1 · pp. 17–18

A.2.2 BEAM SEARCH

  • Beam search iteratively expands sequences and returns B distinct generations at the maximum response length.Source: §A.2.2 · p. 18

A.3 GRADIENT ANALYSIS

The gradient analysis links reward assignments to the granularity of GRPO supervision. The experiments describe dataset preprocessing and identify traditional and LLM-based baseline groups.

A.3 GRADIENT ANALYSIS

  • Assuming equal rewards for all negatives ignores their varying difficulty, yielding only coarse-grained supervision because gradient weights depend on reward-based advantages.Source: §A.3 · p. 18

B.1 DATASETS

  • The experiments use three real-world datasets, filter users and items with fewer than five interactions, and split chronologically into training, validation, and test sets at 8:1:1.Interaction windows vary by dataset, and maximum interaction-sequence length is 10 for all models.Source: §B.1 · p. 19

B.2 BASELINES

  • The baseline comparison includes traditional models and LLM-based recommenders, including GRU4Rec, Caser, SASRec, TIGER, BigRec, D3, S-DPO, and SPRec.Source: §B.2 · p. 19

B.3 IMPLEMENTATION DETAILS

The paper reports implementation settings and evaluation measures, reviews generative recommendation and reinforcement learning for LLMs, and analyzes sampling strategies on Amazon Toys. Beam search is associated with more stable generation diversity and richer ranking information.

B.3 IMPLEMENTATION DETAILS

  • Reinforcement learning data use batches of 512; ReRe trains for two epochs with β = 1e−3 and 16 generated items.The authors report approximately four hours of ReRe training on Amazon Toys.Source: §B.3 · p. 20

B.4 EVALUATION METRICS

  • HR@K and NDCG@K are evaluated at K = 3, 5, and 10, with constrained token generation for LLM-based recommenders.Length normalization is eliminated for D3, S-DPO, SPRec, and ReRe.Source: §B.4 · p. 20

B.5 DENSE REWARDS

  • Semantic rewards use text-embeddings-3-large encodings, while collaborative rewards use SASRec logits for each generated item.The paper also presents Figure 6 as a comparison of reward consistency and NDCG@10.Source: §B.5 · p. 20, Figure 6 · p. 21

C.1 GENERATIVE RECOMMENDATION

  • Generative recommendation work represents items with semantic IDs and generates candidates with transformer models, typically trained using supervised fine-tuning.Prior work also studies semantic-ID encoding and decoding strategies.Source: §C.1 · pp. 20–21

C.2 REINFORCEMENT LEARNING FOR LLM

  • LLM recommendation research includes in-context use, supervised fine-tuning, integration of collaborative information, and preference-alignment stages.RLHF fits a reward model from human preferences, while DPO and related direct-alignment methods modify preference optimization.Source: §C.1 · p. 21, §C.2 · p. 21

D MODEL STUDY

  • The Amazon Toys model study focuses on sampling strategy and reward design, with Table 6 reporting sampling-strategy performance.The supplied table caption identifies the comparison scope but gives no values.Source: D MODEL STUDY · p. 21, Table 6 · p. 22

D.1 SAMPLING STRATEGY

  • Beam search maintains more stable generation diversity while injecting richer ranking information into LLM-based recommenders on Amazon Toys.The comparison is reported in Table 6 and Figure 5a.Source: §D.1 · p. 22, Table 6 · p. 22

D.2 REWARD DESIGN

On Amazon Toys, the rule-based reward outperforms semantic and collaborative rewards, and its values align with recommendation performance. The semantic and collaborative rewards show misalignment with NDCG@10.

D.2 REWARD DESIGN

  • The rule-based reward significantly surpasses semantic and collaborative rewards on Amazon Toys.The authors connect this comparison to the auxiliary ranking reward’s potential to improve fine-grained ranking.Source: §D.2 · p. 22
  • On Amazon Toys, rule-based reward aligns with recommendation performance, while semantic and collaborative rewards show misalignment with NDCG@10.S-DPO’s implicit reward aligns better on Toys than Industrial, but the NDCG changes are too minor to confirm that alignment.Source: §D.2 · p. 22

E STUDY OF OPTIMIZATION ALGORITHM

The study replaces the GRPO objective with DAPO and GSPO as representative alternatives. ReRe maintains high performance with different training algorithms.

E STUDY OF OPTIMIZATION ALGORITHM

  • The objective comparison includes DAPO, with decoupled clipping ranges and token-level policy gradients, and GSPO, with sequence-likelihood importance ratios.The paper reports these as two representative alternatives to GRPO.Source: E STUDY OF OPTIMIZATION ALGORITHM · pp. 22–23
  • ReRe maintains high performance with different training algorithms, indicating generality across the tested objectives.The authors identify recommendation-tailored RL losses as a promising research direction.Source: E STUDY OF OPTIMIZATION ALGORITHM · p. 23, Table 8 · p. 23

F ANALYSIS ON β

The analysis varies β and reports a trade-off between KL restrictions and aggressive updates. The authors also disclose their use of LLMs for writing and programming assistance.

F ANALYSIS ON β

  • Smaller β generally adapts better, but values that are too high restrict model updates and values that are too low can make learning over-aggressive and counterproductive.The analysis varies β across {1e−1, 1e−2, 1e−3, 1e−4}.Source: F ANALYSIS ON β · p. 23

G USE OF LLMS

  • LLMs were used to polish paper sentences and generate code snippets, with authors reviewing revisions and retaining control of implementation.The authors state that generated code was adapted and integrated into their framework.Source: G USE OF LLMS · p. 23

Related papers

Loading 2510.12211v1…