PaperlayerView exact revision on arXiv

Source-linked AI summary

Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning

Yaochen Zhu, Harald Steck, Dawen Liang, Yinhan He, Vito Ostuni, Jundong Li, Nathan Kallus

arXiv:2510.20150v5cs.IR

TL;DR

LLM-based conversational recommenders still struggle to produce catalog-grounded, correctly formatted recommendations with strong rankings through the end of a list. ConvRec-R1 combines a Remap–Reflect–Adjust behavioral-cloning stage with rank-aware Rank-GRPO; on Reddit-v2, it converges faster and achieves higher Recall and NDCG than GRPO-style baselines.

  • Problem

    Pretrained LLMs can produce out-of-catalog items, violate required formats, and generate rankings that degrade toward the list tail.

    Source: §1 · p. 1

  • Method

    ConvRec-R1 uses Remap–Reflect–Adjust to create catalog-grounded demonstrations for warm-starting, then applies Rank-GRPO with rank-level rewards and importance reweighting.

    Source: §1 · p. 2

  • Results

    On Reddit-v2, ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The authors report that the approach shows promise for rank-structured tasks including ranking, retrieval, and other sequential decision-making problems.

    Source: §5 · p. 11

    The reward-shaping design uses Eqs. (8) and (9); more sophisticated methods, such as sliding-window constraints, are left for future work.

    Source: §2.3.3 · p. 6

Abstract

from arXiv · show

Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.

1 INTRODUCTION

LLM-based conversational recommenders can handle nuanced user preferences, but aligning them to real-world recommendation catalogs and ranked-list tasks remains challenging. ConvRec-R1 combines catalog-grounded demonstrations with rank-aware reinforcement learning, and outperforms GRPO-style baselines on Reddit-v2.

1 INTRODUCTION

  • Pretrained LLMs can recommend out-of-catalog or nonexistent items, violate required formats, and produce lower-quality rankings toward list ends.These problems are particularly severe for smaller LLMs commonly used under industrial efficiency and latency constraints.Source: §1 · pp. 1–2
  • Human demonstrations are difficult to scale for large, dynamic catalogs, while sequence-level rewards and token-level updates mismatch rank-style outputs.The mismatch can produce non-causal credit assignment, misaligned importance weighting, and unstable optimization.Source: §1 · p. 2
  • ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines on Reddit-v2.Its full pipeline combines Remap–Reflect–Adjust demonstrations with Rank-GRPO.Source: §1 · p. 2
  • Remap–Reflect–Adjust turns teacher-generated lists into catalog-grounded demonstrations, and Rank-GRPO uses ranks for reward and importance reweighting.Rank-GRPO applies reward masking and a rank-level importance ratio based on the geometric mean of item-token probabilities.Source: §1 · p. 2

2 METHODOLOGY

The methodology frames conversational recommendation as generating catalog-grounded ranked lists, then combines teacher-distilled supervised fine-tuning with reinforcement learning. Rank-GRPO uses rank-level advantages and importance weighting, while reward shaping addresses causal credit assignment and list-length errors.

2.1 PROBLEM FORMULATION

  • The task is to generate N natural-language items from a dialogue as a ranked list, ideally with every item drawn from a predefined catalog.Each rank-wise decision conditions on the dialogue and the items already generated.Source: §2.1 · p. 2

2.2 SUPERVISED FINE-TUNING

  • SFT warm-starts the recommendation policy with catalog knowledge, required formats, and ranking ability using behavior-cloning demonstrations distilled from a powerful teacher LLM.The demonstrations are built from training conversations.Source: §2.2 · p. 3

2.2.1 TEACHER-GENERATED PRELIMINARY DEMONSTRATIONS

  • Teacher-generated zero-shot lists provide preliminary demonstrations, but may contain out-of-catalog items, formatting inconsistencies, and ranking biases.The teacher’s recommendations come from its own recommendation space, which may not match the target catalog.Source: §2.2.1 · p. 3

2.2.2 THE REMAP–REFLECT–ADJUST PIPELINE

  • Remap–Reflect–Adjust maps teacher recommendations into catalog scores, adds contextual ratings, then learns item-specific biases before selecting and formatting the top-N items.Remap uses item similarity and dialogue-item similarity; Reflect asks the teacher to rate top-ranked catalog candidates; Adjust aligns scores with training-item frequencies.Source: §2.2.2 · p. 3

2.3 REINFORCEMENT LEARNING-BASED ALIGNMENT

  • The RL stage further aligns the SFT policy against structured rewards derived from users’ positive-feedback items in the catalog.The RL training data pairs conversations with the in-catalog items users rated positively afterward.Source: §2.3 · p. 4

2.3.1 VANILLA GRPO

  • Vanilla GRPO assigns a sequence-level reward to every token, so later items inherit credit from earlier recommendations, while token-level importance weights mismatch sequence-level advantages.The inherited credit is especially problematic near the list’s end, where recommendation quality can deteriorate.Source: §2.3.1 · pp. 4–5

2.3.2 RANK-GRPO

  • Rank-GRPO treats each list position as an action unit, applying rank-specific advantages and geometric-mean token probabilities for rank-level importance weights.Its rank-wise advantage estimate reuses GRPO trajectories, reducing estimation complexity from exponential to linear.Source: §2.3.2 · pp. 5–6

2.3.3 REWARD SHAPING

  • Rank-level returns retain causal contributions to an item; the exp∞ variant credits only current-item relevance, while incomplete and overflow lists receive penalties.More sophisticated sliding-window reward shaping is left for future work.Source: §2.3.3 · p. 6

3 EXPERIMENTS

On REDDIT-V2, ConvRec-R1 is evaluated through SFT and RL training dynamics, test-set baseline comparisons, and component ablations. The reported results favor the full two-stage pipeline and Rank-GRPO.

3.1 DATASETS

  • The test set contains 10,972 conversations, and SFT demonstrations are built from 25% of training conversations using Remap–Reflect–Adjust.REDDIT-V2 has 383,013 training and 9,421 validation conversations; recommendations must include release years with titles.Source: §3.1 · p. 6

3.3 EVALUATION OF THE SFT STAGE

  • SFT raises in-catalog recommendations above 99% across backbones and increases NDCG@20 over zero-shot by 30×, 3×, and 1.5× for the 0.5B, 1B, and 3B models, respectively.Validation loss plateaus around 800 steps despite steadily decreasing training loss.Source: Figure 1 · p. 7, §3.3 · p. 7

3.2 IMPLEMENTATION

  • ConvRec-R1 is evaluated with three open-source instruction-tuned LLM backbones: Qwen2.5-0.5B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct; models larger than 3B may introduce higher latency and computational overhead that challenge practical deployment.Source: §3.2 · p. 7

3.4 EVALUATION OF THE RL STAGE

  • Rank-GRPO converges faster and achieves higher validation NDCG than GRPO and GSPO off-policy, with larger improvements at higher ranks.Off-policy performance at the same step lags on-policy performance, which the authors attribute to higher variance from importance weighting; trajectory reuse offers an efficiency trade-off.Source: Figure 3 · p. 8, §3.4 · p. 8

3.5 COMPARISON WITH BASELINES

  • The 3B ConvRec-R1 backbone surpasses GPT-4o and CRAG (GPT-4o) on Recall/NDCG@20; the 1B backbone performs on par with GPT-4o.The Qwen2.5-0.5B-Instruct backbone also substantially outperforms GPT-4o-mini.Source: §3.5 · pp. 8–9, Table 1 · p. 9

3.6 ABLATION STUDY

  • Removing Remap weakens catalog grounding, removing Reflect degrades ranking at larger k, and removing SFT weakens RL warm-starting; the full pipeline performs best overall.The ablation removes Remap, Reflect, or the entire SFT stage.Source: §3.6 · p. 9

4 RELATED WORK

Prior work applies language-model generation to conversational recommendation, but zero-shot systems face catalog, formatting, and ranking-tail issues, while RL for ranked outputs remains underexplored. Rank-GRPO addresses the mismatch between sequence-level rewards and rank-structured recommendation.

4.1 LLM-BASED CONVERSATIONAL RECOMMENDER SYSTEMS

  • Natural-language recommendation lists preserve language-model generalization and allow verbal guidance, but zero-shot generation can produce catalog, formatting, and list-tail quality problems.Fine-tuning and retrieval-augmented generation can mitigate these problems, but require large amounts of high-quality data.Source: §4.1 · p. 10

4.2 REINFORCEMENT LEARNING WITH VERIFIABLE REWARD

  • Rank-GRPO uses rank-level advantages and a geometric-mean item-token importance ratio to align policy updates with ranked-list outputs.Earlier recommender applications either match generated descriptions to a catalog or use sequence-level rewards for directly generated lists.Source: §4.2 · p. 10

5 CONCLUSION

ConvRec-R1 combines Remap–Reflect–Adjust demonstrations with Rank-GRPO, and on REDDIT-V2 improves base-model recommendation quality over GRPO-style baselines and large zero-shot black-box models. The appendix details the movie-focused demonstration pipeline and illustrates its intermediate outputs.

5 CONCLUSION

  • ConvRec-R1 improves base-model recommendation quality on REDDIT-V2, surpassing GRPO-style baselines and large zero-shot black-box models in Recall and NDCG.The framework pairs Remap–Reflect–Adjust demonstrations with Rank-GRPO for rank-style outputs.Source: §5 · p. 10

A DETAILS FOR THE REMAP-REFLECT-ADJUST PIPELINE

A.1 STEP 1. REMAP

  • Remap converts teacher recommendations from the teacher’s item space into scores over the target catalog.The remapped scores are built from item similarities, exact catalog matches, and conversation-item relevance.Source: §A.1 · p. 14

A.1.1 METADATA GENERATION

  • Metadata generation handles teacher-known catalog items, out-of-catalog or hallucinated items, and catalog items unknown to the teacher using different information sources.For unknown catalog items, the pipeline uses retrieval-augmented generation and in-context examples with external information.Source: §A.1.1 · p. 14

A.1.2 SIMILARITY AND SCORE CALCULATION

  • Remapping combines semantic item-to-item similarity with conversation-to-catalog-item similarity to produce catalog scores.The item metadata and conversation are embedded with the same sentence-transformer model for cosine-similarity calculations.Source: §A.1.2 · p. 15

A.2 STEP 2. REFLECT

  • Reflect reranks top remapped candidates using the teacher LLM’s contextual relevance judgments, adding scaled ratings to the remapped scores.The judgment covers more candidates than the final recommendation list; the paper uses five rating levels and sets γ to 0.5.Source: §A.2 · p. 16

A.3 ADJUST

  • Adjust learns item-specific multiplicative and additive biases to align reflected scores with positive-feedback items in the training data.The highest-scoring catalog items form the final demonstration list for supervised fine-tuning.Source: §A.3 · p. 16

A.4 EXAMPLE OF DEMONSTRATION GENERATED BY REMAP–REFLECT–ADJUST PIPELINE

  • An ocean-sailing conversation illustrates the teacher’s raw recommendations and how the ranking changes through remapping, reflection, and adjustment.The example accompanies the released behavior-cloning dataset and code.Source: §A.4 · p. 17

B GRADIENT ANALYSIS FOR GRPO, GSPO AND RANK-GRPO

The gradient analysis contrasts token-level GRPO and sequence-level GSPO with Rank-GRPO’s rank-level policy updates. Rank-GRPO treats each recommendation rank as the action unit and defines its advantage and importance ratio at that level.

GRPO

  • GRPO combines sequence-level rewards with token-level importance ratios and updates, creating reward-granularity and importance-sampling mismatches for ranked outputs.The analysis derives the gradients using the log-derivative trick after omitting the PPO-style clipping term for brevity.Source: B GRADIENT ANALYSIS FOR GRPO, GSPO AND… · p. 18

GSPO

Rank-GRPO

  • Rank-GRPO defines its objective, advantage, and importance ratio at rank level, with the item probability computed as the geometric mean of its token probabilities.The analysis presents this as a more natural granularity for credit assignment and importance weighting on rank-structured outputs.Source: B GRADIENT ANALYSIS FOR GRPO, GSPO AND… · p. 19

C IMPLEMENTATION DETAILS

The implementation appendix describes the training and inference setup, the standardized recommendation prompt, and the baseline systems used for comparison.

C IMPLEMENTATION DETAILS

  • The experiments use full-parameter fine-tuning, distributed-training and efficiency techniques, and model-specific GPU setups and learning-rate schedules.The appendix specifies ZeRO-3 offload, gradient checkpointing, FlashAttention2, and bf16 mixed precision.Source: C IMPLEMENTATION DETAILS · pp. 19–20

D.1 PROMPT USED TO GENERATE RECOMMENDATIONS

  • The recommendation prompt requests 20 standardized English movie titles with release years, one per line and without extra text.The same prompt is used across experiments.Source: §D.1 · p. 20

D.2 BASELINES USED IN THE MAIN PAPER •

  • The baselines include knowledge-enhanced conversational recommenders, zero-shot LLM generation, retrieval from similar conversations, and CRAG’s collaborative retrieval and reflection.CRAG uses an LLM-as-a-judge mechanism to refine retrieved items and their final ranking.Source: §D.2 · p. 20

D.3 ADDITIONAL COMPARISON WITH RL-FREE POST-TRAINING-BASED METHODS

The section compares generative ranking with DPO-based RL-free methods, which are primarily designed for candidate selection. The adapted DPO variants improve over zero-shot performance but remain behind Rank-GRPO.

D.3 ADDITIONAL COMPARISON WITH RL-FREE POST-TRAINING-BASED METHODS

  • DPO assumes pairwise response preferences and derives a policy-optimization objective under a Bradley–Terry model of latent reward.The passage notes that ranked-list preferences are difficult to define for DPO.Source: §D.3 · pp. 20–21
  • S-DPO extends DPO to multiple negatives, while SPRec adds self-play with negatives from prior model predictions.Both are included among the DPO-based baselines.Source: §D.3 · p. 21
  • Adapted vanilla DPO, S-DPO, and SPRec improve significantly over the zero-shot model but remain behind Rank-GRPO.The comparison is reported in Table 3 for different backbone models on REDDIT-V2.Source: §D.3 · p. 21, Table 3 · p. 21
  • Rank-GRPO generates a ranked list directly from conversational context and prior items, whereas the DPO adaptations use post-hoc ranking from item scores.The latter pipeline scores many catalog items and is slower than ConvRec-R1, even with a shared cached prefix.Source: §D.3 · p. 21

D.4 ADDITIONAL RESULTS FOR THE LLAMA3.2-1B MODEL

The additional results report Rank-GRPO comparisons for a 1B backbone, off-policy test evaluation, and catalog memorization during RL. Rank-GRPO shows stronger training dynamics and test performance than GRPO-style alternatives, while catalog drift still occurs.

D.4 ADDITIONAL RESULTS FOR THE LLAMA3.2-1B MODEL

  • With Llama3.2-1B-Instruct, Rank-GRPO rewards rise across ranks, validation NDCG improves faster than with vanilla GRPO, and gains accumulate toward list tails.The authors report these dynamics in on- and off-policy settings and state that the pattern generalizes across backbone sizes.Source: §D.4 · pp. 21–22, Figure 4 · p. 22, Figure 5 · p. 22

D.5 TEST SET EVALUATION RESULTS FOR THE OFF-POLICY SETTING

  • Under off-policy training with μ = 2, Rank-GRPO consistently outperforms GRPO and GSPO across nearly all metrics and has the largest improvements over SFT.The authors report all methods perform worse than their on-policy counterparts; Table 4 covers REDDIT-V2 test performance.Source: §D.5 · p. 22

D.6 ADDITIONAL ANALYSIS OF CATALOG MEMORIZATION DURING THE RL SETTING

  • During RL, both GRPO and Rank-GRPO drift away from the catalog, but the drift is slower for Rank-GRPO because out-of-catalog items receive zero in-position rank-level reward.The 0.5B model's in-catalog ratio fluctuates more than the 3B model's; the authors relate this to their learning-rate stabilization strategy.Source: §D.6 · p. 22, Figure 6 · p. 22

D.7 ADDITIONAL RESULTS ON THE REDIAL DATASET

The REDIAL experiments use a smaller dialogue dataset and movie catalog than REDDIT-V2, with Claude-3.7-Sonnet as an alternate teacher. The section identifies the test comparison and SFT training-dynamics figures, without supplying their numerical outcomes.

D.7 ADDITIONAL RESULTS ON THE REDIAL DATASET

  • REDIAL contains multi-turn seeker–recommender dialogues and is smaller than REDDIT-V2 in both conversation count and movie-catalog size.Its standard split has 20,896 training, 2,612 validation, and 2,613 test conversations.Source: §D.7 · pp. 22–23
  • The section includes REDIAL SFT training dynamics for loss, validation metrics, and in-catalog recommendation ratio across backbone models.Table 5 is captioned as an on-policy comparison of ConvRec-R1 and baselines on REDIAL; its caption defines R@k and N@k as Recall@k and NDCG@k.Source: Figure 7 · p. 23, Table 4 · p. 23
  • Claude-3.7-Sonnet serves as an alternate teacher for constructing the catalog-grounded behavior-cloning dataset.The authors use this teacher to demonstrate generalization across large teacher LLMs.Source: §D.7 · p. 23

D.7.1 EVALUATION ON THE SFT STAGE

On REDIAL, SFT with the Remap–Reflect–Adjust pipeline grounds recommendations in the catalog and improves validation ranking over zero-shot models. The section also identifies figures comparing RL-stage rank rewards and validation NDCG.

D.7.1 EVALUATION ON THE SFT STAGE

  • Across three backbones, the in-catalog recommendation ratio approaches nearly 100%, and validation NDCG@20 improves substantially over zero-shot models after SFT.The authors describe this SFT initialization as a warm start for the subsequent RL stage.Source: §D.7.1 · pp. 23–24
  • Figure 8 covers Rank-GRPO reward dynamics by rank and compares on-policy validation NDCG between GRPO and Rank-GRPO on REDIAL.The caption assigns rank-wise rewards to the left and middle panels and the validation NDCG comparison to the right panel.Source: Figure 8 · p. 24
  • Figure 9 covers Rank-GRPO reward dynamics by rank and an off-policy validation-NDCG comparison on REDIAL.The supplied caption identifies the comparison as off-policy but does not state its outcome.Source: Figure 9 · p. 25

D.7.2 EVALUATION ON THE RL STAGE

The RL-stage evaluations report improved ranking performance over GRPO-style baselines, including on REDIAL, while sensitivity and larger-backbone studies probe robustness and generalization. The discussion identifies the movie-focused evaluation scope and outlines ways the framework could extend to other recommendation domains.

D.7.2 EVALUATION ON THE RL STAGE

  • Across three backbones, Rank-GRPO reaches higher validation NDCG than vanilla GRPO on-policy and compares favorably with GRPO and GSPO off-policy, with larger gains at larger k.Off-policy curves lag on-policy results but improve over SFT and vanilla GRPO; importance weighting adds variance.Source: §D.7.2 · p. 25

D.7.3 SENSITIVITY W.R.T. REWARD SHAPING HYPERPARAMETER

  • Recall@k is noticeably worse without the output-length penalty; with a negative penalty, performance is robust to its magnitude, with only mild fluctuations.Figure 10 presents the Qwen2.5-0.5B-Instruct sensitivity sweep.Source: Figure 10 · p. 25, §D.7.3 · pp. 25–26

D.7.2 EVALUATION ON THE RL STAGE

  • On REDIAL, Llama-3.2-3B-Instruct + Rank-GRPO (exp∞) achieves the best overall performance, surpassing zero-shot Claude-3.7 and a Claude-3.7 RAG system at substantially lower inference cost.Source: §D.7.2 · p. 25, Table 6 · p. 26

D.7.4 GENERALIZATION TO LARGER MODELS

  • On REDIAL, Qwen2.5-7B-Instruct follows the smaller-model trend: SFT improves over zero-shot, and Rank-GRPO further boosts ranking ability.The 7B model slightly surpasses the Llama-3.2-3B-Instruct counterpart on most metrics; training it on full REDDIT-V2 was beyond the available computational budget.Source: Table 7 · p. 26, §D.7.4 · p. 26

E DISCUSSION ON GENERALIZATION OF CONVREC-R1

  • The experiments cover movie recommendations; INSPIRE is excluded because its 1,000 conversations are insufficient for reliable fine-tuning of LLMs with billions of parameters.Source: E DISCUSSION ON GENERALIZATION OF… · p. 26
  • For domains such as e-commerce or short-video recommendation, the discussion proposes representing items with pretrained textual tokens or a special-token codebook, then applying SFT and rank-wise RL.SFT can teach catalog awareness and preliminary ranking over a predefined catalog; RL can align generations with user preferences using verifiable rank-wise rewards.Source: E DISCUSSION ON GENERALIZATION OF… · p. 27

F QUALITATIVE EXAMPLES

The qualitative examples show how recommendation lists and NDCG@20 scores change from zero-shot outputs through SFT and RL for three backbones. In these examples, the processed recommendation scores are higher after training than in the corresponding zero-shot cases.

Qwen2.5-0.5B example

  • In the displayed recommendation outputs, NDCG@20 increases from 0.000 for zero-shot raw output to 0.249 after SFT at checkpoint 1500, then to 0.444 after RL at checkpoint 15800.Source: F QUALITATIVE EXAMPLES · p. 28

Llama-3.2-1B example

  • Processed recommendation NDCG@20 is 0.000 zero-shot, 0.342 after SFT (Checkpoint-1500), and 0.471 after RL (Checkpoint-15800).The RL raw output is reported at NDCG@20=0.129.Source: F QUALITATIVE EXAMPLES · p. 29

Llama-3.2-3B example

  • For the displayed recommendation example, NDCG@20 was 0.147 zero-shot, 0.385 after SFT (Checkpoint-1500; +0.238), and 0.475 after RL (Checkpoint-15800; +0.237).Source: F QUALITATIVE EXAMPLES · p. 30
Loading 2510.20150v5…