RPSD: Retrieval-Privileged On-Policy Self-Distillation with Optimal Transport

1 INTRODUCTION

• Prior sequential recommenders use opaque item identifiers and co-occurrence statistics, limiting generalization when histories are short and providing little explanation of a recommendation.Source: §1 · p. 1
• Existing reward adaptations still share one number across all tokens within a rank or reasoning step, and their reward comes from the user being scored rather than similar users.This leaves credit-blind optimization and collaborative-blind supervision unresolved.Source: §1 · p. 2
• Temporally aware optimal transport compares user trajectories using item similarity and temporal structure, while diversity-aware selection retains complementary trajectories as privileged evidence.Source: §1 · pp. 2–3
• RPSD re-scores the student’s on-policy response using privileged knowledge, then uses the student–teacher confidence difference to reshape sequence-level advantage into token-specific credit.The teacher’s privileged knowledge combines the ground-truth item with retrieved behavioral evidence from similar users.Source: §1 · p. 3
• RPSD is presented as the first method to bring on-policy self-distillation to sequential recommendation and use cross-user behavior as privileged knowledge.Source: §1 · p. 3
• Experiments on three benchmark datasets show RPSD consistently outperforms strong state-of-the-art baselines.Source: §1 · p. 3

2.1 REINFORCEMENT LEARNING FOR LLM-BASED RECOMMENDATION

• Rank-GRPO assigns rewards per rank and RecZero uses explicit reasoning steps, but tokens within each unit still share a signal that scores recommendation correctness rather than which tokens support it.S-DPO and C-APO shape preference objectives, while RPSD instead densifies supervision to the token level using retrieved behavior.Source: §2.1 · p. 3

2.2 ON-POLICY SELF-DISTILLATION

• On-policy self-distillation uses a model as both student and teacher, with the teacher re-scoring sampled tokens under privileged context; RPSD grounds that context in retrieved behavior from similar users.Prior OPSD extensions use label- or stronger-model-based privilege, and instance-level labels can let a teacher rationalize answers without grounding them in input.Source: §2.2 · p. 3

2.3 RETRIEVAL-AUGMENTED RECOMMENDATION AND KNOWLEDGE DISTILLATION

• Unlike ReLLa and RALLRec, which feed retrieved neighbors at inference, RPSD uses similar users’ continuations only as training-time privileged signal, leaving the ordinary prompt at inference.The same model serves as teacher and student; the teacher additionally receives retrieved evidence and the ground-truth item.Source: §2.3 · pp. 3–4

3.1 PROBLEM FORMULATION

• The task predicts a user's next item from recent interaction history, with a candidate set containing the ground-truth item and sampled non-interacted items.Training also gives the teacher the ground-truth item and retrieved evidence; inference uses only the student prompt.Source: §3.1 · p. 4

3.2 RETRIEVAL-BASED PRIVILEGED EVIDENCE

• Retrieved histories are re-ranked by unbalanced entropic optimal transport, whose pairwise cost combines semantic dissimilarity with a penalty for temporally inconsistent alignments.A coarse recall stage uses recency-weighted history vectors before OT re-ranking.Source: §3.2 · p. 5
• Diversity-aware selection retains three complementary windows, each describing a user's recent items and subsequent interaction for the teacher.The selected windows form the behavioral evidence block.Source: §3.2 · p. 5

3.3 RETRIEVAL-PRIVILEGED ON-POLICY SELF-DISTILLATION

• RPSD has the same model sample recommendations as a student and re-score them as a teacher using the ground-truth item and retrieved behavioral evidence.The token-level confidence gap redistributes GRPO's response-level advantage across tokens.Source: §3.3 · p. 5

• Teacher-student token confidence differences are centered and clipped before conversion into advantage-aware token weights.The token scores compare the same sampled response under the student and privileged contexts.Source: Non-LLM · p. 6
• When the response-level advantage is positive, stronger teacher support reinforces a token; when negative, teacher-disfavored tokens receive stronger negative credit.The guidance strength is bounded, and setting λ = 0 or ϵw = 0 recovers standard GRPO.Source: Non-LLM · p. 6
• At inference, the model conditions only on the student prompt, so retrieved evidence is used during training to reshape token credit.Source: Non-LLM · p. 6

4.1 PERFORMANCE COMPARISON AGAINST SOTA METHODS (RQ1)

• RPSD performs strongly against Non-LLM and LLM-based baselines, consistently outperforming same-backbone GRPO and remaining effective with a larger candidate pool.Qwen3-4B + SFT is comparable to or weaker than strong Non-LLM baselines, while reinforcement learning improves performance; with nc = 100, RPSD remains effective as ranking ambiguity increases.Source: §4.1 · p. 7, Table 2 · p. 7

4.2 ABLATION STUDY ON THE CONTRIBUTION OF EACH PROPOSED MODULE (RQ2)

• Without retrieved evidence, RPSD still improves recommendation by reallocating token-level credit; the full model matches or outperforms that variant across all metrics.The w/o OT teacher receives only the ground-truth item, while retrieved evidence adds preference patterns that help connect the history to the target.Source: §4.2 · pp. 7–8, Table 3 · p. 7

4.3 ABLATION STUDY ON DIFFERENT DESIGNS OF RETRIEVED EVIDENCE (RQ3)

• Diversity selection is consistently important, while component importance varies by dataset; temporal alignment and semantic similarity contribute comparably overall.Removing diversity selection causes the largest degradation on MovieLens-1M and MicroLens; recency weighting is equally decisive on Netflix, and removing OT re-ranking can underperform removing retrieved evidence.Source: §4.3 · p. 8, Table 4 · p. 8

4.4 CROSS-DOMAIN EVALUATION (RQ4)

• When trained on MovieLens, RPSD performs best across all metrics on MicroLens, but SFT remains slightly stronger than both post-training methods on Netflix.The results suggest transferability depends on compatibility between source and target domains; the similar GRPO drop indicates the Netflix result is not specific to the privileged teacher.Source: §4.4 · pp. 8–9, Figure 3 · p. 8

4.5 TRAINING DIAGNOSTICS OF RPSD (RQ5)

• During training, RPSD’s reward rises and response variability falls, while student entropy and the teacher–student gap decrease and KL divergence eventually stabilizes.The gap remains above zero after entropy nearly flattens, so retrieved evidence continues providing token-level signal late in training.Source: §4.5 · p. 9, Figure 4 · p. 9

4.6 PERFORMANCE ACROSS TRAINING STEPS (RQ6)

• RPSD outperforms GRPO from the first evaluated checkpoint through convergence, while SFT shows little improvement over training.The gains are larger in NG than in HR, suggesting RPSD both retrieves the target more often and ranks it higher.Source: §4.6 · p. 9, Figure 5 · p. 9

• RPSD enriches the OPSD teacher with cross-user evidence retrieved through temporally aware optimal transport and diversity-aware selection, enabling token-level supervision of target correctness and collaborative patterns.Experiments on multiple benchmarks show that RPSD consistently outperforms state-of-the-art methods.Source: §5 · pp. 9–10

• Generative AI tools polished the manuscript, and the authors reviewed the AI-assisted content and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A APPENDIX OVERVIEW

• The appendix overview covers response-level versus token-level RPSD, prompt examples, algorithms, implementation, datasets, evaluation, baselines, and training dynamics.Source: A APPENDIX OVERVIEW · p. 12
• The appendix studies training dynamics with and without OT, and compares RPSD with vanilla OPSD, reporting that token-specific credit improves over response-level weighting.The vanilla OPSD comparison reports consistent gains from retaining the reward-grounded objective while reweighting token-level credit.Source: A APPENDIX OVERVIEW · p. 12

B ALGORITHM DETAILS

• The training procedure samples student responses, scores them with rule-based rewards, then uses teacher–student confidence differences to form token-level advantages for policy updates.Source: B ALGORITHM DETAILS · pp. 12–13
• Evidence retrieval first finds candidate windows by recency-weighted cosine similarity, then reranks them with unbalanced entropic OT and greedily selects a diverse subset.Source: B ALGORITHM DETAILS · p. 13
• The OT cost combines item-title semantic dissimilarity with temporal misalignment, while marginal relaxation permits some interaction mass to remain unmatched.Source: B ALGORITHM DETAILS · p. 13

C IMPLEMENTATION DETAILS

• Implementation details specify Qwen3-4B initialization, LoRA training, sequence construction, and ten-item candidate sets with one target and nine negatives.Source: C IMPLEMENTATION DETAILS · pp. 13–15
• The retrieval setup uses three diverse evidence windows; the teacher scores the student’s sampled completions under privileged context without generating a separate response.The evidence is used only by the teacher during training and is not needed at inference.Source: C IMPLEMENTATION DETAILS · p. 15

• The benchmarks span interaction densities differing by more than two orders of magnitude, from large sparse datasets to a small, dense movie dataset.This range tests whether retrieved evidence helps when users’ own histories are short or already informative.Source: D DATASET DETAILS · p. 15
• MicroLens has 25,411 users, 41,081 items, density 2.1 × 10−4, and an average of 8.79 interactions per user.Source: D DATASET DETAILS · p. 15
• Netflix users average 5.23 interactions at density 3.0×10−4, giving it the shortest histories among the three datasets.Source: D DATASET DETAILS · p. 15
• MovieLens has density 4.2 × 10−2 and average sequence length 165.56; its small catalog and long, information-rich histories make it a setting where standard sequential models already perform well.Instances retain up to nine recent interactions, use the final item as the prediction target and preceding items as input history, and are split 7:1:2 into training, validation, and test sets.Source: D DATASET DETAILS · p. 15

E EVALUATION PROTOCOL

• Each test candidate set combines the ground-truth next item with sampled non-interacted items; the default has nc = 10, with nc = 100 also evaluated.All methods use the same data splits and candidate-construction protocol.Source: E EVALUATION PROTOCOL · pp. 15–16
• Training predicts one next item, while evaluation prompts ranked lists and reports HR@k and NDCG@k, averaged over five runs.Top-3 and Top-5 lists are generated independently; both metrics are also evaluated at k = 5 and 10 with the larger candidate set.Source: E EVALUATION PROTOCOL · p. 16
• HR@k measures whether the target appears in the top k, while NDCG@k additionally rewards placing it higher.Source: E EVALUATION PROTOCOL · p. 16

F BASELINE DETAILS

• Baselines cover identifier-based collaborative, multimodal, and LLM-based sequential recommenders, including a GRPO-posttrained LLM method.Source: F BASELINE DETAILS · pp. 16–17
• The Qwen3 SFT, GRPO, and RPSD variants share the backbone, prompt, sampling, and reward, differing only in method design.Source: F BASELINE DETAILS · p. 17

G TRAINING DIAGNOSTICS OF RPSD

• Across three datasets, rule-based reward rises without collapse, entropy falls, and KL rises initially before staying flat, indicating stable training near the reference.Source: G TRAINING DIAGNOSTICS OF RPSD · p. 17, Figure 8 · p. 18, Figure 9 · p. 18
• The teacher–student confidence gap is largest early and shrinks as the policy converges, so retrieved evidence contributes most while the student is uncertain.Source: G TRAINING DIAGNOSTICS OF RPSD · p. 18
• Token weights move toward one as the gap shrinks, automatically shifting from strong early correction to gentler guidance without a manual annealing schedule.Source: G TRAINING DIAGNOSTICS OF RPSD · p. 18
• Group agreement increases during training as responses share rewards more often, while reward variation and the teacher’s token-level correction weaken.Source: G TRAINING DIAGNOSTICS OF RPSD · p. 18

H TRAINING DYNAMICS OF RPSD W/O OT

• Without retrieved behavioral evidence, optimization remains stable, but the teacher correction is weak on denser datasets and larger yet noisy on sparse Netflix.The label-only teacher does not destabilize training or pull the policy away from the reference.Source: H TRAINING DYNAMICS OF RPSD W/O OT · p. 19
• The label-only teacher’s confidence gap depends on history density: it is small for long-history users but stays wide on sparse Netflix, where removing retrieval hurts accuracy most.Source: H TRAINING DYNAMICS OF RPSD W/O OT · p. 19

I COMPARISON WITH VANILLA OPSD

• Unlike vanilla OPSD’s teacher-distribution target, RPSD retains reward-grounded GRPO and lets the teacher modulate token-credit magnitude without reversing the reward’s update direction.RPSD’s teacher additionally receives retrieved cross-user behavioral evidence alongside the ground-truth item.Source: I COMPARISON WITH VANILLA OPSD · pp. 20–21
• RPSD surpasses both vanilla OPSD and GRPO, whereas vanilla OPSD improves over SFT but falls below GRPO on recommendation performance.RPSD keeps the reward-grounded objective and uses teacher guidance to reweight token credit rather than directly matching the teacher distribution.Source: I COMPARISON WITH VANILLA OPSD · p. 21, Table 6 · p. 23

J RESPONSE-LEVEL VERSUS TOKEN-LEVEL CREDIT

• Response-level RPSD improves over GRPO but remains well below token-level RPSD, which better distinguishes informative from generic or formatting tokens.The larger NDCG improvement further suggests fine-grained credit helps rank targets more accurately.Source: J RESPONSE-LEVEL VERSUS TOKEN-LEVEL… · p. 22, Table 7 · p. 23

K CASE STUDY OF STUDENT AND TEACHER PROMPTS

• The student sees only interaction history and candidates at inference; during training, the teacher also sees similar-user continuations with similarity scores and the ground-truth item.Source: K CASE STUDY OF STUDENT AND TEACHER… · p. 23, Figure 18 · p. 24