PaperlayerView exact revision on arXiv

Source-linked AI summary

SPRec: Self-Play to Debias LLM-based Recommendation

Chongming Gao, Ruijun Chen, Shuai Yuan, Kexin Huang, Yuanqing Yu, Xiangnan He

arXiv:2412.09243v3cs.IR

TL;DR

DPO can bias LLM recommendations toward a few items, worsening filter bubbles and user experience. SPRec alternates SFT with DPO using the model’s previous outputs as negative samples, adaptively suppressing biased items. Experiments report improved recommendation accuracy and fairness.

  • Problem

    DPO aligns LLMs with user preferences but can bias recommendations toward a few items, exacerbating filter bubbles and degrading user experience.

    Source: §1 · p. 2

  • Method

    SPRec alternates SFT on offline interaction data with DPO that uses outputs from the previous iteration as negative samples and reweights loss using model predictions.

    Source: §1 · p. 2

  • Results

    SPRec improves recommendation accuracy and fairness in experiments on public datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SPRec places bias mitigation in the tuning mechanism rather than treating it as a post-hoc correction.

    Source: §7 · p. 8

    The work primarily addresses bias in DPO-based tuning and leaves popularity bias already present in SFT unaddressed.

    Source: §7 · p. 8

Abstract

from arXiv · show

Large language models (LLMs) have attracted significant attention in recommendation systems. Current work primarily applies supervised fine-tuning (SFT) to adapt the model for recommendation tasks. However, SFT on positive examples only limits the model's ability to align with user preference. To address this, researchers recently introduced Direct Preference Optimization (DPO), which explicitly aligns LLMs with user preferences using offline preference ranking data. However, we found that DPO inherently biases the model towards a few items, exacerbating the filter bubble issue and ultimately degrading user experience. In this paper, we propose SPRec, a novel self-play framework designed to mitigate over-recommendation and improve fairness without requiring additional data or manual intervention. In each self-play iteration, the model undergoes an SFT step followed by a DPO step, treating offline interaction data as positive samples and the predicted outputs from the previous iteration as negative samples. This effectively re-weights the DPO loss function using the model's logits, adaptively suppressing biased items. Extensive experiments on multiple real-world datasets demonstrate SPRec's effectiveness in enhancing recommendation accuracy and fairness. The implementation is available via https://github.com/RegionCh/SPRec

1 Introduction

LLM recommenders use SFT and DPO to align recommendations with user preferences, but DPO can concentrate recommendations on similar or popular items and create filter bubbles. SPRec uses self-play to suppress bias without additional data or expert knowledge, and experiments report improvements in accuracy, diversity, and fairness.

1 Introduction

  • DPO can overrepresent items with common title tokens or popularity, producing homogeneous recommendations and filter bubbles that may degrade user experience.The examples include movies with “the” in their titles and popular series such as Batman.Source: §1 · p. 2
  • Prior bias-mitigation methods may depend on carefully crafted rules or external knowledge, limiting their applicability to general recommendation systems.Source: §1 · p. 2
  • SPRec alternates SFT on offline positive samples with DPO using prior-iteration model outputs as negatives, reweighting loss to penalize highly ranked items.The framework requires neither additional data nor expert knowledge.Source: §1 · p. 2
  • Experiments validate that SPRec improves accuracy, diversity, and fairness, while ablations indicate that self-play negative samples contribute significantly to the improvements.Source: §1 · p. 2

2 Related work

Related work covers LLM recommendation paradigms, bias and fairness concerns, and self-play as an iterative learning mechanism. The paper identifies DPO bias in LLM recommenders and proposes self-play as a mitigation that needs neither prior knowledge nor additional models.

2.1 LLMs for Recommendation

  • LLM recommenders have been used as decision makers, contextual assistants, or user simulators, with work progressing from prompting to SFT and DPO post-training.Source: §2.1 · p. 2
  • The paper identifies DPO's tendency to favor frequently exposed items as a previously overlooked source of bias in LLM recommendation.Source: §2.1 · p. 2

2.2 Biases in Recommender Systems

  • Prior work describes recommender biases including popularity, selection, and position bias, while LLM recommenders can also inherit social biases related to gender and race.Existing mitigation approaches often use predefined target distributions or external guidance; SPRec instead modifies the tuning process without prior knowledge or additional models.Source: §2.2 · p. 2

2.3 Self-Play Mechanism

  • Self-play addresses whether algorithms can improve iteratively without additional data, drawing on the idea of generating training data from prior iterations and learning a new policy.Prior LLM work describes self-play as interacting with instances of the model itself.Source: §2.3 · p. 3

3 Preliminary

The preliminary section introduces SFT and DPO for recommendation and frames bias and fairness as a mismatch between ground-truth and model-predicted preference distributions. It also identifies MGU and offline category distributions as the evaluation approach.

3 Preliminary

  • The section previews LLM alignment technologies for recommendation and bias evaluation.Source: §3 · p. 3

3.1 Supervised Fine-tuning (SFT)

  • SFT fine-tunes model parameters on offline recommendation demonstrations to maximize log-likelihood over the training data.The input-output pairs represent user context and interaction history as input and the target item as output; the empirical item distribution is item popularity.Source: §3.1 · p. 3

3.2 Direct Preference Optimization (DPO)

  • DPO uses prompts with preferred and rejected answers to encourage higher model probabilities for preferred items than less desirable ones.In recommendation, the prompt represents user context, and preferred and rejected answers correspond to positive and negative samples.Source: §3.2 · p. 3

3.3 Evaluating Bias via Distribution Alignment

  • Bias and unfairness are framed as a mismatch between ground-truth user-preference and model-predicted preference distributions.The evaluation approximates ground-truth preferences with offline training-data category distributions and uses MGU to measure mismatch.Source: §3.3 · p. 3

4 Problem of DPO: Amplify Popularity Bias

The paper examines how DPO amplifies popularity bias, first empirically and then through a theoretical account of its loss. Its analysis links this bias to recommendation homogeneity and reduced exposure to diverse items.

4.1 Empirical Analysis

  • DPO concentrates recommendations almost entirely on the most popular group, while SPRec mitigates this amplification.In the cold-start analysis, SFT adds only slight bias toward that group; the study uses MovieLens and Goodreads.Source: §4.1 · pp. 3–4

4.2 Theoretical Analysis

  • With uniformly distributed negative samples and 0 < β < 1, DPO's loss favors popular items; as β approaches zero, its optimal policy collapses to the most popular items.The theoretical account attributes the bias to higher conditional item popularity pD(y|x).Source: §4.2 · p. 4
  • SFT's forward KL is mass-covering, whereas DPO's reverse KL is mode-seeking and focuses on distribution peaks.The paper presents this contrast as a byproduct of DPO's loss function.Source: §4.2 · p. 4
  • In recommendation, this mode-seeking bias can limit exposure to diverse items and undermine user interests through filter bubbles.Source: §4.2 · p. 4

5 Method

SPRec uses self-play to suppress DPO's learned peaks, dynamically drawing negative samples from the model's previous predictions. Its architecture repeats SFT on offline positives followed by DPO using those generated negatives.

5.1 Solution: Suppress Biases through Self-Play

  • Self-play draws negative samples from the previous iteration's predictions, making the DPO objective weighted by their probabilities and adaptively emphasizing likely biased items.Unlike predefined negatives or weights, the approach dynamically selects negatives during learning, which the paper says mitigates filter bubbles and enhances recommendation diversity.Source: §5.1 · p. 5

5.2 Architecture of SPRec

  • Each SPRec iteration constructs preference pairs from offline positives and current-model predictions, then applies SFT to positives and DPO to the pairs.The process repeats for T iterations per epoch and can extend to multiple negative samples.Source: §5.2 · p. 5
  • Adding an SFT step in every self-play iteration further enhances performance in the authors' experiments.Source: §5.2 · p. 5

6 Experiments

Across four real-world datasets, SPRec improves recommendation diversity and over-recommendation metrics over BIGRec and performs better on most fairness metrics. The experiments compare recommendation methods under a shared evaluation setup and examine where existing approaches fall short.

6 Experiments

  • The evaluation asks how SPRec compares with baselines, how its components contribute, and how negative-sampling choices affect performance.Results are reported for accuracy, diversity, and fairness.Source: §6 · p. 5

6.1.1 Datasets.

  • Experiments use MovieLens, Steam, Goodreads, and Amazon Reviews CDs and Vinyl, with chronological 8:1:1 splits and equal-sized samples for all methods.Each method uses 4,096 training interactions, 512 validation interactions, and 1,000 test interactions per dataset.Source: §6.1.1 · p. 5

6.1.3 Metrics.

  • Evaluation measures accuracy with NDCG@5 and HR@5, diversity with DivRatio, over-recommendation with ORRatio, and fairness with MGU.MGU captures category-level discrepancies between recommendations and user history.Source: §6.1.3 · p. 5

6.1.4 Baseline.

  • The comparison includes sequential SASRec and LLM-based methods: BIGRec, Re-weighting, D3, DMPO, SDPO, RosePO, and SPRec.BIGRec is the foundation for SPRec; the other LLM-based baselines use SFT- or DPO-based approaches.Source: §6.1.4 · p. 6

6.2.1 Limitations of SFT-based Methods.

  • BIGRec shows low diversity on Goodreads (DivRatio 0.142) and high concentration on Steam (ORRatio 0.391), illustrating popular-item bias in SFT-based recommendations.The reported Steam ORRatio means over 39% of recommendations are concentrated on the three most popular items.Source: §6.2.1 · p. 6

6.2.2 Limitations of DPO-based Methods.

  • Random-sampling DPO methods improve accuracy but perform poorly on diversity and fairness; RosePO's results depend on dataset semantics and generalize poorly elsewhere.The paper concludes that existing DPO-based methods fail to address fairness issues in LRSs.Source: §6.2.2 · pp. 6–7

6.2.3 Superiority of SPRec.

  • SPRec improves DivRatio and ORRatio over BIGRec across datasets and outperforms it on most fairness metrics.The authors describe its self-play framework as a general approach across diverse scenarios, unlike methods tailored to specific datasets.Source: §6.2.3 · p. 7
  • SPRec surpasses all baselines on DivRatio and ORRatio, and achieves the highest MGU on MovieLens and Goodreads and second-highest on Steam and CDs and Vinyl.It also has the best category-level unfairness results in 7 of 8 MovieLens categories.Source: Figure 4 · p. 6, Table 1 · p. 6, §6.2.3 · p. 7

6.3 Ablation Study (RQ2)

The ablations show that SFT, DPO, and self-play negative sampling each have distinct roles in SPRec. On MovieLens, more random negatives reduce performance, whereas increasing the number of self-play negatives improves diversity and fairness without changing accuracy.

6.3 Ablation Study (RQ2)

  • The ablation study compares SPRec with variants that remove SFT or DPO or replace self-play negative samples with random items.The study examines each component's contribution to the training framework.Source: §6.3 · p. 7

6.3.1 SPRec without SFT.

  • Removing SFT gives the lowest accuracy on all datasets except Steam; on MovieLens, NDCG steadily declines during training without SFT.The authors identify SFT as important for maintaining recommendation performance.Source: §6.3.1 · p. 7, Figure 5 · p. 7

6.3.2 SPRec without DPO.

  • Removing DPO leaves further SFT, which does not improve diversity or fairness; MovieLens training curves fluctuate minimally, indicating prior SFT had converged.This comparison tests whether gains come simply from adding more training data.Source: §6.3.2 · p. 7, Figure 5 · p. 7

6.3.3 Randomly sampling negative items.

  • Replacing self-play negatives with random samples fails to improve DivRatio and ORRatio on MovieLens and Goodreads and worsens fairness compared with SPRec.On CDs and Vinyl, DivRatio rises but ORRatio remains poor, with a pronounced long-tail effect.Source: §6.3.3 · pp. 7–8

6.4 Impact of Negative Samples (RQ3)

  • The negative-sample experiments vary random-sampling proportion and negative-sample count, reporting MovieLens results averaged over the last three iterations.The experiments use 1,024 training samples and otherwise retain the Section 6.2 settings.Source: §6.4 · p. 8

6.4.1 Effect of Random Sampling Ratio.

  • Replacing more self-play negatives with random negatives steadily lowers diversity and accuracy, while fairness deteriorates toward recommendations skewed to a few popular items.This trend is reported as the random-sampling proportion increases.Source: §6.4.1 · p. 8, Figure 6 · p. 8

6.4.2 Effect of Negative Sample Size.

  • Increasing self-play negative sample size keeps recommendation accuracy stable while significantly improving diversity and fairness and reducing focus on popular items.The samples are selected by beam-searching 2N items, deduplicating, and choosing the top N by predicted probability.Source: §6.4.2 · p. 8, Figure 7 · p. 8

7 Conclusion & Discussion

The work connects preference alignment with fairness-aware recommendation, arguing that conventional DPO-based tuning conflicts with equitable recommendation. SPRec redesigns the learning mechanism for bias suppression, while the paper identifies popularity bias in SFT and long-term alignment as remaining challenges.

7 Conclusion & Discussion

  • The authors argue that conventional DPO-based tuning conflicts with equitable recommendation by creating self-reinforcing popularity biases.They state that traditional debiasing approaches fail to address these biases.Source: §7 · p. 8
  • SPRec replaces post-hoc bias correction with self-regulated competition between model generations.The authors describe the framework as a blueprint for self-correcting systems aligned with user preferences and ethical constraints.Source: §7 · p. 8
  • The work addresses bias in DPO-based tuning but overlooks popularity bias already present in SFT due to cross-entropy loss.The authors identify token-to-item-to-list preference optimization and long-term alignment as open challenges requiring new datasets, benchmarks, and models.Source: §7 · p. 8

A Mathematical Derivations

The appendix derives DPO's relationship to an RLHF objective and recasts reward optimization over the catalog as optimization of a vector of latent item rewards. It then obtains gradients for that objective.

A Mathematical Derivations

  • DPO's optimal policy matches the solution to the corresponding optimization problem, allowing analysis through that objective.The passage identifies this equivalence as the basis for analyzing the DPO loss.Source: A Mathematical Derivations · p. 10
  • For a fixed context, the transformed reward vector represents latent rewards for every item in the recommendation dataset.The derivation defines the transformed reward as the exponential of the reward model output.Source: A Mathematical Derivations · p. 10
  • The derivation sets gradients to zero to obtain the optimal reward, then calculates the objective's gradient with respect to the transformed reward.It begins from the RLHF objective and uses the Bradley–Terry reward model defined in the appendix.Source: A Mathematical Derivations · p. 10

B Dataset Statistics

The datasets cover movies, books, music, and games, with varied dataset sizes and user interaction patterns. Full dataset statistics are reported, although only a subset of interaction sequences is sampled for LLM fine-tuning.

B Dataset Statistics

C Implementation Details

The LLM experiments use Llama-3.2-1B-Instruct, with SPRec run for five SFT–DPO iterations and half of the training data sampled in each iteration. SASRec uses the same training and validation datasets as the LLM-based methods.

C Implementation Details

  • SPRec uses five iterations, with each SFT and DPO phase trained for one epoch and 2,048 interactions randomly sampled per iteration.The backbone is Llama-3.2-1B-Instruct, and all baselines and SPRec use the same dataset as the SFT training phase.Source: C Implementation Details · p. 10
  • The LLM experiments are run on four RTX 3090 GPUs, each with 24GB of VRAM.Source: C Implementation Details · p. 10
  • SASRec uses the same training and validation datasets as the LLM-based methods, with embedding size 64 and dropout 0.1.Training uses randomly sampled negative items, Adam, and a learning rate of 4e-3.Source: C Implementation Details · p. 10

Related papers

Loading 2412.09243v3…