PaperlayerView exact revision on arXiv

Source-linked AI summary

On Softmax Direct Preference Optimization for Recommendation

Yuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang, Leheng Sheng, Enzhi Zhang, Xiang Wang, Tat-Seng Chua

arXiv:2406.09215v3cs.IRcs.AI

TL;DR

LM-based recommenders often optimize next-token prediction on positive items, rather than personalized rankings that use negative preference information. S-DPO extends DPO to multiple negatives and partial Plackett–Luce rankings; across three sequential-recommendation datasets, it outperforms the baselines and provides better rewards for preferred items.

  • Problem

    LM-based recommenders often train on positive-item responses with language-modeling loss, which does not directly optimize personalized rankings or fully use preference data.

    Source: §1 · p. 2

  • Method

    S-DPO extends DPO with multiple negative items and a Plackett–Luce loss generalized from full to partial rankings.

    Source: §1 · p. 2

  • Results

    S-DPO surpasses traditional and LM-based baselines on three sequential-recommendation datasets, with Hit Ratio@1 gains of 11.10%–47.03% over the second-best baseline and better rewards for preferred items than DPO.

    Source: §1 · p. 2, §4.1 · p. 8, §7 · p. 10

  • Takeaways & Limitations

    The authors connect S-DPO to softmax loss over negative sampling and report an inherent benefit in mining hard negatives.

    Source: §1 · p. 2, §7 · p. 10

    Experiments cap the number of negative samples at 15, and adding more negatives increases training costs, especially as their number grows.

    Source: §6 · p. 10

Abstract

from arXiv · show

Recommender systems aim to predict personalized rankings based on user preference data. With the rise of Language Models (LMs), LM-based recommenders have been widely explored due to their extensive world knowledge and powerful reasoning abilities. Most of the LM-based recommenders convert historical interactions into language prompts, pairing with a positive item as the target response and fine-tuning LM with a language modeling loss. However, the current objective fails to fully leverage preference data and is not optimized for personalized ranking tasks, which hinders the performance of LM-based recommenders. Inspired by the current advancement of Direct Preference Optimization (DPO) in human preference alignment and the success of softmax loss in recommendations, we propose Softmax-DPO (S-DPO) to instill ranking information into the LM to help LM-based recommenders distinguish preferred items from negatives, rather than solely focusing on positives. Specifically, we incorporate multiple negatives in user preference data and devise an alternative version of DPO loss tailored for LM-based recommenders, which is extended from the traditional full-ranking Plackett-Luce (PL) model to partial rankings and connected to softmax sampling strategies. Theoretically, we bridge S-DPO with the softmax loss over negative sampling and find that it has an inherent benefit of mining hard negatives, which assures its exceptional capabilities in recommendation tasks. Empirically, extensive experiments conducted on three real-world datasets demonstrate the superiority of S-DPO to effectively model user preference and further boost recommendation performance while providing better rewards for preferred items. Our codes are available at https://github.com/chenyuxin1999/S-DPO.

1 Introduction

LM-based recommenders commonly train on positive-item responses with a language-modeling objective, although recommendation is a personalized ranking task that also depends on negatives. S-DPO incorporates multiple negatives into preference data and extends DPO to softmax ranking, connecting the approach to softmax loss and hard-negative mining.

1 Introduction

  • The usual language-modeling objective predicts the next token rather than directly modeling personalized rankings, and often omits negative items from training.This leaves user preference data incompletely used for recommendation.Source: §1 · p. 2
  • S-DPO is presented as a recommendation-tailored preference-alignment stage that helps LMs distinguish preferred items from negatives rather than focusing only on positives.The framework contrasts this objective with language-modeling losses that are not tailored for recommendation.Source: §1 · p. 2, Figure 1 · p. 3
  • S-DPO pairs prompts with a positive item and multiple sampled negatives, then extends DPO through a Plackett–Luce model generalized to partial rankings.The authors describe partial rankings as a more natural fit for recommendation tasks.Source: §1 · p. 2
  • The authors connect S-DPO to softmax loss over negative sampling and report that it mines hard negatives, provides more effective ranking gradients, and gives preferred items better rewards than DPO.They also report that hard-negative mining accelerates training.Source: §1 · p. 2

2 Preliminary

This section frames sequential recommendation as aligning an LM with user preferences over candidate items, while describing current supervised fine-tuning as next-token prediction of a preferred item. It also outlines RLHF and DPO and motivates extending pairwise preference optimization to account for multiple negatives.

2 Preliminary

  • Sequential recommendation asks an LM-based recommender to select a user-preferred item from candidates using the user's chronological interaction history.The preferred item is required to outrank the other candidates.Source: §2 · p. 3
  • Current recommendation fine-tuning pairs a prompt containing interaction history and candidates with the preferred item's title, then applies language-modeling loss.The objective predicts title tokens based on preceding tokens.Source: §2 · p. 3
  • Recommendation is framed as preference alignment, unlike language modeling that considers only positive responses, motivating further alignment of LMs with user preference.The section characterizes this alignment as underexplored.Source: §2 · p. 3
  • RLHF adds reward-model training and reinforcement-learning optimization after supervised fine-tuning, using pairwise preference data.The preference-model account maps rewards to a preference distribution and optimizes the likelihood of preference data.Source: §2 · pp. 3–4
  • DPO directly extracts an optimal policy from pairwise preference data, but its pairwise design overlooks other negative items in recommendation.The section introduces S-DPO as an extension that considers multiple negatives.Source: §2 · p. 4

3 Methodology

S-DPO derives a preference loss from partial rankings of one preferred item against multiple dispreferred items, using the Plackett-Luce model. Its gradients account for the relative rewards of negatives, and its connection to softmax motivates its use for multi-negative preference alignment.

3.1 Derivation of S-DPO loss

  • S-DPO builds a loss from partial rankings in which one preferred item outranks each dispreferred item, deriving the preference distribution with the Plackett-Luce model.The resulting multi-negative distribution is substituted into the reward-learning objective to formulate the loss.Source: §3.1 · p. 5
  • With N = 2 candidates—one dispreferred item—S-DPO reduces to DPO.Source: §3.1 · p. 5
  • S-DPO raises the preferred item's likelihood and lowers each dispreferred item's likelihood, weighting updates by reward deviation and the relative rewards of multiple negatives.Hard negatives receive greater extra weight and therefore a larger likelihood decrease than easy negatives.Source: §3.1 · pp. 5–6

3.2 Properties of S-DPO

  • DPO and S-DPO correspond to BPR and softmax-loss patterns, respectively; because softmax works better than BPR in multi-negative scenarios, S-DPO is argued to be more tailored to multi-negative alignment.When the reference model is approximately uniform, β log πθ(e|xu) serves as an implicit reward reflecting item preference.Source: §3.2 · p. 6

4 Experiments

The experiments compare S-DPO with recommendation baselines and study its ranking objective, negative samples, and β. S-DPO leads the reported benchmarks, while analyses examine optimization, reward behavior, and parameter effects.

4.1 Overall Performance Comparison (RQ1)

  • S-DPO exceeds all traditional and state-of-the-art LM-based recommenders on all three datasets, improving Hit Ratio@1 by 11.10%–47.03% over the second-best baseline.Table 1 covers three real-world datasets; the reported improvement range is for Hit Ratio@1.Source: Table 1 · p. 7, §4.1 · p. 8

4 Experiments

  • The experiments compare traditional, LM-enhanced, and LM-based recommenders using Hit Ratio@1 and Valid Ratio across three real-world datasets.The datasets are Movielens, Goodreads, and LastFM.Source: §4 · pp. 6–7

4.1 Overall Performance Comparison (RQ1)

  • For LM-based recommenders, fine-tuned models outperform untuned ones, and S-DPO further improves ranking by aligning the objective with multi-negative preference data.The passage reports that LLaRA outperforms TALLRec across datasets, while S-DPO extends DPO with a softmax ranking loss.Source: §4.1 · pp. 7–8

4.2 Study on S-DPO

  • DPO outperforms SFT, and S-DPO performs best among the compared variants, supporting the value of ranking optimization and multiple negatives.The ablation compares SFT, single-negative DPO, and S-DPO with multiple negatives.Source: §4.2 · p. 8, Figure 2 · p. 8
  • S-DPO’s validation loss decreases faster and more substantially than DPO’s, while its preferred-item rewards increase more continuously, significantly, and stably.The reported comparisons are on LastFM; the authors connect the loss trend to more meaningful optimization gradients and the reward trend to better distinction of preferred items.Source: §4.2 · p. 8, Figure 2 · p. 8
  • Performance improves as the number of negative samples rises from one to 15, which the authors attribute to more effective ranking gradients.The tested sample counts are 1, 3, 5, 8, 10, and 15.Source: §4.2 · pp. 8–9, Figure 3 · p. 9
  • With K negatives, S-DPO has lower stated computational complexity than multi-negative DPO and achieves comparable performance with fewer training steps.The reported complexities are Θ((K + 1)(CM + 1)St) for S-DPO and Θ(2KCMSt) for DPO.Source: §4.2 · p. 9, Table 2 · p. 9
  • β controls deviation from the reference policy: too little preference influence can lower validity and Hit Ratio, while excessive β impedes learning ranking relationships.The experiments use β values 0.1, 0.5, 1, 3, and 5; the authors set β to 1 for their main experiments and studies as a balance.Source: §4.2 · p. 9, Figure 3 · p. 9

5 Related Work

Prior LM recommenders either use LMs as representations or employ them directly, with fine-tuning and item representations as adaptation strategies. Preference-alignment work has developed DPO and its variants, while multi-negative DPO for recommendation remains relatively unexplored.

5.1 LM for Recommendation

  • LM-enhanced recommenders use LM embeddings or features but retain traditional recommenders for final recommendations, whereas LM-based systems directly make recommendations with LMs.LM-based systems have used in-context learning, supervised fine-tuning, and varied item representations.Source: §5.1 · pp. 9–10
  • Existing fine-tuned LM recommenders optimize language generation, while S-DPO explicitly optimizes item ranking using preference data.Source: §5.1 · p. 10

5.2 Preference Alignment of Language Models

  • DPO avoids RL optimization through reward-model parameterization, and later variants modify preference optimization in different ways.The cited variants include IPO, ODPO, KTO, GPO, f-DPO, and RSO.Source: §5.2 · p. 10
  • Despite DPO’s use across NLP and multimodal LMs, adapting it to recommendation and handling multiple negatives remain largely unexplored.Source: §5.2 · p. 10

6 Limitation

The experiments cap negative samples at 15 and do not fully explore their potential; adding negatives also increases training costs, especially at larger counts.

6 Limitation

  • Experiments cap negative samples at 15, leaving their potential incompletely explored; more negatives also increase training costs, increasingly so at larger counts.Source: §6 · p. 10

7 Conclusion

S-DPO uses multiple negatives to encode ranking information in LM-based recommenders. It outperforms traditional and LM-based baselines on three sequential recommendation datasets and is theoretically linked to softmax loss and hard-negative mining.

7 Conclusion

  • S-DPO surpasses traditional and LM-based baselines on three datasets in sequential recommendation tasks.The conclusion reports that it also provides better rewards for preferred items than DPO.Source: §7 · p. 10
  • S-DPO provides better rewards for preferred items than DPO.Source: §7 · p. 10
  • The theoretical analysis connects S-DPO to softmax loss and identifies an inherent benefit for mining hard negatives with larger, more effective optimization gradients.Source: §7 · p. 10

A Mathematical Derivations

The appendix derives a probability for multi-negative preference rankings, relates S-DPO to DPO, and develops its gradient. It also describes the baseline methods used for comparison.

A.1 Deriving Preference Distribution

  • The multi-negative preference event ranks the positive item above every negative, with its probability derived by decomposing PL permutations after the positive.Source: §A.1 · p. 17

A.2 Connection Between DPO and S-DPO

A.3 Deriving the Gradient of S-DPO Loss

  • The gradient derivation expresses the gradient of V and uses sigmoid and log-sigmoid identities.Source: §A.3 · p. 18

B.1 Baselines

  • The comparison includes traditional sequential methods GRU4Rec, Caser, and SASRec; MoRec; and LM-based LLaMA2, Chat-REC, TallRec, and LLaRA.Source: §B.1 · pp. 18–19

B.2 Datasets

The evaluation uses MovieLens100K, Goodreads, and LastFM, retaining item titles as textual descriptions and chronologically splitting interactions into training, validation, and test sets.

B.2 Datasets

  • The three datasets are MovieLens100K for movies, Goodreads for books, and LastFM for listening histories and artists.Source: §B.2 · p. 19
  • Item titles are retained as textual descriptions, and Goodreads users and books with fewer than 20 interactions are removed.Source: §B.2 · p. 19
  • Interactions are ordered chronologically and split into training, validation, and test sets in an 8:1:1 ratio to prevent potential information leakage.Source: §B.2 · p. 19

B.3 Implementation Details

Experiments use Llama2-7B as S-DPO’s language-model backbone, with a separate preference-training stage and selected hyperparameters for the main results.

B.3 Implementation Details

  • S-DPO uses Llama2-7B, randomly varies prompt formats, and is implemented with Python 3.9.7, PyTorch 2.2.2, and Transformers 4.38.2 on four NVIDIA A100s.Source: §B.3 · p. 19
  • LM methods train for at most five epochs, with batch size 128 and the checkpoint with lowest validation loss selected; learning rates use warm-up and cosine scheduling.Source: §B.3 · p. 19
  • S-DPO preference training runs for three additional epochs at batch size 128 and learning rate 1e-5; main results use β = 1 and search 3 or 5 negatives.Source: §B.3 · p. 19

B.4 Evaluation Metrics

S-DPO outperformed language-modeling loss across the tested backbones while maintaining or improving output validity; broader validation remains future work.

B.4 Evaluation Metrics

  • Evaluation uses HR@1 to measure whether the correct item ranks first among 21 candidates and Valid Ratio to assess instruction adherence and response appropriateness.The 21 candidates comprise the correct item and 20 randomly selected non-interacted items; NDCG is considered unsuitable because LMs struggle to produce ranked results.Source: §B.4 · p. 19

C Study on Backbone Language Models

  • S-DPO consistently outperformed language-modeling loss across three backbones and two datasets, with performance improving from 3 to 8 negatives while validity was maintained or improved.Table 4 compares LLAMA1-7b, Pythia-2.8b, and Mistral-7b on LastFM and MovieLens.Source: Table 4 · p. 20, C Study on Backbone Language Models · p. 20

D Broader Impact

  • Further exploration with more negative samples and validation across varied settings is left for future work.Source: D Broader Impact · p. 20

Related papers

Loading 2406.09215v3…