Source-linked AI summary
MSL: Not All Tokens Are What You Need for Tuning LLM as a Recommender
Bohao Wang, Feng Liu, Jiawei Chen, Xingyu Lou, Changwang Zhang, Jun Wang, Yuegang Sun, Yan Feng, Chun Chen, Can Wang
TL;DR
LLM recommenders are often fine-tuned with Language Modeling Loss, which can diverge from recommendation ranking and treat fictitious item descriptions as negatives. The paper introduces Masked Softmax Loss with an adaptive temperature strategy to address these issues. Across four real-world datasets, it reports a 42.24% average NDCG@10 improvement over LML.
Problem
Method
MSL masks invalid tokens from the loss calculation, while ATS adaptively adjusts temperature to mitigate MSL’s gradient vanishing without extensive tuning.
Source: §1 · p. 2Results
42.24% average improvement in NDCG@10 over LML is reported across four real-world datasets.
Source: §1 · p. 3Takeaways & Limitations
The analysis presents MSL as closely connected to ranking objectives and a tight upper bound of NDCG metrics.
Source: §1 · p. 2The theoretical analysis relies on an assumption that a Gaussian distribution nearly holds.
Source: §4.2.3 · p. 6
Abstract
from arXiv · showhide
Large language models (LLMs), known for their comprehension capabilities and extensive knowledge, have been increasingly applied to recommendation systems (RS). Given the fundamental gap between the mechanism of LLMs and the requirement of RS, researchers have focused on fine-tuning LLMs with recommendation-specific data to enhance their performance. Language Modeling Loss (LML), originally designed for language generation tasks, is commonly adopted. However, we identify two critical limitations of LML: 1) it exhibits significant divergence from the recommendation objective; 2) it erroneously treats all fictitious item descriptions as negative samples, introducing misleading training signals. To address these limitations, we propose a novel Masked Softmax Loss (MSL) tailored for fine-tuning LLMs on recommendation. MSL improves LML by identifying and masking invalid tokens that could lead to fictitious item descriptions during loss computation. This strategy can effectively avoid the interference from erroneous negative signals and ensure well alignment with the recommendation objective supported by theoretical guarantees. During implementation, we identify a potential challenge related to gradient vanishing of MSL. To overcome this, we further introduce the temperature coefficient and propose an Adaptive Temperature Strategy (ATS) that adaptively adjusts the temperature without requiring extensive hyperparameter tuning. Extensive experiments conducted on four public datasets further validate the effectiveness of MSL, achieving an average improvement of 42.24% in NDCG@10. The code is available at https://github.com/WANGBohaO-jpg/MSL.
1 Introduction
LLM recommenders commonly fine-tune on positive-item descriptions with Language Modeling Loss (LML), but that objective can diverge from personalized ranking and penalize fictitious items as negatives. The paper proposes Masked Softmax Loss (MSL) and an Adaptive Temperature Strategy (ATS) to address these issues; experiments report a 42.24% average NDCG@10 improvement over LML.
1 Introduction
- LML focuses on generating valid item descriptions and offers limited guidance for distinguishing positive from negative items, diverging from recommendation’s ranking objective.It also treats other generated descriptions, including fictitious items, as negatives, although users may favor some semantically similar fictitious items.Source: §1 · p. 2
- MSL masks invalid tokens corresponding to fictitious item descriptions, avoiding their penalization and aligning the loss more closely with ranking objectives.The theoretical analysis describes MSL as a tight upper bound of NDCG metrics.Source: §1 · p. 2
- MSL can face gradient vanishing because fewer tokens in the softmax denominator can produce small gradients and loss values.ATS sets temperature adaptively based on the dataset’s average number of valid tokens, mitigating the issue without extensive hyperparameter tuning.Source: §1 · p. 2
- S-DPO integrates Direct Preference Optimization into LLM recommendation, but retains LML and requires negative-item sampling.The introduction also identifies training instability and higher computational cost as concerns; the cited comparison reports approximately four times the runtime versus MSL.Source: §1 · pp. 2–3
- 42.24% average improvement in NDCG@10 over LML is reported for MSL across four real-world datasets.The contribution summary reports the overall comparison.Source: §1 · p. 3
2 LLM-based Recommendation
This section frames sequential recommendation as ranking a user’s next positive item from historical interactions. It describes LLM recommendation prompts and supervised fine-tuning with positive-item descriptions and token-wise Language Modeling Loss.
2 LLM-based Recommendation
- Sequential recommendation infers a user’s next positive item from historical interactions and seeks to rank that item highly.The section notes that ranking metrics such as NDCG are commonly used to evaluate performance.Source: §2 · p. 3
- LLM recommendation prompts encode historical item descriptions and the task, then ask the model to predict a likely next-item description.Table 1 presents prompt templates using the Toy dataset as an example.Source: §2 · p. 3
- Supervised fine-tuning pairs a prompt with the description of the target positive item as a training instance.The section introduces Language Modeling Loss as the objective for this setup.Source: §2 · p. 3
- Token-wise LML increases the logits of target-description tokens while decreasing logits for other vocabulary tokens.The loss uses a softmax over the token vocabulary.Source: §2 · p. 3
3 Analyses on Language Modeling Loss
LML differs from recommendation ranking because it predicts across the full language space and penalizes fictitious descriptions as negatives. Its valid-item and positive-over-negative components behave unevenly during training, limiting its recommendation guidance.
3 Analyses on Language Modeling Loss
- LML retrieves positive-item descriptions from the full language space, whereas recommendation retrieves positive items from the system’s valid item set.The language space also includes fictitious items imagined by LLMs.Source: §3 · pp. 3–4
- LML combines a component that lifts valid items over invalid items with one that lifts positive items over negative items; only the latter aligns with recommendation ranking.The analysis connects the positive-over-negative component to NDCG metrics.Source: §3 · p. 4
- L1_LML dominates LML training gradients and hinders convergence of L2_LML, while directly optimizing L2_LML makes its training loss decrease quickly.The analysis uses gradient norms and L2_LML training-loss curves to contrast the two optimization settings.Source: §3 · p. 4, Figure 3 · p. 4, Figure 4 · p. 4
- LML penalizes invalid tokens as negative signals, although fictitious items may resemble positive items or simply be unreleased, potentially confusing preference learning.The passage gives “Iron Man: AI Rebellion” and “Iron Man 4” as examples.Source: §3 · p. 4
- S-DPO has higher NDCG@10 deviation than MSL across random seeds, with performance losses reaching 16.4% on Toy and 15.3% on Book.The passage also reports that S-DPO requires nearly four times the runtime of MSL and LML.Source: Table 3 · p. 4, §3 · p. 5
4 Methodology
The methodology introduces MSL to address LML’s limitations, then analyzes a potential gradient-vanishing issue and develops ATS to address it.
4.1 Masked Softmax Loss
- MSL masks invalid-token penalties, and under valid-token ranking its loss is a tighter upper bound of −log NDCG than LML.The stated ranking premise is naturally satisfied when generation masks invalid tokens using constrained beam search with a large beam size.Source: §4.1 · p. 5
4.2.1 Potential Gradient Vanishing Issue.
- 61% of Office samples with weights below 0.1 fall within the first three item tokens, where reduced MSL weights can approach zero.The authors attribute possible gradient vanishing to fewer terms in the valid-token denominator, especially when positive-token logits are larger.Source: §4.2.1 · p. 6, Figure 5 · p. 6
4.2.2 The Introduction of Temperature.
- Temperature modulates gradient magnitude and significantly improves performance; adjusting learning rate or adding a negative-token balancing coefficient does not yield satisfactory results.The performance comparison is shown in Figure 6.Source: §4.2.2 · p. 6, Figure 6 · p. 6
4.2.3 Adaptive Temperature Strategy.
- ATS adaptively adjusts temperature to keep valid-token probability near a target, using a global setting based on model state and average valid-token count.The strategy uses a Gaussian approximation for valid-token logits, which the authors report nearly holds.Source: §4.2.3 · pp. 6–7
5 Experiments
Experiments evaluate MSL against recommendation baselines on four real-world Amazon datasets, examining accuracy, loss components, and efficiency. MSL outperforms baselines across datasets, while temperature adjustment and ATS address its gradient challenge.
5 Experiments
- The experiments are organized around comparative performance, component ablations, and joint accuracy-efficiency evaluation.These correspond to RQ1, RQ2, and RQ3.Source: §5 · p. 7
5.1.1 Datasets.
- The evaluation uses four Amazon datasets, 5-core filtering, length-11 sequence windows, and chronological 8:1:1 train-validation-test splits.For Amazon Books, 100,000 items are randomly retained before 5-core processing; models use LLaMA3 8B, train for 10 epochs, and report the epoch with the highest NDCG@5.Source: §5.1.1 · p. 7, §5.1.3 · p. 7
5.1.2 Baselines.
5.2 Performance Comparison (RQ1)
- MSL consistently outperforms all baselines across the four datasets in the performance comparison.The comparison covers recommendation accuracy; the supplied passages do not give its table values.Source: §5.2 · p. 7
5.3 Ablation Study (RQ2)
- Temperature adjustment improves MSL, while ATS rivals or surpasses brute-force temperature search and removes exhaustive manual tuning.Without temperature, MSL underperforms LML on Toy and Office; alternative learning-rate and negative-token adjustments remain inferior to temperature adjustment.Source: §5.3 · p. 8
5.4 Efficiency Comparison (RQ3)
- 315% and 324% efficiency gains over S-DPO on Toy and Book, respectively; gains over LML are 4.4% and 6.7% on those datasets.Figure 7 compares recommendation accuracy and efficiency.Source: §5.4 · p. 8, Figure 7 · p. 9
- MSL reduces computation by restricting the softmax to valid tokens; S-DPO's negative-sample requirement produces roughly four times MSL's runtime.Trie construction is the only additional MSL overhead, and its reported time is within one second for all datasets.Source: §5.4 · pp. 8–9
6 Related Work
Related work covers sequential recommendation and two ways of applying LLMs to recommendation. It describes fine-tuning as a response to task-objective mismatch and identifies limitations in prior loss-based approaches.
6.1 Sequential Recommendation
- Sequential recommendation predicts a user's next item from historical interactions, with models including recurrent, convolutional, self-attention, and distributionally robust approaches.The cited examples include GRU4Rec [18], Caser [48], SASRec [22], BERT4Rec [46], and DROS.Source: §6.1 · p. 9
6.2 LLMs for Recommendation
- LLMs are used directly as recommenders or in auxiliary roles, while fine-tuned LLM recommenders commonly use LML and S-DPO adds sampled ranking pairs.The work characterizes S-DPO [9] as unstable, limited in effectiveness, and inefficient; it presents MSL as addressing these issues.Source: §6.2 · p. 9
7 Conclusion
The conclusion presents MSL as a recommendation-specific loss that masks invalid tokens, with ATS addressing gradient vanishing. The authors report theoretical and empirical support for improved LLM recommendation performance and suggest specialized architectures as future work.
7 Conclusion
- MSL masks invalid tokens during loss computation to align fine-tuning with recommendation objectives and avoid erroneous negative signals.The conclusion says MSL can encounter gradient vanishing; temperature and ATS adaptively adjust temperature without extensive hyperparameter tuning.Source: §7 · p. 9
- The authors report theoretical and empirical validation and improved performance for state-of-the-art LLM-based recommenders.Source: §7 · p. 9
- Future work could explore specialized LLM architectures to improve their suitability for recommendation systems.Source: §7 · p. 9
A The proof of lemma 1
The proof relates a rank-based expression for the positive item to NDCG, then uses the softmax denominators to compare MSL and LML.
A The proof of lemma 1
- Starting from item probabilities summing to 1, the proof uses positive-item rank π(p) and log(1 + π(p)) ≤ π(p) to relate a rank term to NDCG.Source: A The proof of lemma 1 · p. 10
- The proof states that MSL is no greater than LML because each MSL softmax denominator is smaller.Source: A The proof of lemma 1 · p. 10
B The proof of lemma 2
The proof expands an expectation involving exp(f/τ_t), uses a standard Gaussian integral, and proceeds through an approximation to a quadratic equation.
B The proof of lemma 2
- The proof expands E_f[exp(f/τ_t)] as an intermediate step.The supplied passage introduces the expansion but does not show its terms.Source: B The proof of lemma 2 · p. 10
- A standard Gaussian integral is set equal to 1 in the derivation.The passage uses this result before continuing the proof.Source: B The proof of lemma 2 · p. 10
- The derivation then approximates an equation and solves the resulting quadratic equation.The passages provide the sequence but not the equations or solution.Source: B The proof of lemma 2 · p. 10
C Validation of Gaussian distribution for logits
The section reports that the logits distribution closely follows a Gaussian distribution, with fitting results shown for the Toy and Book datasets.
C Validation of Gaussian distribution for logits
- The logits distribution closely follows a Gaussian distribution, with Gaussian fitting results shown for the Toy and Book datasets.Figure 8 presents the distribution and Gaussian fit.Source: C Validation of Gaussian distribution… · p. 10, Figure 8 · p. 10