PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Effectiveness of Sampled Softmax Loss for Item Recommendation

Jiancan Wu, Xiang Wang, Xingyu Gao, Jiawei Chen, Hongcheng Fu, Tianyu Qiu

arXiv:2201.02327v2cs.IR

TL;DR

Recommendation research has rarely used sampled softmax (SSM) as its main objective or examined its properties, despite the cost of full softmax. The paper analyzes SSM’s theoretical properties and representation-magnitude limitation, then evaluates it across recommender models. History- and graph-based models perform strongly with SSM, while matrix factorization degrades in most cases.

  • Problem

    Few recommendation studies use SSM as the main learning objective or thoroughly examine its properties and advantages over pointwise and pairwise losses.

    Source: §1 · p. 3

  • Method

    The paper theoretically analyzes SSM’s recommendation properties and representation-magnitude limitation, then evaluates it with ID-, history-, and graph-based collaborative-filtering models.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    History-based and graph-based models achieve leading performance with SSM, while matrix factorization shows large performance degradation in most cases.

    Source: §1 · p. 3

  • Takeaways & Limitations

    SSM’s magnitude limitation can be compensated for by message-passing models that adjust representation magnitude according to node degree.

    Source: §3.2 · p. 8

    The paper analyzes magnitude adjustment from the item side, noting that the user-side analysis can be derived in the same way.

    Source: §3.2 · p. 8

Abstract

from arXiv · show

The learning objective plays a fundamental role to build a recommender system. Most methods routinely adopt either pointwise or pairwise loss to train the model parameters, while rarely pay attention to softmax loss due to its computational complexity when scaling up to large datasets or intractability for streaming data. The sampled softmax (SSM) loss emerges as an efficient substitute for softmax loss. Its special case, InfoNCE loss, has been widely used in self-supervised learning and exhibited remarkable performance for contrastive learning. Nonetheless, limited recommendation work uses the SSM loss as the learning objective. Worse still, none of them explores its properties thoroughly and answers ``Does SSM loss suit for item recommendation?'' and ``What are the conceptual advantages of SSM loss, as compared with the prevalent losses?'', to the best of our knowledge. In this work, we aim to offer a better understanding of SSM for item recommendation. Specifically, we first theoretically reveal three model-agnostic advantages: (1) mitigating popularity bias; (2) mining hard negative samples; and (3) maximizing the ranking metric. However, based on our empirical studies, we recognize that the default choice of cosine similarity function in SSM limits its ability in learning the magnitudes of representation vectors. As such, the combinations of SSM with the models that also fall short in adjusting magnitudes may result in poor representations. One step further, we provide mathematical proof that message passing schemes in graph convolution networks can adjust representation magnitude according to node degree, which naturally compensates for the shortcoming of SSM. Extensive experiments on four benchmark datasets justify our analyses, demonstrating the superiority of SSM for item recommendation. Our implementations are available in both TensorFlow and PyTorch.

1 INTRODUCTION

The paper addresses the limited study of sampled softmax (SSM) as a primary recommendation objective, despite its computational advantages over full softmax. It analyzes SSM’s properties and model-specific limitations, then evaluates its performance across recommender models.

1 INTRODUCTION

  • Pointwise losses fit interaction labels, pairwise losses rank observed items above unobserved ones, and softmax normalizes predictions across items.SSM substitutes a sampled subset of negatives for all items, reducing training cost while retaining softmax’s ranking property.Source: §1 · p. 2
  • Prior recommendation work rarely studies learning objectives; SSM as a main objective has received limited use and its properties remain underexplored.Full softmax can be costly at item scales of millions and intractable when streaming data do not expose the complete item space.Source: §1 · pp. 2–3
  • The paper theoretically identifies three model-agnostic SSM advantages: reducing popularity bias, mining hard negatives, and maximizing DCG.The authors describe SSM optimization as consistent with maximizing DCG, a metric suited to top-K tasks.Source: §1 · p. 3
  • History-based and graph-based models achieve leading performance with SSM, while matrix factorization suffers large performance degradation in most cases.The authors link the contrast to whether models can adjust representation magnitudes.Source: §1 · p. 3
  • The authors argue that history- and graph-based recommenders can compensate for SSM’s weakness in adjusting representation magnitudes.Their empirical analysis contrasts these models with MF, which lacks a mechanism to adapt representation magnitude.Source: §1 · p. 3

2 SSM FOR RECOMMENDATION

The section formulates recommendation with sampled softmax by contrasting an observed item against sampled negatives and maximizing its normalized probability among the candidates. It specifies in-batch negative sampling and temperature-aware cosine similarity as the paper’s default setup.

2 SSM FOR RECOMMENDATION

  • For each observed user–item interaction, NCE samples N negative items from a noise distribution and labels the observed item positive and the negatives zero.The resulting candidate set contains the observed item and the sampled negatives.Source: §2 · p. 3
  • The posterior probability of each candidate’s label is expressed using its conditional data likelihood, the number of negatives, and the positive and negative class priors.The paper models the positive-label posterior as a sigmoid of the user–item affinity score.Source: §2 · p. 4
  • The sampled-softmax objective applies a negative logarithm to the normalized conditional probability that the observed item is the sole positive among the candidate items.The probability is compared against alternatives in which another candidate is labeled positive.Source: §2 · p. 4
  • The default setup uses in-batch negatives, equivalent in expectation to sampling items by empirical dataset frequency, and temperature-aware cosine similarity.The paper notes that cosine similarity with temperature can mine hard negatives in a bounded interval; parameters are typically optimized with mini-batch SGD such as Adam.Source: §2 · p. 4

3 THEORETICAL ANALYSES

The analyses identify three advantages of SSM for recommendation: mitigating popularity bias, mining hard negatives, and aligning optimization with DCG. They also show that cosine-based SSM does not learn representation magnitudes, while message passing can compensate through degree-dependent magnitude adjustment.

3.1 Properties of SSM

  • As negative-sampling probability p_n(i) rises, the nearly closed-form optimal score f*(u,i) falls, offering an explanation for SSM’s popularity-bias mitigation.The analysis assumes N→∞ and relates the result to inverse propensity weighting.Source: §3.1 · p. 5
  • Alternative negative samplers can provide flexibility, but sampling N negatives independently for each interaction comes at the expense of training speed.Source: §3.1 · pp. 5–6
  • With a suitable temperature, harder negatives yield larger gradients, while overly hard negatives near x=1 are downweighted because they may be potential positives.The paper reports an empirical temperature range of 0.1 to 0.2 and describes the in-batch approach as adding little training complexity.Source: §3.1 · p. 6
  • Because −log DCG is a lower bound of SSM, minimizing SSM is equivalent to maximizing DCG over the item set.Source: §3.1 · p. 7

3.2 Potential Limitations of SSM

  • Cosine similarity leaves representation magnitudes outside SSM’s optimization, and MF’s lack of a magnitude-adjustment mechanism is associated with poor representations.The analysis sets aside the regularization term.Source: §3.2 · p. 7
  • Message passing can adjust magnitudes according to node degree; the expected squared item-representation magnitude increases linearly with item degree, though the optimal solution remains open.Source: §3.2 · pp. 8–9

4 EXPERIMENTS

The experiments evaluate SSM across four benchmark datasets, examining recommendation performance, long-tail behavior, and the effects of sampling, similarity, and message passing. They also assess efficiency with in-batch negative sampling and temperature-aware cosine similarity, including how time and memory costs change as the candidate pool grows on Amazon-Book.

4.1 Experimental Setup

  • The evaluation compares BCE, BPR, softmax, CCL, and SSM across MF, SVD++, NGCF, and LightGCN on four benchmark datasets.It uses all-ranking evaluation with Recall@20 and NDCG@20, and reports results averaged over five runs.Source: §4.1 · pp. 9–10

4.2 Overall Performance Comparison

  • SSM achieves the best performance on each dataset, leads with history- and graph-based recommenders but performs poorly with MF, and helps NGCF match or surpass LightGCN in some cases.The authors attribute MF’s weakness to its lack of magnitude adjustment and popularity modeling; message-passing methods can adjust representation magnitudes using node degree.Source: §4.2 · pp. 11–12, Table 2 · p. 12

4.3 Long-tail Recommendation

  • With LightGCN, BCE, BPR, and softmax concentrate Recall@20 on popular items, whereas CCL and SSM perform well on long-tail groups.SSM is more competitive in waist and head groups, while CCL performs better in the smallest-ID groups on three of four datasets; SSM’s Amazon-Book group contributions are nearly uniform.Source: §4.3 · pp. 12–14, Figure 1 · p. 13

4.4 Component Analysis on SSM

  • SSM outperforms the compared in-batch variants across datasets, while in-batch sampling improves long-tail Recall@20 but worsens head-group performance.BCE-IB, BPR-IB, and CCL-IB generally lose performance, with BPR-IB on Amazon-Book the stated exception; unlike those variants, SSM balances long-tail and overall recommendation accuracy.Source: §4.4 · pp. 14–16, Table 3 · p. 15, Figure 2 · p. 15
  • Negative-sampling distribution affects Recall@20: larger β usually produces harder negatives, and a suitable temperature benefits hard-negative mining, while extreme temperatures degrade performance.β = 1.0 is usually best, although β = 0.9 improves Gowalla over the Table 2 result; on Amazon-Book, τ = 0.1 works better than τ = 0.15 or 0.05.Source: Figure 3 · p. 16, §4.4 · pp. 16–17
  • As the negative-sample count increases, SSM improves on all four datasets and surpasses BPR once N is sufficient, while BPR can deteriorate beyond a threshold.The text gives N > 8 on Amazon-Book as an example where SSM surpasses BPR, and notes that generalized BPR is more than 50x slower per epoch at N = 2048 on Amazon-Book.Source: §4.4 · p. 17, Figure 4 · p. 18
  • Cosine similarity is better for SSM training, but inner product is better at testing; MF-SSM learns flat item magnitudes, whereas graph models can adjust them.BPR’s inner-product magnitude learning can amplify popularity bias in graph models, where popular items obtain larger magnitudes and similarity scores.Source: §4.4 · pp. 18–19, Table 4 · p. 19, Figure 5 · p. 20
  • SVD++ performance varies considerably with message-passing normalization, and the best α settings differ between user-side and item-side propagation.The best user-side setting is α0 = 1.0 and α1 = 0, while the best item-side setting is α0 = α1 = 0.5 across all four datasets; item-side propagation performs better on three datasets, but not Alibaba-iFashion.Source: §4.4 · pp. 19–20
  • SSM with in-batch negative sampling and temperature-aware cosine similarity is reported to lead performance while outperforming other baselines in training time per epoch and epochs to best performance.The passage refers to Table 2 but provides no numerical values for these comparisons.Source: §4.4 · p. 21
  • On Amazon-Book with a three-layer LightGCN, SM’s time cost increases linearly as padding items are added and it runs out of memory at 500,000 padding items.Training and testing samples were held unchanged while the candidate item pool was enlarged.Source: §4.4 · p. 21
  • Across different padding counts, SSM’s memory cost is low and its running time remains almost unchanged, suggesting it may suit large-scale recommender systems better than SM.The passage reports that the comparison used Amazon-Book and a three-layer LightGCN.Source: §4.4 · p. 21

5 RELATED WORK

The section reviews pointwise and pairwise objectives alongside softmax-based losses for item recommendation. It describes SSM as a lower-cost alternative to full softmax and contrasts prior uses with this paper’s broader analysis.

5 RELATED WORK

  • Pointwise losses fit interaction labels, whereas pairwise losses optimize the relative ordering of positive and unobserved items.BPR is described as optimizing AUC; WARP and other pairwise variants also target ranking.Source: §5 · pp. 21–22
  • Softmax models recommendation scores as a probability distribution but is costly to compute; SSM reduces that cost by using only a subset of negatives.Prior work also treated SSM as biased relative to full softmax and proposed methods to estimate or correct the sampling distribution.Source: §5 · p. 22
  • Using observed interactions to form positive pairs makes InfoNCE equivalent to SSM, but few studies use SSM as the main recommendation objective.Prior work explored contrastive objectives and debiasing false negatives; MSCL empirically studied SSM for graph-based recommenders.Source: §5 · pp. 22–23
  • Unlike MSCL, which empirically verified SSM for graph-based recommendation, this work theoretically analyzes popularity bias, hard negatives, ranking metrics, and representation magnitudes.The authors also distinguish their analysis of SSM’s magnitude-adjustment shortcoming from MSCL’s empirical focus.Source: §5 · p. 23

6 CONCLUSION AND FUTURE WORK

The paper concludes that SSM has model-agnostic advantages but may be limited in adjusting representation magnitudes, which message-passing methods can address. Experiments on four benchmarks show superior performance for history- and graph-based models on normal and long-tail recommendation; future work includes robustness and broader model settings.

6 CONCLUSION AND FUTURE WORK

  • Across four benchmark datasets, SSM-trained history- and graph-based models show superior performance on both normal and long-tail recommendation tasks.The paper also reports theoretical analyses of SSM’s popularity-bias, hard-negative, and ranking-metric advantages.Source: §6 · p. 23
  • SSM may be limited in adjusting representation magnitudes, while message-passing methods can adjust magnitudes.The paper frames the three SSM advantages as model-agnostic and its magnitude limitation as model-specific.Source: §6 · p. 23
  • Future work will explore distributionally robust optimization, additional architectures such as diffusion models, and SSM for feature-based CTR models.The authors identify robustness and applicability to richer model settings as directions for further study.Source: §6 · p. 23

Related papers

Loading 2201.02327v2…