PaperlayerView exact revision on arXiv

Source-linked AI summary

Understanding Embedding Scaling in Collaborative Filtering

Yicheng He, Zhou Kaiyu, Haoyue Bai, Fengbin Zhu, Yonghui Yang

arXiv:2509.15709v2cs.IR

TL;DR

Prior evidence suggested embedding-dimension scaling could hurt collaborative-filtering performance, while its causes and consistency across models and datasets remained unclear. The authors study four models across 10 datasets and report double-peak, logarithmic, and single-peak trends, alongside an analysis relating double peaks to noise.

  • Problem

    Prior observations suggested embedding expansion could fail to improve collaborative-filtering performance, but its underlying causes and prevalence across models and datasets remained unclear.

    Source: §1 · p. 1

  • Method

    The authors test embedding-dimension scaling across 10 datasets and four representative models, then analyze interaction noise and model noise robustness.

    Source: §1 · p. 2

  • Results

    Across models and datasets, embedding scaling showed double-peak and logarithmic patterns in addition to single-peak behavior, with BPR double-peaked and SGL tending logarithmic.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors prefer logarithmic scaling for its expected stability and identify interaction noise as a key factor in double-peak behavior.

    Source: §1 · p. 2

    Computational resources limited experiments to a narrow model range, and observations relied primarily on Top@K with K=20.

    Source: §6 · p. 10

Abstract

from arXiv · show

Scaling recommendation models into large recommendation models has become one of the most widely discussed topics. Recent efforts focus on components beyond the scaling embedding dimension, as it is believed that scaling embedding may lead to performance degradation. Although there have been some initial observations on embedding, the root cause of their non-scalability remains unclear. Moreover, whether performance degradation occurs across different types of models and datasets is still an unexplored area. Regarding the effect of embedding dimensions on performance, we conduct large-scale experiments across 10 datasets with varying sparsity levels and scales, using 4 representative classical architectures. We surprisingly observe two novel phenomena: double-peak and logarithmic. For the former, as the embedding dimension increases, performance first improves, then declines, rises again, and eventually drops. For the latter, it exhibits a perfect logarithmic curve. Our contributions are threefold. First, we discover two novel phenomena when scaling collaborative filtering models. Second, we gain an understanding of the underlying causes of the double-peak phenomenon. Lastly, we theoretically analyze the noise robustness of collaborative filtering models, with results matching empirical observations.

1 Introduction

The introduction asks why embedding-dimension scaling can degrade collaborative-filtering performance and tests whether the previously observed single-peak pattern holds more broadly. Across 10 datasets and four models, it reports double-peak and logarithmic trends alongside single-peak behavior, then investigates interaction noise and model noise robustness.

1 Introduction

  • The study addresses why embedding expansion may fail, given prior reports of embedding collapse and an assumed single-peak relationship between dimension and performance.Earlier work also treated embedding dimensionality as a learnable parameter based on the idea of an optimal dimension.Source: §1 · pp. 1–2
  • Across 10 datasets and four collaborative-filtering models, scaling produced double-peak and logarithmic trends alongside single-peak behavior, with patterns varying across models and datasets.BPR exhibited a double-peak pattern, while SGL followed a logarithmic trend.Source: §1 · p. 2, Figure 1 · p. 2
  • The authors regard logarithmic scaling as the more stable pattern and double-peaked scaling as a limitation in scalability.They suggest future work focus on filtering clean interactions and further scaling SGL.Source: §1 · p. 2
  • Interaction noise, in addition to overfitting, is identified as a key factor causing the double-peak phenomenon.The authors propose sample dropping to help models scale, reporting expansion up to 32,768 dimensions without significant risk of collapse.Source: §1 · p. 2
  • The paper analyzes noise robustness in BPR, NeuMF, LightGCN, and SGL, finding that BPR is less robust and more often double-peaked while SGL is more robust and tends logarithmic.The theoretical analysis is reported to align largely with empirical observations.Source: §1 · p. 2

2 Background: Collaborative Filtering

Collaborative filtering represents users and items with embeddings and uses model-specific methods to estimate preference scores. The background introduces four representative models, including graph-based and self-supervised graph-based approaches.

2 Background: Collaborative Filtering

  • Collaborative filtering represents user and item preferences with embedding vectors and uses them to compute a user's preference score for an item.The user and item embeddings are vectors in R^k, where k is the embedding dimensionality.Source: §2 · p. 3
  • BPR computes user–item preference through embedding interactions, while NeuMF combines generalized matrix factorization and a multilayer perceptron in a two-tower architecture.NeuMF combines the two outputs to produce its prediction.Source: §2 · p. 3
  • LightGCN removes feature transformations and nonlinear activations, then aggregates neighboring-node embeddings with degree-aware symmetric normalization.The stated design is intended to capture high-order connectivity in user–item graphs.Source: §2 · p. 3
  • SGL augments graph structure and node features and adds contrastive objectives to improve robustness and alleviate sparsity.Its contrastive loss encourages consistency between augmented views, with γ controlling that task's contribution.Source: §2 · p. 3

3 Analysis of Scaling: Scale it up! Scale it up again!

Across collaborative-filtering models and datasets, increasing embedding dimensions can produce double-peak or logarithmic performance patterns. The paper investigates their scope and attributes double peaks to interaction noise and overfitting, then examines noise mitigation.

3.1 Observation Settings

  • The study evaluates BPR, NeuMF, LightGCN, and SGL across ten datasets; Figure 2 doubles embedding dimension at each step for its displayed models and datasets.The experiments use RecBole and a setup with eight 80G GPUs.Source: §3.1 · p. 4, Figure 2 · p. 5

3.2 Double-peak and Logarithmic Phenomenon

  • As dimensions grow, BPR double-peaks on ML-100K but rises continuously on Modcloth, while LightGCN and SGL show logarithmic growth on ML-100K.The authors describe the double-peak sequence as performance rising, falling, rising again, then falling; logarithmic growth continues with diminishing gains.Source: §3.2 · p. 4
  • Experiments across seven additional datasets confirm that the observed phenomena are widespread.The study uses ten datasets overall, with varying scale and sparsity, and four representative collaborative-filtering models.Source: §3.2 · p. 4, §3.1 · p. 4

3.3 Why Two Phenomena Occur

  • The authors propose that negative dataset factors affect BPR and NeuMF more than LightGCN and SGL, and identify interaction noise alongside overfitting as a cause of double peaks.They connect dimension growth to reduced network sparsity and describe clean-interaction learning, noise memorization, a robustness-improving stage, and eventual overfitting.Source: §3.3 · pp. 4, 6
  • SGL tends toward logarithmic growth, which the authors attribute to its contrastive-learning framework’s regularization and robustness enhancement.They suggest other methods may deviate from this pattern because they lack such a mechanism.Source: §3.3 · p. 5
  • Figure 3 compares standard BPR with denoising strategy-based BPR_Drop on ML-100K and Douban, reporting significantly improved performance degradation in both datasets relative to standard BPR.Source: Figure 3 · p. 6
  • The analysis draws on sparse double descent, a pruning pattern in which test performance falls, improves, and falls again as sparsity increases.The authors say embedding-dimension scaling is related to neural-network sparsification in recommendation tasks.Source: §3.3 · p. 6

4 One of the Keys to Scaling: Improve Robustness to Noisy Interactions

This section analyzes why noisy interactions affect scaling across BPR, NeuMF, graph collaborative filtering, and self-supervised graph collaborative filtering. It connects model-specific noise mechanisms to their differing robustness.

4.1 Analysis of the Resistance to Noise of BPR

  • BPR’s growing embedding gradients make noise shifts large; dynamic loss-threshold dropping improves performance in high-dimensional spaces.Theorem 1 relates representation-quality degradation quadratically to noise ratio δ and linearly to ∥∆noise∥; the double peak is attributed to overfitting noise.Source: §4.1 · p. 7

4.2 Analysis of the Resistance to Noise of NeuMF

  • NeuMF’s Jacobian can grow multiplicatively with depth, yielding exponentially amplified gradient sensitivity and potentially greater noise susceptibility than BPR.The comparison contrasts NeuMF’s depth-dependent Jacobian with BPR’s Jacobian, which grows linearly with embedding size.Source: §4.2 · p. 8

4.3 Analysis of the Resistance Noise of Graph Collaborative Filtering

  • LightGCN’s neighbor aggregation is likened to noise-resistant Mixup; graph convolution low-pass filters embeddings, attenuating high-frequency and retaining low-frequency components.This filtering leads to low-rank embeddings.Source: §4.3 · pp. 8–9

4.4 Analysis of the Resistance Noise of Self-supervised Graph Collaborative Filtering

  • SGL adds a contrastive subspace constraint to graph convolution’s filtering, reducing noise variance and improving robustness.SGL creates augmented graph views and its contrastive loss encourages shared noise-invariant information.Source: §4.4 · p. 9

5 Discussion

The discussion links dataset quality, especially noisy interactions, to scaling behavior and contrasts SGL’s stability with other models. It reports that SGL maintains performance better as embedding dimensions increase.

5 Discussion

  • Datasets with less interaction noise allow smoother scaling, while high noise muddles patterns and is associated with suboptimal performance as scale increases.The discussion identifies dataset quality, particularly noisy interactions, as substantially influential on scaling performance.Source: §5 · p. 10

Collaborative filtering models with noise-resistant structures have more advantages when it

  • SGL maintains stable or near-peak performance at substantially increased embedding dimensions, unlike BPR, which experiences a sharp decline.The discussion attributes SGL’s stronger scaling capability to its noise-resistant structure.Source: Collaborative filtering models with… · p. 10

6 Conclusion and Limitations

The conclusion reports two newly identified embedding-scaling patterns and attributes the double peak to noisy interactions, supported by a denoising loss and theoretical analysis. It also bounds the evaluation by model availability and its primary use of Top@20.

6 Conclusion and Limitations

  • Across 10 datasets and 4 representative models, the study identifies double-peak and logarithmic scaling beyond the previously recognized single-peak pattern.The authors attribute the double peak to noise interactions, introduce a denoising loss as empirical support, and report theoretical noise-robustness results aligned with observations.Source: §6 · p. 10
  • Available computational resources limited experiments to a narrower set of collaborative filtering models, and evaluation relied primarily on Top@20.The authors suggest testing additional K values and finer-grained metrics to further validate generality.Source: §6 · p. 10

A Notions

  • The notation appendix defines user/item latent representations, clean and noisy interaction sets, noise proportion, loss components, and representation-quality symbols.The proof setup models the noisy objective as a mixture of clean-data and noise-distribution losses, under smoothness and local strong-convexity assumptions.Source: A Notions · p. 14, §C.1 · p. 15

B.1 Scaling Recommendation.

  • Prior scaling work reports limited performance impact from parameter scaling relative to data scaling and identifies embedding collapse as an impediment.The cited work describes collapsed embeddings as leaning toward a low-dimensional subspace.Source: §B.1 · p. 14

B.2 Efficient Recommendation.

  • Efficient-recommendation methods learn embedding dimensions through architecture search, pruning, or mixed dimensions, often assuming a single optimal dimension.The passage names neural architecture search, weight pruning, and mixed-dimension approaches.Source: §B.2 · pp. 14–15

B.3 Denoising Recommendation

  • For implicit feedback, drop-based denoising uses sample loss to identify likely noisy interactions and filters them before training.Higher-loss samples are treated as more likely noisy than lower-loss samples.Source: §B.3 · p. 15

C.2 Proof of Theorem 2

The proof links NeuMF’s gradient sensitivity to MLP depth and uses a contrastive-loss condition to constrain noise variance in embeddings.

C.2 Proof of Theorem 2

  • The MLP term dominates the Jacobian’s depth behavior, while the inner-product term contributes only a bounded term proportional to the embedding norms.Source: §C.2 · p. 16
  • NeuMF’s gradient norm can grow multiplicatively with MLP layer spectral norms, implying exponential sensitivity in depth.The activation-dependent constant is bounded for standard activations such as ReLU and sigmoid.Source: §C.2 · p. 16

C.3 Proof of Theorem 3

  • The contrastive-loss condition constrains noise variance to at most ϵ², keeping embeddings primarily in the clean-signal subspace.Source: §C.3 · p. 17

C.4 Derivation of Definition 4

LightGCN propagation acts as a low-pass filter: it retains low-frequency embedding components, attenuates high-frequency components, and yields a rank bound based on significant eigenvalues.

C.4 Derivation of Definition 4

  • The spectral filter preserves components with eigenvalues near 1, corresponding to low frequencies.Source: §C.4 · pp. 17–18
  • The filter attenuates components with eigenvalues near 0, suppressing high-frequency noise.Source: §C.4 · p. 18
  • The final LightGCN embedding matrix has rank bounded by the number of significant low-frequency eigenvalues.The proof establishes this low-rank property after L propagation layers.Source: §C.4 · p. 18

D Experimental Settings

The experiments vary embedding size and use a shared training and evaluation setup across models; the BPR_Drop appendix provides its loss implementation.

D.1 Hyperparameters

  • Embedding dimensions range from 2 to 2^16, with NDCG@20 used to measure recommendation performance.Source: §D.1 · p. 18
  • Models train for up to 300 epochs with Adam at learning rate 0.001, uniform static negative sampling, and early stopping after 10 validation epochs without improvement.The batch size is 2,048, with one negative per positive interaction and no weight decay or gradient clipping.Source: §D.1 · p. 18

D.2 Pytorch Code of BPR_Drop

  • BPR_Drop computes per-sample pairwise losses from positive and negative scores using a sigmoid score difference and a small gamma term.Its implementation exposes save_ratio and get_low as configuration parameters.Source: §D.2 · p. 19

D.3 Detailed description of the datasets

The evaluation covers varied dataset scales and sparsity levels, and the selected models represent several major collaborative-filtering architectures.

D.3 Detailed description of the datasets

  • The experiments use 10 datasets spanning small to extra-large interaction scales and a range of sparsity levels.Source: §D.3 · p. 19
  • Most datasets are ultra-sparse, including Amazon_Beauty, while ML-1M is cited as a relatively dense example.Source: §D.3 · p. 19

D.4 Reason for Choosing Models

E.1 All Observations

Across additional datasets, SGL degrades more slowly than LightGCN, which degrades more slowly than NeuMF, while BPR performs similarly to SGL. The results also cover recurring embedding-scaling patterns and BPR_Drop’s behavior at different save ratios.

E.1 All Observations

  • Double-peak and logarithmic trends recur across datasets alongside the widely discussed single-peak pattern, supporting the generality of these observations.Source: §E.1 · p. 20
  • SGL’s performance maintenance as dimensions grow is attributed largely to contrastive-learning regularization and robustness, whereas LightGCN is described as more prone to overfitting or degradation.Source: §E.1 · p. 20
  • Across Figures 4–6, SGL degrades more slowly than LightGCN, which degrades more slowly than NeuMF, while BPR performs similarly to SGL.Source: §E.1 · p. 19, Figure 4 · p. 21, Figure 5 · p. 22

E.2 Effect of Varies Dropping Ratio

  • Figure 7 reports significantly improved performance degradation for BPR_Drop versus standard BPR on ML-100K and Douban, with BPR_Drop’s curves varying by save ratio.Source: §E.2 · p. 20, Figure 7 · p. 23

Related papers

Loading 2509.15709v2…