PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Representation Alignment and Uniformity in Collaborative Filtering

Chenyang Wang, Yuanqing Yu, Weizhi Ma, Min Zhang, Chong Chen, Yiqun Liu, Shaoping Ma

arXiv:2206.12811v1cs.IR

TL;DR

CF research has largely emphasized stronger encoders, leaving the desired properties of user and item representations less studied. The paper analyzes alignment and uniformity, connects them theoretically and empirically to BPR, and proposes DirectAU; experiments report significant improvements even with simple matrix factorization. It concludes that these properties are useful targets for CF representation learning, while leaving other objectives for future study.

  • Problem

    Few CF studies investigate desired representation properties, although doing so is important for understanding existing methods and designing learning objectives.

  • Method

    The paper measures alignment and uniformity, theoretically connects them to BPR, empirically analyzes CF learning dynamics, and proposes DirectAU to optimize both directly.

  • Results

    Experiments on three public datasets show significant performance improvements over state-of-the-art CF methods when DirectAU uses a simple matrix factorization encoder.

  • Takeaways & Limitations

    Better alignment or better uniformity contributes to higher recommendation performance, motivating direct optimization of both properties in CF.

  • Takeaways & Limitations

    The paper leaves investigation of other learning objectives that favor alignment and uniformity to future work.

Abstract

from arXiv · show

Collaborative filtering (CF) plays a critical role in the development of recommender systems. Most CF methods utilize an encoder to embed users and items into the same representation space, and the Bayesian personalized ranking (BPR) loss is usually adopted as the objective function to learn informative encoders. Existing studies mainly focus on designing more powerful encoders (e.g., graph neural network) to learn better representations. However, few efforts have been devoted to investigating the desired properties of representations in CF, which is important to understand the rationale of existing CF methods and design new learning objectives. In this paper, we measure the representation quality in CF from the perspective of alignment and uniformity on the hypersphere. We first theoretically reveal the connection between the BPR loss and these two properties. Then, we empirically analyze the learning dynamics of typical CF methods in terms of quantified alignment and uniformity, which shows that better alignment or uniformity both contribute to higher recommendation performance. Based on the analyses results, a learning objective that directly optimizes these two properties is proposed, named DirectAU. We conduct extensive experiments on three public datasets, and the proposed learning framework with a simple matrix factorization model leads to significant performance improvements compared to state-of-the-art CF methods. Our implementations are publicly available at https://github.com/THUwangcy/DirectAU.

1 INTRODUCTION

Collaborative filtering commonly learns shared user and item representations with an encoder and BPR, while research has focused more on encoder power than representation properties. This paper studies alignment and uniformity, connects them to BPR, and proposes DirectAU to optimize them directly.

  • Motivation: CF embeds users and items into a shared space and commonly uses BPR to distinguish positive interactions from unobserved ones.Existing work has mainly emphasized more powerful encoders, while newer objectives beyond BPR have also shown robust improvements.
  • Representation properties: Representation quality can be characterized by alignment of positive user-item pairs and uniformity on the unit hypersphere.The paper argues that both properties are important for informative representations.
  • Approach: The paper theoretically shows that BPR favors alignment and uniformity, then empirically analyzes these properties across typical CF learning dynamics.These analyses motivate a learning objective that directly optimizes both properties, named DirectAU.
  • Results: DirectAU with a simple matrix factorization encoder yields significant performance improvements over state-of-the-art CF methods in experiments on three public datasets.The reported experiments evaluate the proposed learning framework rather than requiring a more complex encoder.

2 PRELIMINARIES

The paper formulates CF as predicting scores for unobserved user-item pairs from learned representations and commonly trains these representations with pairwise BPR. It measures alignment and uniformity to connect representation quality with CF objectives.

  • Collaborative filtering: CF infers scores for unobserved user-item pairs from observed interactions and recommends each user the highest-scoring items.The observed interaction set is denoted R, and s(u,i) indicates the predicted likelihood of interaction.
  • Collaborative filtering: Most CF encoders map users and items into low-dimensional representations, with predicted scores defined by representation similarity such as a dot product.Matrix factorization uses an embedding table, whereas graph-based models additionally use neighborhood information.
  • Learning objective: Pairwise BPR trains CF models to make each observed target item score higher than a randomly sampled unobserved negative item.The negative item i− is sampled from items the user has not interacted with.
  • Alignment and uniformity: Alignment measures the expected distance between normalized embeddings of positive pairs, while uniformity measures the logarithm of their average pairwise Gaussian potential.Together, the metrics capture closeness of positive instances and scattering of random instances on the hypersphere.

3 ALIGNMENT AND UNIFORMITY IN COLLABORATIVE FILTERING

This section defines alignment and uniformity for normalized collaborative-filtering representations, proves their connection to BPR, and examines how they evolve across CF methods. The analyses show that better alignment or uniformity is associated with higher recommendation performance.

  • Theoretical Analyses: Perfect alignment makes positive user-item representations identical, whereas perfect uniformity makes user and item representations uniformly distributed on S_d−1.Perfect alignment can sacrifice uniformity when all inputs map to one representation.
  • Theoretical Analyses: The BPR loss favors alignment and uniformity on the hypersphere when perfectly aligned and uniform encoders are realizable.Under this condition, such encoders are exact minimizers of the BPR loss.
  • Empirical Observations: During BPR training on Beauty, alignment improves quickly, uniformity initially worsens and later improves, while NDCG@20 rises with better alignment and uniformity.The experiment trains an MF model with an embedding-table encoder.
  • Empirical Observations: BPR-DS achieves better uniformity but worse alignment than BPRMF, while LGCN achieves superior alignment but worse uniformity; both outperform BPRMF.BPR-DS uses harder negatives, whereas LGCN aggregates neighborhood information.
  • Empirical Observations: ENMF attains the best performance with promising alignment and uniformity, supporting simultaneous optimization of both properties.Its pointwise MSE training uses all negative interactions but hurts uniformity at the later training stage.
  • Empirical Observations: Different CF methods follow distinct alignment-uniformity learning trajectories, and either property can contribute to higher recommendation performance.Figure 2 tracks both metrics each epoch and marks converged points, with lower metric values preferred.

4 DIRECTLY OPTIMIZING ALIGNMENT AND UNIFORMITY (DIRECTAU)

DirectAU is proposed to optimize representation alignment and uniformity directly for collaborative filtering. It uses normalized embeddings, separately measures user and item uniformity, and avoids additional negative samples by relying on in-batch positive pairs.

  • DirectAU: The framework encodes positive user-item pairs with an embedding table and l2-normalizes their embeddings onto the hypersphere.The default encoder maps user and item IDs to embeddings.
  • DirectAU: DirectAU directly optimizes alignment for positive user-item pairs and uniformity for users and items.The framework is designed around the two representation properties identified in the preceding analysis.
  • DirectAU: Alignment increases similarity between positive user-item representations, while uniformity measures how representations scatter on the hypersphere.User and item uniformity are calculated separately because their data distributions may differ.
  • DirectAU: DirectAU jointly optimizes alignment and uniformity with trade-off hyperparameter γ, whose desired value depends on dataset characteristics.The weight γ controls the desired degree of uniformity.
  • DirectAU: Unlike prior CF methods that use negative sampling, DirectAU uses only in-batch positive pairs and computes uniformity from in-batch pairwise distances.This is intended to better match the empirical user and item distributions.
  • DirectAU: The DirectAU framework can be combined with other encoders, while experiments report remarkable performance from its simple embedding-table encoder.Graph neural network combinations are tested separately.

5 EXPERIMENTS

Experiments on three public datasets evaluate DirectAU against state-of-the-art CF methods, examine alignment and uniformity, test encoder integration, and measure efficiency and parameter sensitivity. DirectAU consistently performs strongly with simple encoders, while balancing alignment and uniformity and maintaining competitive training time.

  • Experimental Settings: Experiments use Beauty, Gowalla, and Yelp2018 with standard preprocessing and 80%/10%/10% train-validation-test splits.Repeated interactions are removed, and every user and item is required to have at least five associated interactions.
  • Experimental Settings: DirectAU is compared with matrix-factorization, variational-autoencoder, graph-based, negative-sample-free, and contrastive-learning CF baselines.The baselines include BPRMF, BPR-DS, ENMF, RecVAE, LGCN, DGCF, BUIR, and CLRec.
  • Overall Performance: DirectAU yields impressive performance improvements and consistently balances alignment and uniformity across datasets.The best baseline varies by dataset, whereas DirectAU is reported to achieve the best performance overall and the lowest alignment and uniformity losses compared with state-of-the-art CF methods.
  • Representation Properties: Methods with better alignment and uniformity achieve better recommendation performance on Beauty, while optimizing only one property produces poor results on Yelp2018.Alignment-only training approaches perfect alignment but degenerates in uniformity; uniformity-only training maintains uniformity but performs poorly, whereas jointly optimizing both properties performs better.
  • Encoder Integration: DirectAU consistently improves different encoders, including MF and LGCN variants with different numbers of graph layers.The comparison evaluates original losses against corresponding DirectAU-loss variants.
  • Efficiency Analyses: DirectAU has slightly higher per-epoch training time than BPRMF and ENMF but converges quickly, giving total training time similar to them and much faster than LGCN.The efficiency comparison is conducted on Yelp2018 under the same implementation framework and fixed batch size.
  • Parameter Sensitivity: DirectAU uses one tunable hyperparameter, γ, whose performance typically rises and then falls as the uniformity-loss weight increases.Higher uniformity weights may suit Gowalla and Yelp2018, while the useful range may require wider and finer-grained searches in practice.

6 RELATED WORK

Collaborative filtering learns user and item representations from interactions, while recent work explores stronger encoders and alternative learning objectives. Contrastive representation learning motivates examining alignment and uniformity as properties of representation quality.

  • Latent factor models learn user and item representations from observed interactions to predict scores for unobserved interactions.
  • Graph neural network CF models represent observed user-item interactions as a bipartite graph to capture high-order connection information.
  • CF research has increasingly examined learning objectives and negative sampling alongside encoder structure, including MSE optimization and positive-only learning.
  • Contrastive representation learning identifies alignment for similar instances and uniformity for spreading representations across the unit hypersphere.

7 CONCLUSION

The paper studies representation quality in collaborative filtering through alignment and uniformity, connecting these properties to BPR and analyzing their learning dynamics. It also identifies future work on additional objectives favoring alignment and uniformity.

  • The paper measures collaborative-filtering representation quality from the perspectives of alignment and uniformity.
  • It theoretically investigates the connection between the commonly adopted BPR loss and alignment and uniformity.
  • Future work will examine other learning objectives that favor alignment and uniformity to improve effectiveness and efficiency.

A APPENDIX

The appendix documents DirectAU’s learning algorithm and explains how alignment and uniformity losses are calculated for entire learned collaborative-filtering embeddings.

  • The appendix presents the learning algorithm used by DirectAU.
  • It details the calculation of alignment and uniformity losses when measuring learned embeddings in collaborative filtering.
  • The loss calculations cover the entire set of learned embeddings in collaborative filtering.

A.1 Learning Algorithm of DirectAU

DirectAU trains encoder parameters using alignment and uniformity losses, with interaction data, an encoder structure, a uniformity weight, and an embedding dimension as inputs. The appendix also provides PyTorch-style pseudocode for reproducibility.

  • PyTorch-style pseudocode is provided to facilitate reproducibility during training.
  • Algorithm 1 takes interaction data, an encoder structure, the uniformity-loss weight γ, and embedding dimension d as inputs.
  • The algorithm outputs encoder parameters θ after randomly initializing all parameters.
  • DirectAU combines alignment loss with γ times the average of user-side and item-side uniformity losses.The objective is LDirectAU = Align(x, y) + γ·(Uni(x) + Uni(y)) / 2.
  • Algorithm 1 defines separate alignment and uniformity loss functions for training.
  • The uniformity function computes the logarithm of the mean of exp(-2 times pairwise distance).

A.2 Alignment and Uniformity Calculation

The section explains how to calculate alignment and uniformity from positive-interaction distributions, then presents popularity-weighted sampling for efficient uniformity estimation.

  • Alignment uses observed user-item interactions, while uniformity should sample user-user and item-item pairs from their corresponding popularity distributions.The training batch naturally follows these distributions because it is constructed from positive interactions.
  • Naively estimating uniformity by sampling interaction pairs requires traversing the interaction set twice, yielding O(|R|2) complexity that is usually intractable.The method also incurs redundant computations.
  • The proposed alternative samples directly from user and item sets with popularity weighting to match the required distributions.The weighting uses the number of related interactions as popularity, with normalization factors over users or items.
  • The direct-set and interaction-based calculations are exactly equivalent, while the former is substantially cheaper because user and item sets are usually smaller than R.This enables efficient measurement of both alignment and uniformity.
  • Popularity weighting makes uniformity emphasize distances between popular users and items, encouraging frequently involved representations to scatter on the hypersphere.Popular entities are more likely to become closely aligned, so scattering them is treated as desirable.
Loading 2206.12811v1…