Source-linked AI summary
Connecting User and Item Perspectives in Popularity Debiasing for Collaborative Recommendation
Ludovico Boratto, Gianni Fenu, Mirko Marras
TL;DR
Imbalanced feedback can reinforce popular-item recommendations and compromise beyond-accuracy qualities such as novelty and coverage. The paper defines two item-level bias metrics and proposes an in-processing mitigation approach. Experiments report lower popularity bias and competitive accuracy and beyond-accuracy estimates compared with state-of-the-art alternatives.
Problem
Imbalanced feedback can reinforce recommendations of popular items over niche items of interest, while bias mitigation needs to account for both user preferences and item visibility.
Source: §1 · p. 3, §6.5 · p. 25Method
The paper defines metrics for item recommendation probability and item true positive rate, then combines balanced training-example sampling with a regularized loss to mitigate popularity bias.
Source: §1 · p. 4, §7 · p. 28Results
Treated models nearly reduce popularity bias while achieving competing ranking accuracy and beyond-accuracy estimates, generalizing across user populations and domains.
Source: §7 · p. 28Takeaways & Limitations
The combined mitigation procedure can improve item-level parity, equal opportunity, and beyond-accuracy estimates while reducing the gap in pair-wise accuracy between observed head and mid items.
Source: §7 · p. 28The offline experimental protocol cannot establish real-world effects on user satisfaction, which the authors say requires online evaluation.
Source: §6.5 · p. 27
Abstract
from arXiv · showhide
Recommender systems learn from historical users' feedback that is often non-uniformly distributed across items. As a consequence, these systems may end up suggesting popular items more than niche items progressively, even when the latter would be of interest for users. This can hamper several core qualities of the recommended lists (e.g., novelty, coverage, diversity), impacting on the future success of the underlying platform itself. In this paper, we formalize two novel metrics that quantify how much a recommender system equally treats items along the popularity tail. The first one encourages equal probability of being recommended across items, while the second one encourages true positive rates for items to be equal. We characterize the recommendations of representative algorithms by means of the proposed metrics, and we show that the item probability of being recommended and the item true positive rate are biased against the item popularity. To promote a more equal treatment of items along the popularity tail, we propose an in-processing approach aimed at minimizing the biased correlation between user-item relevance and item popularity. Extensive experiments show that, with small losses in accuracy, our popularity-mitigation approach leads to important gains in beyond-accuracy recommendation quality.
1. Introduction
Popularity-skewed feedback can reinforce recommendations of already-popular items, while reducing novelty and catalog coverage. The paper introduces item-level metrics and a mitigation framework to measure and reduce this bias.
1. Introduction
- Historical feedback imbalances can create a loop in which popular items gain visibility and selection likelihood while niche items of interest are under-recommended.Repeated recommendations can further concentrate training data on a small set of items.Source: §1 · p. 3
- Popularity bias can hinder novelty and catalog coverage, even though popularity-focused recommenders can achieve competitive accuracy.The authors frame mitigation as a trade-off between accuracy and beyond-accuracy quality.Source: §1 · p. 3
- Existing mitigation metrics often omit users’ preferences, motivating measures that connect item popularity with user tastes and recommendation visibility.The paper introduces one metric for equality of recommendation probabilities and another for equality of item true positive rates.Source: §1 · pp. 3–4
- The proposed approach balances training examples by relative item popularity and regularizes the correlation between user-item relevance and item popularity.The authors report less biased treatment of items, small accuracy losses, and gains in novelty and catalog coverage.Source: §1 · p. 4
2. Related Work
Prior work addresses popularity bias through data preparation, in-processing changes, and reranking, while fairness research provides related metric and optimization ideas. The paper positions its contribution as item-level measurement and in-processing mitigation without requiring head/tail membership.
2. Related Work
- The paper states that its research relates to literature from two research communities: the recommender-system community and the machine-learning community, without identifying a narrower specific connection in the cited passage.Source: §2 · p. 4
2.1. Item Popularity in Recommendation
- Popularity-bias approaches include preprocessing, in-processing, and post-processing, with trade-offs including dependence on head/tail labels and possible efficiency or accuracy costs.The paper balances observed-versus-unobserved item popularity in training examples and proposes a regularizer intended to generalize across learning-to-rank methods.Source: §2.1 · pp. 4–5
2.2. Bias Mitigation in Machine Learning
- The proposed metrics monitor individual items’ top-k recommendation probabilities rather than group outcomes or predefined popularity groups.The approach reframes constraint-based fairness strategies around correlation between relevance and popularity, with sampling that does not require group membership.Source: §2.2 · p. 6
3. Preliminaries
The paper formalizes recommendation from observed user-item feedback and introduces two item-level metrics for popularity bias. Item Statistical Parity compares recommendation rates across items, while Item Equal Opportunity compares their true positive rates among relevant items.
3.1. Recommender System Formalization
- The recommender-system setup represents users, items, and observed feedback in a user-item matrix, then predicts relevance scores for unobserved entries to rank recommendations.Feedback may be implicit or explicit, and items are ranked by decreasing predicted relevance.Source: §3.1 · pp. 6–7
3.2. Popularity Bias Metric Formalization
- Item Statistical Parity measures equality in item recommendation probabilities, with a score of 1 for a perfectly equal distribution.The metric uses the inverse Gini index across items; low values correspond to more unequal recommendation distributions.Source: §3.2 · pp. 7–8
- Equal recommendation rates can favor items with little user interest, so equalizing true positive rates offers a distinct objective that retains ground-truth preferences.Metric selection and trade-offs depend on stakeholder objectives.Source: §3.2 · pp. 8–9
- Item Equal Opportunity measures equality of item true positive rates, defined as the probability of recommending an item in a user’s top-k when it is relevant in the test set.Its score is the inverse Gini index across these item probabilities and ranges from 0 to 1.Source: §3.2 · pp. 8–9
4. Exploratory Analysis on Popularity Bias
The exploratory analysis evaluates popularity bias in representative point-wise and pair-wise recommenders using two datasets and newly proposed item-level metrics. It finds popularity-linked disparities in ranking and relevance, alongside relationships between lower bias, novelty, and catalog coverage.
4.1. Datasets
- The study tests representative point-wise and pair-wise algorithms on MovieLens1M and COCO600k, treating ratings as positive feedback.It notes that the analysis and mitigation approach can also apply to rating or frequency matrices.Source: §4.1 · p. 9, §4 · p. 9
4.2. Recommendation Algorithms and Protocols
- The four algorithms are Random, MostPop, NeuMF, and BPR, evaluated with a temporal split that holds out each user's most recent 20% of ratings.NeuMF and BPR use four unobserved-item examples or triplets per observed item, respectively.Source: §4.2 · pp. 9–10
4.3. Ranking Accuracy and Beyond-Accuracy Observations
- On the balanced test subset, BPR, NeuMF, and MostPop accuracy scores decreased, indicating that their full-test accuracy benefits from recommending popular items.MostPop's full-test ranking accuracy was competitive with BPR and NeuMF, especially on ML1M.Source: §4.3 · pp. 10–11, Figure 2 · p. 11
- Point-wise and pair-wise methods show unequal Item Statistical Parity and Item Equal Opportunity, with stronger inequalities for pair-wise optimization, sparse data, and low cutoffs.The analysis interprets low equal-opportunity values as potentially reflecting higher true positive rates for popular items and low or zero rates for others.Source: §4.3 · p. 12, Figure 3 · p. 12
- Higher Item Statistical Parity and Item Equal Opportunity accompany newer, catalog-wider recommendations, without inversely affecting ranking accuracy on a balanced test set.BPR and NeuMF recommend more novel items and cover the catalog better than MostPop, though the differences are often small and scores remain far from optimal.Source: §4.3 · pp. 13–14, Figure 4 · p. 13
4.4. Internal Mechanics Analysis
- Observed mid-popularity items often receive lower relevance than observed head items, and BPR and NeuMF pair-wise accuracy depends on the popularity of both compared items.The models more frequently fail to rank observed mid items above unobserved head items; observed head items are more often assigned higher relevance and over-represented in top-k lists.Source: §4.4 · pp. 14–16, Table 1 · p. 15
5. The Proposed Mitigation Procedure
The procedure combines training-example mining and regularized optimization to reduce biased correlation between predicted relevance and item popularity while also minimizing the recommendation algorithm’s loss.
5. The Proposed Mitigation Procedure
- The procedure jointly minimizes the recommendation algorithm’s loss and the biased correlation between predicted relevance and observed-item popularity.The regularization weight controls the trade-off: λ = 0 uses only the accuracy loss, while λ = 1 minimizes only the regularization loss.Source: §5 · pp. 16–18
- The procedure is demonstrated with BPR and NeuMF but is described as applicable to other algorithms from the same point-wise and pair-wise families.Training-example mining runs after each epoch, while regularized optimization is computed per batch until convergence.Source: §5 · pp. 16, 18
- Unlike approaches that rely on known group memberships, the method examines relative relevance and popularity for individual items without an arbitrary head–mid split.Its design focuses on training processes and model facets that the authors describe as under-explored for popularity debiasing.Source: §5 · pp. 16–17
- Training-example mining balances popularity comparisons by sampling unobserved items less popular than the observed item for half the examples and more popular for the other half.This strategy is applied to point-wise pairs and pair-wise triplets; the observed interaction is replicated in the point-wise setting.Source: §5 · p. 17
- The regularizer compares predicted residuals with observed-item popularity and penalizes models when higher predicted relevance depends on item popularity.The paper describes the correlation regularization as applicable to a wide range of learning-to-rank procedures.Source: §5 · p. 18
6. Experimental Evaluation
The evaluation tests how sampling and regularization affect accuracy, popularity bias, internal mechanics, broader recommendation quality, and comparisons with other mitigation procedures. Across these experiments, sam+reg improves popularity-related and beyond-accuracy measures, with accuracy trade-offs that depend on the test setting and model.
6.1. Effects of Mitigation Elements (RQ1)
- Sampling, alone or with regularization, improved ISP and IEO; adding regularization to sampling raised IEO but did not significantly change ISP.On the balanced test set, accuracy differences across models were small; on ML1M with NeuMF, combining the treatments slightly improved accuracy.Source: Figure 8 · p. 19, Figure 9 · p. 20, §6.1 · p. 21
6.2. Impact on Internal Mechanics (RQ2)
- Mitigation reduced the head–mid relevance gap, especially for pair-wise approaches and sparse datasets; pair-wise accuracy improved for (mid, head) but slightly decreased on setups involving observed head items.When no head–mid bias was present for NeuMF on COCO, performance decreased across the observed/unobserved-item setups; the authors advise applying mitigation only when the gap is considerable.Source: §6.2 · pp. 21–23, Figure 10 · p. 22, Table 2 · p. 22
6.3. Linking Regularization Weight and Recommendation Quality (RQ3)
- Across tested regularization weights, mitigation traded some full-test NDCG for comparable or better balanced-test NDCG, large ISP and IEO gains, higher novelty, and wider coverage.The authors describe a reasonably stable performance at various λ values and identify α = 0.2 as a reasonable accuracy–quality balance.Source: §6.3 · p. 23, Figure 11 · p. 24
6.4. Comparison with Other Mitigation Procedures (RQ4)
- sam+reg largely improved ISP on both datasets and was competitive on IEO, novelty, and catalog coverage, with higher full-test NDCG loss but balanced-test NDCG comparable to Pop-Weighted.The comparison covers the mitigation procedures’ ranking, popularity-bias, and beyond-accuracy metrics.Source: §6.4 · p. 25, Table 3 · p. 26
6.5. Discussion
- The authors report that combining sampling with correlation-based regularization lowers popularity bias at an accuracy cost, while noting limits in metrics, algorithms, data, and real-world evidence.The study uses ratings, focuses on selected metrics and two optimization families, and does not establish the real-world impact of mitigated recommendations on user satisfaction.Source: §6.5 · pp. 27–28
7. Conclusions
The conclusions report popularity-related differences in relevance scores and pair-wise accuracy for head and mid items, alongside gains from the proposed mitigation. Treated models nearly reduce popularity bias while retaining competing ranking accuracy and beyond-accuracy estimates.
7. Conclusions
- The paper monitors popularity bias in ranking with two metrics and proposes mitigation using a new training-example sampling strategy and regularized loss.Source: §7 · p. 28
- Observed head and mid items have differently biased predicted relevance distributions, with higher average relevance scores for head items.Source: §7 · p. 28
- Pair-wise accuracy is lower for observed mid items than for observed head items, leaving mid items under-ranked regardless of users’ interests.Source: §7 · p. 28
- The combined sampling strategy and regularized loss lower the pair-wise accuracy gap between observed head and mid items and improve fairness and beyond-accuracy estimates.The reported gains include higher statistical parity, equal opportunity, and beyond-accuracy estimates.Source: §7 · p. 28
- Treated models have comparable accuracy to the original model when each item has the same number of test observations.The paper identifies this as a proper testing setup for considering popularity bias.Source: §7 · p. 28
- Treated models nearly reduce popularity bias while achieving competing ranking accuracy and beyond-accuracy estimates compared with state-of-the-art alternatives.The authors report that this generalizes across users’ populations and domains.Source: §7 · p. 28
- Future work will investigate time-sensitive popularity and relevance metrics, post-processing countermeasures, and stakeholder-specific beyond-accuracy objectives.Source: §7 · p. 28