Source-linked AI summary
User-centered Evaluation of Popularity Bias in Recommender Systems
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, Edward Malthouse
TL;DR
Popularity bias mitigation is usually evaluated from the items’ perspective, overlooking how users with different tolerance for popular items are affected. This paper proposes user-centered evaluation and mitigation, showing that its method addresses different users while improving existing metrics.
Problem
Existing popularity-bias metrics focus on items and insufficiently capture how users with different interest in popular items are affected.
Method
The paper introduces User Popularity Deviation (UPD) and Calibrated Popularity (CP), a user-centered re-ranking method, and evaluates existing approaches from both perspectives.
Results
User-centered popularity-bias mitigation outperforms existing solutions from users’ perspectives while also improving existing item-centered metrics across experiments on two publicly available datasets.
Takeaways & Limitations
Popularity-bias evaluation should complement item-centered metrics with user-centered measures that account for users’ tolerance toward popular items.
Takeaways & Limitations
Existing metrics can report favorable results when an algorithm recommends a few extremely non-popular items to everyone, and item- and user-centered measures’ connection remains unresolved.
Abstract
from arXiv · showhide
Recommendation and ranking systems are known to suffer from popularity bias; the tendency of the algorithm to favor a few popular items while under-representing the majority of other items. Prior research has examined various approaches for mitigating popularity bias and enhancing the recommendation of long-tail, less popular, items. The effectiveness of these approaches is often assessed using different metrics to evaluate the extent to which over-concentration on popular items is reduced. However, not much attention has been given to the user-centered evaluation of this bias; how different users with different levels of interest towards popular items are affected by such algorithms. In this paper, we show the limitations of the existing metrics to evaluate popularity bias mitigation when we want to assess these algorithms from the users' perspective and we propose a new metric that can address these limitations. In addition, we present an effective approach that mitigates popularity bias from the user-centered point of view. Finally, we investigate several state-of-the-art approaches proposed in recent years to mitigate popularity bias and evaluate their performances using the existing metrics and also from the users' perspective. Our experimental results using two publicly-available datasets show that existing popularity bias mitigation techniques ignore the users' tolerance towards popular items. Our proposed user-centered method can tackle popularity bias effectively for different users while also improving the existing metrics.
1 INTRODUCTION
Popularity bias favors a small set of popular items, while item-centered evaluation overlooks how users with different interests are affected. The paper proposes user-centered measurement and mitigation to address this gap.
- Popularity bias favors a small set of popular items, even when less popular items are preferred by interested users.
- Long-tail recommendations can provide unfamiliar content, support fuller preference understanding, and create consumer surplus.
- Existing research mainly evaluates how algorithms amplify popular items, overlooking that users differ in their interest in popularity.
- The paper evaluates popularity bias from users’ perspectives by measuring whether recommended-item popularity matches each user’s historical interest.
- The paper proposes User Popularity Deviation, investigates recent mitigation methods, and presents a user-centered mitigation approach.
- Experiments on two publicly available datasets compare the proposed metric and approach with existing bias-mitigation methods.
2 POPULARITY BIAS IN RECOMMENDER SYSTEMS
Popularity bias concentrates recommendations on a small head of items and can distort recommendations for users with different preferences. The section illustrates this item- and user-level impact using item groups and three user groups.
- 2.1 Impact on Items: Popularity bias causes a few popular items to dominate recommendations while less popular items receive little or no exposure.
- 2.1 Impact on Items: Figure 1 partitions items into popular H, medium M, and less popular T groups, and users into G1, G2, and G3 by interest in popularity.
- 2.1 Impact on Items: In Most-Popular and Item-CF, items constituting less than 3% of the catalog account for 100% of recommendations.
- 2.2 Impact on Users: The user groups are Blockbuster-focused G1, Diverse Taste G2, and Niche-focused G3, defined by their interest in popular and less popular items.
- 2.2 Impact on Users: Most-Popular and Item-CF recommend only Head items, including to G3 users whose profiles contain diverse item groups.
- 2.2 Impact on Users: All user groups are affected, but G3 users are most poorly served because recommendations omit their long-tail interests.
3 POPULARITY BIAS MITIGATION
The section reviews item-centered metrics for popularity-bias mitigation, then motivates user-centered evaluation and introduces User Popularity Deviation and Calibrated Popularity. It also describes existing approaches whose evaluation can overlook whether recommended-item popularity matches users’ interests.
- 3.1.1 Existing metrics: item-centered evaluation: Existing metrics assess popularity bias through item exposure, using ARP, Aggregate Diversity, and Gini Index.These metrics respectively address recommended-item popularity, catalog coverage, and exposure inequality.
- 3.1.1 Existing metrics: item-centered evaluation: ARP can appear favorable when an algorithm recommends a few extremely unpopular items to everyone, while Aggregate Diversity ignores recommendation frequency.Gini Index is used to assess whether exposure is distributed evenly across items.
- 3.1.2 Proposed metric: user-centered evaluation: Two hypothetical algorithms receive identical item-centered scores despite producing different popularity matches for blockbuster-focused, diverse-taste, and niche-focused users.The example motivates evaluating how recommendations align with each user’s tolerance and tendency toward item popularity.
- 3.1.2 Proposed metric: user-centered evaluation: User Popularity Deviation measures how much each user group’s recommended-item popularity distribution deviates from historical interest, with lower values preferred.The metric uses Jensen–Shannon divergence and supports overall and group-level evaluation.
- 3.2 Algorithmic Solutions: The paper implements four prior bias-mitigation algorithms and compares them with a new user-centered approach using existing and user-centered evaluation.The reviewed approaches target long-tail representation, diversity, exposure distribution, or personalized balance between popular and less-popular items.
- 3.2 Algorithmic Solutions: Calibrated Popularity re-ranks recommendations to match each user’s historical distribution across popularity groups while balancing relevance and popularity calibration.Its scoring uses predicted relevance and the divergence between popularity distributions in the user profile and recommendation list.
4 EXPERIMENTAL METHODOLOGY
Experiments use two public datasets, with fixed training/test splits and a RankALS base recommender for post-processing evaluation.
- Datasets: The experiments use a Last.fm album dataset with 274,707 ratings from 2,697 users across 6,006 albums.Users with fewer than 20 ratings were removed, and interaction frequencies were converted into ratings.
- Datasets: MovieLens 1M contains 1,000,209 ratings from 6,040 users across 3,706 movies.
- Experimental setup: Each dataset uses 80% for training and 20% for testing, with RankALS generating the initial recommendation lists.The base list contains 100 items, and the final recommendation list contains 10 items.
- Experimental setup: MovieLens 20M produced similar results, so MovieLens 1M was retained for efficiency.
5 RESULTS
The results show that popularity-bias mitigation metrics can conceal user-group differences. CP generally provides the most consistent user-centered performance, while other methods trade off precision, item exposure, and outcomes across user groups.
- Overall metrics: Increasing λ reduces precision for all algorithms on MovieLens, reflecting a relevance–popularity-bias mitigation trade-off.XQ and FS lose less precision, whereas DM and RG lose more; CP eventually levels off.
- Overall metrics: At λ=0.5 on MovieLens, CP and DM have equal precision and ARP, but CP’s UPD is 0.15 versus DM’s 0.3.The result indicates that CP better matches recommendation popularity to users’ tolerance.
- Overall metrics: On Last.fm, CP outperforms the other approaches on almost every metric, including item-centered and user-centered measures.This shows that user-centered mitigation can also perform well on item-centered metrics.
- Overall metrics: RG performs poorly on both datasets, consistent with prior findings that model-based mitigation generally underperforms re-ranking approaches.
- Metric comparisons: DM performs well on MovieLens Agg-Div and Gini but poorly on UPD, illustrating disagreement between item-centered and user-centered metrics.The algorithms were compared at λ values chosen to yield similar precision.
- Metric comparisons: Figure 4 shows that DM’s strong Agg-Div and Gini results can coincide with near-zero exposure for many items.Exposure frequencies are calculated across users and sorted for comparison.
- User-group analysis: FS increases exposure of mid-popularity items but does little for tail items, while CP produces a smoother balance across item groups.DM and XQ include tail items mainly by reducing mid-popularity exposure rather than high-popularity concentration.
- User-group analysis: CP has the lowest UPD for every user group and maintains consistently low deviation, unlike methods whose performance varies across groups.For example, FS favors G1 over G3, while its UPD is lower than DM’s for G1 but higher for G3 on MovieLens.
6 DISCUSSION
The paper finds that popularity bias affects user groups differently, while existing item-centered metrics miss whether recommendations match users’ desired popularity ranges. It proposes user-centered evaluation with UPD, while treating it as complementary to item-centered metrics and noting that popularity calibration does not ensure content calibration.
- Niche-oriented users are more affected by popularity bias because recommendations deviate more from their expected popularity levels.
- Aggregate diversity, Gini index, and average recommendation popularity can measure overall mitigation but miss user-specific popularity matching.
- UPD measures whether each user’s recommendation-list popularity matches their desired range and complements existing item-centered metrics.
- Popularity calibration does not guarantee that recommended content matches users’ broader preferences, such as genre preferences.
7 CONCLUSION AND FUTURE WORK
The paper studies popularity bias from users’ perspectives, showing that user tolerance affects experienced impact and that commonly used metrics can overlook this variation. It proposes a user-centered metric and mitigation approach, while identifying method simplicity, dataset differences, and temporal effects as future-work concerns.
- Popularity bias affects users differently depending on their tolerance toward popular items, motivating user-centered evaluation and mitigation.
- The proposed user-centered approach improved existing item-centered metrics while addressing popularity bias from users’ perspectives.
- Existing approaches improved item-centered metrics but did not perform well when evaluated from users’ perspectives.
- The CP method was relatively simple, leaving substantial scope for future enhancement of the user-centered algorithm.
- Future work should examine how dataset characteristics and feedback loops shape popularity-bias mitigation across user groups over time.