Source-linked AI summary
The Unfairness of Popularity Bias in Recommendation
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher
TL;DR
Popularity bias causes recommenders to overrepresent popular items, potentially mismatching users’ expectations. This paper analyzes the mismatch across three user groups and several algorithms. The results show that users with less interest in popular items receive the most unfair recommendations.
Problem
The paper asks how popularity bias makes recommendations deviate from users’ expected proportion of popular and non-popular items.
Method
The authors analyze MovieLens 1M user profiles and compare several algorithms’ popularity-bias propagation across three user groups.
Results
All tested algorithms produced recommendations more popular than users’ rated profiles, with the greatest mismatch affecting niche users.
Takeaways & Limitations
Popularity bias affects users unequally, and recommendations can be extremely unfair to users with lower interest in popular items.
Abstract
from arXiv · showhide
Recommender systems are known to suffer from the popularity bias problem: popular (i.e. frequently rated) items get a lot of exposure while less popular ones are under-represented in the recommendations. Research in this area has been mainly focusing on finding ways to tackle this issue by increasing the number of recommended long-tail items or otherwise the overall catalog coverage. In this paper, however, we look at this problem from the users' perspective: we want to see how popularity bias causes the recommendations to deviate from what the user expects to get from the recommender system. We define three different groups of users according to their interest in popular items (Niche, Diverse and Blockbuster-focused) and show the impact of popularity bias on the users in each group. Our experimental results on a movie dataset show that in many recommendation algorithms the recommendations the users get are extremely concentrated on popular items even if a user is interested in long-tail and non-popular items showing an extreme bias disparity.
1 INTRODUCTION
Popularity bias causes recommenders to overrepresent frequently rated items, potentially diverging from users’ expectations. This paper shifts attention to users’ differing interests in popular and long-tail items.
- Popularity bias makes popular items appear too frequently in recommendations while limiting exposure for most other items.
- Long-tail items help reveal users’ preferences and are especially relevant for systems exploring less-known parts of user profiles.
- The paper studies popularity bias from users’ perspective, comparing expected and recommended ratios of popular versus non-popular items.
- Three user groups are defined by interest in popular items, with niche users receiving the greatest bias disparity and generally rating more items.
- The analysis asks how users differ in interest in popular items and how algorithmic popularity bias affects those groups.
2 RELATED WORK
Prior work has studied popularity bias through accuracy, long-tail exposure, and broader recommendation fairness. This paper instead evaluates fairness by asking whether recommendations match users’ expected popularity mix.
- Earlier studies primarily examined recommendation accuracy under long-tail rating distributions or proposed methods to increase long-tail exposure.
- This paper defines recommendation fairness as matching the popularity composition users expect from their rated-item profiles.
- The approach is related to calibration, but measures popularity rather than content categories and does not use item content.
- Other fairness work addresses demographic discrimination or equitable exposure across item categories, whereas this study centers users’ expectations.
- Comparisons of algorithms have documented differing popularity bias, while this paper emphasizes how that bias affects users with different expectations.
3 POPULARITY BIAS IN DATA
The MovieLens 1M rating data has a long-tail popularity distribution that algorithms can amplify. Users differ in their popularity preferences, and larger profiles tend to contain proportionally fewer popular items.
- MovieLens 1M contains 1,000,209 ratings from 6,040 users on approximately 3,900 movies.
- Rating data is skewed toward a few highly rated items, leaving most items with substantially fewer ratings.
- Algorithms can amplify this imbalance by repeatedly recommending popular items, increasing their opportunities to receive further ratings.
- 3.1 User Propensity for Popular Items: More than 4,000 users have at least 20% non-popular items in their profiles, while nearly all remaining users also include some non-popular items.
- 3.2 Users Profile Size and Popularity Bias: Popular items are defined as those in the top 20% of item popularities; larger profiles have a lower ratio of popular items because users exhaust the small popular-item pool.
- 3.2 Users Profile Size and Popularity Bias: Users less inclined to rate popular items tend to provide more ratings, giving them a potentially greater effect on recommender-system performance.
4 DIFFERENT GROUPS OF USERS IN TERMS OF PROPENSITY FOR POPULARITY
Users are grouped by the proportion of popular items in their profiles into Niche, Diverse, and Blockbuster-focused categories. In MovieLens 1M, Diverse users are the largest group, while Niche users have the longest profiles and Blockbuster-focused users the shortest.
- Niche users are the bottom 20% by popular-item ratio, with more than half of their profiles consisting of long-tail items.
- Blockbuster-focused users are the top 20% by popular-item ratio and average more than 85% popular items in their profiles.
- Diverse users comprise the remaining users who are neither Niche nor Blockbuster-focused.
- More than 3,500 users belong to the Diverse group, making it the largest user category in MovieLens 1M.
- More than 4,500 users, around 75% of the population, belong jointly to the Niche and Diverse groups.
- Niche users have the largest average profiles, followed by Diverse users, while Blockbuster-focused users have the shortest average profiles.
5 ALGORITHMIC PROPAGATION OF POPULARITY BIAS
The paper evaluates how recommendation algorithms propagate popularity bias and whether recommendations preserve the popularity mix expected by different user groups. Across analyses, algorithms disproportionately favor popular items, with niche users experiencing the largest mismatch.
- General popularity propagation: Except for Random, all algorithms leave many items almost never recommended, while Most popular shows the strongest popularity-recommendation correlation.User KNN and Item KNN also show strong correlation; Biased MF has sparse recommendations despite weak apparent correlation, and SVD++ shows positive correlation.
- User-group disparities: For every user group except under Random, recommendation lists significantly increase the ratio of popular items, with Niche users experiencing the largest discrepancy.SVD++ recommendations appear more consistent with users’ expected ratio of popular and non-popular items.
- Group-level measurement: GAP measures the average popularity of items in a user group’s profiles or recommendation lists, while ∆GAP measures the change imposed by an algorithm.A ∆GAP of 0 represents fair preservation of users’ preferences toward item popularity in recommendations.
- User-group disparities: Most popular produces the highest positive ∆GAP, Item KNN follows, Random produces negative ∆GAP, and Niche users have the largest ∆GAP for every algorithm.Negative ∆GAP means recommendations are less popular on average than the users’ profiles.
- Overall implication: These findings demonstrate that popularity bias is unfair because its effect varies with users’ interest in popular items.The results frame popularity propagation as a user-specific disparity rather than only a catalog-coverage issue.
6 DISCUSSION
The analysis finds substantial variation in users’ interest in popular items, while recommendation algorithms disproportionately serve popular items to users less interested in them. Thus, popularity bias affects users unevenly, with the greatest unfairness among users with lesser interest in popular items.
- Users differ in their interest in popular items, and many also expect non-popular items in their recommendations.Users with larger profiles are typically less likely to have rated only popular items and show some interest in long-tail items.
- Recommendations from all evaluated algorithms were extremely unfair to users with lesser interest in popular items, because almost all received recommendations were popular.The results indicate that not every user is affected by popularity bias to the same extent.
7 CONCLUSION AND FUTURE WORK
The paper concludes that recommendation algorithms amplify popularity bias from the users’ perspective, especially for users interested in non-popular items. It defines three user groups and identifies additional datasets and satisfaction measures as future research directions.
- Many algorithms recommend popular items more frequently while providing insufficient exposure to less popular items.
- The study defines Niche, Diverse and Blockbuster-focused user groups according to their interest in popular items.
- All algorithms recommended items much more popular than those rated by users in the three groups, especially Niche users.Niche users received particularly inappropriate recommendations when recommendation popularity was compared with their interests.
- Future work will extend the analysis to more datasets and examine satisfaction measures including relevance and diversity.