Source-linked AI summary
The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation
Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher
TL;DR
The paper investigates whether popularity bias contributes to unequal miscalibration across user groups. Using miscalibration and popularity-lift measures across two real-world datasets, it finds that groups more affected by popularity bias receive less calibrated recommendations.
Problem
Unequal miscalibration across user groups may indicate unfair recommendation performance, motivating investigation of factors behind these differences.
Method
The paper measures users’ preference and recommendation-category distributions with miscalibration, and measures algorithmic popularity bias using popularity lift.
Results
Groups less interested in popular items experience greater popularity lift and higher miscalibration across recommendation algorithms.
Takeaways & Limitations
Popularity bias is connected to miscalibrated recommendations and affects user groups differently according to their interest in popular items.
Abstract
from arXiv · showhide
Recently there has been a growing interest in fairness-aware recommender systems including fairness in providing consistent performance across different users or groups of users. A recommender system could be considered unfair if the recommendations do not fairly represent the tastes of a certain group of users while other groups receive recommendations that are consistent with their preferences. In this paper, we use a metric called miscalibration for measuring how a recommendation algorithm is responsive to users' true preferences and we consider how various algorithms may result in different degrees of miscalibration for different users. In particular, we conjecture that popularity bias which is a well-known phenomenon in recommendation is one important factor leading to miscalibration in recommendation. Our experimental results using two real-world datasets show that there is a connection between how different user groups are affected by algorithmic popularity bias and their level of interest in popular items. Moreover, we show that the more a group is affected by the algorithmic popularity bias, the more their recommendations are miscalibrated.
1 INTRODUCTION
The paper frames fairness as consistent recommendation performance across user groups and investigates whether popularity bias contributes to unequal miscalibration. It defines popularity lift to measure algorithmic amplification and reports a connection between users’ interest in popular items, popularity-bias impact, and calibration.
- Motivation: Miscalibration measures whether recommendations match the spectrum of items in a user’s prior ratings.A user whose ratings are 70% action and 30% romance might expect recommendations with a similar distribution.
- Motivation: Unequal miscalibration across users or groups may indicate unfair treatment, even though miscalibration for everyone could simply reflect insufficient personalization.
- Popularity bias: Popularity bias occurs when algorithms amplify existing popularity differences, measured here by popularity lift between profile and recommendation-list popularity.
- Research focus: The paper conjectures that popularity bias contributes to miscalibration and examines whether users with different interest in popular items experience different effects.
- Contributions: The authors report that users more affected by algorithmic popularity bias tend to receive less calibrated recommendations.
2 RELATED WORK
Related work studies popularity bias, fairness, and unequal recommendation performance across user groups. Prior research addresses accuracy, long-tail exposure, demographic discrimination, and balanced utility across stakeholders.
- Popularity bias: Prior popularity-bias research mainly examines overall recommendation accuracy under long-tail rating distributions.
- Popularity bias: Other work proposes algorithms that control popularity bias by giving long-tail items more opportunities to be recommended.
- Fairness: Fairness research considers reducing discrimination against demographic groups and balancing exposure among item categories or providers.
- User-group performance: Research on user-group performance finds that some algorithms provide significantly less accurate recommendations to groups defined by age or gender.
- User-group performance: Balanced distributions of utility across stakeholders are presented as important for fair recommender systems.
3 POPULARITY BIAS AND MISCALIBRATION
This section defines fairness-related calibration measures and popularity lift by comparing user profiles with recommendation lists. It motivates testing whether algorithms amplify popularity bias differently across user groups and thereby produce miscalibrated recommendations.
- Calibration and fairness: Fairness is treated as consistent recommendation performance across users or groups, with unequal miscalibration indicating potentially unfair treatment.
- Calibration and fairness: A calibrated recommender reflects a user’s various interests in appropriate proportions within the recommended list.
- Miscalibration: For each user, miscalibration compares category distributions in rated items, P_u, and recommended items, Q_u.
- Miscalibration: The category ratios p(c|u) and q(c|u) count feature-bearing rated and recommended items using an indicator function over Γ_u and Λ_u.
- Miscalibration: Miscalibration is computed using Hellinger distance between the rated-item and recommended-item category distributions, then averaged across users in each group.
- Motivation: Because rating data is popularity-skewed, algorithms may over-recommend popular items and produce miscalibrated lists for different user groups.
- Popularity lift: Popularity lift measures how much an algorithm amplifies popularity by comparing average item popularity in group recommendations with group profiles.
- Popularity lift: Positive popularity lift indicates amplification, negative values indicate less concentration on popular items, and zero indicates no amplification.
4 METHODOLOGY
The experiments use two public movie-rating datasets, several neighborhood and latent-factor recommenders, and a most-popular baseline. Users are divided into ten groups according to the average popularity of items in their profiles.
- Datasets: MovieLens 1M contains 1,000,209 ratings from 6,040 users on approximately 3,900 movies, while core-10 Yahoo Movies contains 173,676 ratings from 7,012 users on 2,131 movies.
- Experimental setup: The experiments use an 80% training split and a 20% test split.
- Algorithms: Algorithms include UserKNN, ItemKNN, SVD++, biased matrix factorization, and a most-popular method representing extreme popularity bias.
- User groups: Users are grouped into ten groups according to the average popularity of the items in their profiles, from least to most popular.
5 RESULTS
The results connect algorithmic popularity bias with disparate miscalibration across user groups. Groups with niche tastes receive greater popularity lift and miscalibration, while item-level popularity bias can amplify genres associated with popular items rather than the most frequent genres.
- Popularity bias and disparate miscalibration: Popularity bias can contribute to unfairness when different user groups experience different levels of miscalibration.The paper frames reducing popularity bias as a possible approach to addressing disparate miscalibration.
- User-group effects: Groups with the lowest average item popularity experience the greatest algorithmic popularity lift, while groups with more popular tastes are less affected.Figure 1 compares groups’ average profile-item popularity with their observed popularity lift across two datasets.
- User-group effects: For all algorithms, niche-focused group G1 has higher popularity lift and miscalibration than blockbuster-focused group G10.The comparison concerns the two extreme groups in Table 1.
- Popularity bias and miscalibration: Popularity lift can over-represent popular genres in recommendation lists, producing overall miscalibration relative to users’ interactions.Miscalibration can theoretically also arise when non-popular genres are over-represented, but the observed mechanism involves popular items and their associated genres.
- Popular item or popular feature?: In MovieLens and Yahoo Movies, algorithms amplify genres associated with popular movies even when those genres are not the most frequent in the rating data.MovieLens examples include Action, Thriller, Sci-Fi, and Adventure; Yahoo Movies shows amplification of Action, Adventure, Sci-Fi, Fantasy, Crime, and Gangster.
6 CONCLUSION AND FUTURE WORK
The paper finds that popularity bias affects user groups differently according to their interest in popular items, and that less popular-item-oriented users experience the greatest bias and miscalibration. It proposes future experiments to study causality and whether mitigating popularity bias reduces miscalibration and unfairness.
- Conclusion: User groups are affected differently by popularity bias depending on their interest in popular items.The conclusion focuses on popularity bias from the user perspective.
- Conclusion: The group least interested in popular items experiences the greatest popularity bias and the highest miscalibration.This comparison concerns two extreme groups on the item-popularity spectrum.
- Future work: Future work will study causality by controlling popularity bias in data and examining effects on miscalibration and fairness.The authors also plan to investigate whether mitigating algorithmic popularity bias lowers miscalibration and unfairness.