Source-linked AI summary
A Survey on Popularity Bias in Recommender Systems
Anastasiia Klimashevskaia, Dietmar Jannach, Mehdi Elahi, Christoph Trattner
TL;DR
Popularity bias may limit discovery and long-tail opportunities when recommender systems focus heavily on popular items. The paper surveys 123 studies and their motivations, mitigation techniques, and evaluation methods, finding that the literature is dominated by offline computational evaluation and often relies on application-independent assumptions. It also notes that popularity-based recommendations can be beneficial in some settings, including cold-start scenarios.
Problem
Popularity bias may limit recommendation value, discovery, and long-tail exposure when systems focus on popular items, with possible harmful effects in some settings.
Method
The paper reviews and categorizes 123 papers by research motivations, mitigation approaches, and evaluation methodologies.
Results
The literature is dominated by offline computational evaluation, while in-process mitigation techniques are the most common mitigation approach.
Takeaways & Limitations
Popularity-focused recommendations are not uniformly harmful: they can support cold-start systems by helping new users rate familiar popular items.
Takeaways & Limitations
The literature lacks an agreed definition of popularity bias and often assumes that the appropriate balance between popular and long-tail recommendations is already known.
Abstract
from arXiv · showhide
Recommender systems help people find relevant content in a personalized way. One main promise of such systems is that they are able to increase the visibility of items in the long tail, i.e., the lesser-known items in a catalogue. Existing research, however, suggests that in many situations todays recommendation algorithms instead exhibit a popularity bias, meaning that they often focus on rather popular items in their recommendations. Such a bias may not only lead to the limited value of the recommendations for consumers and providers in the short run, but it may also cause undesired reinforcement effects over time. In this paper, we discuss the potential reasons for popularity bias and review existing approaches to detect, quantify and mitigate popularity bias in recommender systems. Our survey, therefore, includes both an overview of the computational metrics used in the literature as well as a review of the main technical approaches to reduce the bias. Furthermore, we critically discuss todays literature, where we observe that the research is almost entirely based on computational experiments and on certain assumptions regarding the practical effects of including long-tail items in the recommendations.
1 Introduction
Popularity bias can make recommender systems overemphasize popular items, limiting discovery and long-tail opportunities despite their potential value. This survey systematically reviews 123 papers, examining motivations, mitigation approaches, and evaluation practices.
- Motivation: A field test found that niche-item sales increased, but popular-item sales increased more and aggregate sales diversity decreased.These observations illustrate unexpected effects associated with popularity bias in practical settings.
- Motivation: Overemphasis on popular items can make recommendations obvious and reduce discovery for consumers while limiting providers’ long-tail sales opportunities.Providers may mainly promote items customers would have bought or consumed anyway.
- Survey scope: The survey systematically reviewed and categorized 123 papers by research motivations, technical approaches, and evaluation methodologies.The review covers the literature’s approaches to understanding and addressing popularity bias.
- Research motivations: Most reviewed works assume that focusing on popular items is inherently problematic, while application-specific conditions for harm or unfairness are mostly missing.The literature commonly links popularity focus with limited exposure, bias reinforcement, and reduced recommendation quality.
- Mitigation approaches: Mitigation methods are categorized as pre-processing, in-process modeling, and post-processing, with in-process techniques most common because they jointly consider competing objectives.The survey identifies in-process approaches as the predominant mitigation form.
- Evaluation: The literature relies heavily on offline experiments and abstract computational metrics, while user studies and field tests are very rare.The authors associate this pattern with a potential abstraction trap that may overlook application-specific details.
2 Background
Popularity bias is defined and discussed as recommendations concentrating on popular items, but its value or harm depends on application context, time horizon, and normative goals. The survey distinguishes impacts on discovery, diversity, fairness, and sales while noting that research often relies on application-independent assumptions and metrics.
- Sources and amplification: Popularity bias may reinforce existing item imbalances through feedback loops, reducing exposure opportunities for lesser-known items over time.Recommendations can influence consumption, giving already popular items greater benefits from increased exposure.
- Potential effects: Popularity bias is not inherently harmful or unfair: popular items may best serve provider objectives, cold-start users, or use cases centered on known content.In cold-start settings, popular items can support preference elicitation, though their ratings may provide little information.
- Definition: Popularity bias concerns recommendations that focus on popular items enough to limit system value or create harm for stakeholders.The proposed definition does not prescribe a popularity measure, bias source, or time horizon.
- Relationship to novelty and diversity: Popular-item recommendations can reduce discovery and catalog coverage, although their effects on diversity depend on the particular diversity notion and application.Aggregate diversity measures catalog coverage, while sales diversity measures concentration of sales volume.
- Relationship to diversity: A field study found that recommendations increased long-tail sales but increased popular-item sales more strongly, while sales diversity decreased.This result illustrates that absolute long-tail gains can coexist with stronger concentration effects.
- Relationship to fairness: Reducing popularity bias does not necessarily improve fairness because the outcome depends on the specific normative claim and application context.The survey focuses mainly on application-independent technical approaches rather than application-specific interactions among quality dimensions.
3 Methodology
The survey used a semi-systematic literature search combining database queries, snowballing, and Connected Papers, then manually filtered the results. It retained 54 papers for analysis and focused on popularity-bias mitigation and evaluation methodologies.
- Search strategy: The authors applied systematic-review principles while using additional discovery methods to identify relevant popularity-bias research.The process is described as semi-systematic because it combined structured review practices with supplementary discovery tools.
- Search strategy: Database queries targeted recommender-system papers published from January 1, 2000, to January 31, 2024, using popularity-bias and recommender-related terms in abstracts or keywords.Searching titles was considered too narrow, while searching full text produced too many irrelevant results.
- Search strategy: Snowballing references and the Connected Papers tool supplemented the initial database search to locate additional relevant works.These methods expanded retrieval beyond the initial keyword-based query.
- Search strategy: The search covered the ACM Digital Library, SpringerLink, ScienceDirect, and IEEE Xplore.These four digital libraries formed the database sources for the initial retrieval step.
- Corpus construction: After duplicate removal and manual relevance filtering, the survey retained 54 papers and shared the detailed list online for reproducibility.The retained papers were used in the subsequent analyses.
- Scope: The survey emphasizes in-depth coverage of popularity bias, technical mitigation approaches, and common evaluation methodologies.This scope distinguishes it from broader work whose technical treatment of popularity bias is less detailed.
4 Survey results: a landscape of research
The survey maps how research on popularity bias characterizes the problem, motivates its study, and contributes methods across application domains. It finds broad reliance on technical mitigation and offline evaluation, alongside concerns about oversimplified assumptions and weak domain-specific justification.
- Survey scope: The survey organizes the research landscape by problem characterizations, motivations, application domains, and contribution types.The analysis codes researchers’ problem statements and negative-effect descriptions, groups datasets into application domains, and classifies papers by their main novel contributions.
- Problem characterizations: Most papers characterize popularity bias as recommendations focusing too strongly on popular items, while long-tail underexposure is the second most frequent characterization.Only a small number of works characterize the phenomenon primarily through underlying data, and few explicitly discuss whether relevant long-tail items are hindered.
- Critical observations: The literature often assumes that popularity focus is inherently problematic, while application-specific harms and the rationale for selected evaluation domains are frequently underexplained.The survey notes that these patterns may oversimplify the problem, and that popularity may not be problematic in every use case.
- Research motivations: Researchers report potential harms for users, platforms, and providers, including limited personalization, diversity, novelty, discovery, exposure, business opportunities, and consumer trust.Potential reinforcement effects are also mentioned, but many studies do not investigate them longitudinally.
- Application domains: The literature is concentrated in media applications, with movies dominating, while e-commerce and tourism receive less attention and many other domains appear only sparsely.Application domains were identified mainly from datasets used in offline experiments.
- Types of contributions: Most papers propose technical approaches to mitigate popularity bias; fewer quantify existing bias or develop computational metrics, and only a limited number use popularity information to improve recommendations.Papers can belong to more than one contribution category.
5 Technical approaches to deal with popularity bias
The survey organizes popularity-bias research around quantification, mitigation, and utilization, while reviewing technical strategies by processing stage and highlighting practical and methodological caveats.
- Bias quantification: Popularity-bias research quantifies bias, studies its effects on users, and compares recommendation algorithms using measures such as accuracy.Studies may examine the entire user base or demographic and preference-based subgroups.
- Bias over time: User-feedback simulations indicate that different algorithms can either reinforce or reduce popularity bias over time.These simulations model recommendations, user acceptance, feedback, and retraining as an evolving loop.
- Bias over time: Auditing with bots found that YouTube recommended increasingly popular but topically unrelated videos.The study simulated user experience through random walks over recommended videos.
- Processing-stage taxonomy: Mitigation approaches are categorized as pre-processing, in-processing, or post-processing according to where they intervene in the recommendation pipeline.Pre-processing adapts data, in-processing incorporates mitigation into learning, and post-processing re-ranks output lists.
- Processing-stage taxonomy: The processing-stage categorization is partly interpretive, especially when distinguishing in-processing from post-processing approaches.An output-oriented perspective could classify almost every approach as in-processing.
- Selected technical approaches: Propensity-score methods address selection bias by increasing exposure for item groups and can incorporate causal, counterfactual, dynamic, explicit-rating, or implicit-feedback considerations.One implicit-feedback approach models both positive propensities from clicks and negative propensities from missing data.
- Scope and caveats: Popularity is not uniformly undesirable: some work distinguishes genuine item quality from other popularity effects, while other work shows popularity bias can enable attacks that skew distributions.Such attacks may harm trustworthiness and provider fairness.
6 Evaluation approaches
The surveyed literature relies predominantly on offline evaluations, while user studies and field tests are rare. Differences in evaluation methods, datasets, metrics, and preprocessing make direct comparisons difficult, and the limited user evidence suggests that popularity mitigation can involve trade-offs and context-dependent effects.
- Datasets and scope: Frequently used datasets may not represent real-world popularity-bias settings, while public data from critical domains such as healthcare and banking remain scarce.The survey questions whether findings and mitigation techniques from domains such as movie recommendation generalize to these application areas.
- Evaluation approaches: Offline experiments strongly dominate the surveyed evaluation approaches across domains and contribution types.The same tendency appears regardless of the application domain or research contribution.
- Comparability: Evaluation results are difficult to compare because methodologies, datasets, algorithms, hyper-parameters, popularity thresholds, and metrics vary across studies.The surveyed literature also applies different preprocessing procedures, and some are not documented in detail.
- Human-in-the-loop online evaluation: User studies report context-dependent effects: popular recommendations performed best in low season, whereas a hybrid method performed best in high season.The hotel recommendation study found that popularity can be effective in some cases and is not necessarily always harmful.
- Human-in-the-loop online evaluation: Computational popularity-bias metrics often do not correlate well with user perceptions, emphasizing the need to assess metric validity against user judgments.Two user studies examined whether users notice recommendation popularity and debiasing and how these perceptions align with calculated metrics.
- User studies: User studies are uncommon and often involve only a few dozen participants, limiting assessment of generalizability.Only one cited study involved a larger sample, while some user studies complemented offline experiments with few reported details.
7 Discussion, research gaps, and future directions
The survey identifies oversimplified assumptions, inconsistent evaluation practices, and limited real-world evidence as major gaps in popularity-bias research. It calls for application-specific definitions, broader datasets, and evaluation methods involving users, stakeholders, simulations, and field studies.
- Definition, applications, and datasets: No agreed-upon definition of popularity bias has emerged despite substantial research uptake.
- Definition, applications, and datasets: Researchers often assume that popular recommendations are inherently harmful without establishing context-specific harms or mitigation success criteria.A 10% reduction on one metric may still leave many popular items in recommendations.
- Definition, applications, and datasets: Practical applications require determining when an item is unpopular and how to balance users’ preferences with exploration.Academic studies sometimes define the least popular 30% as items that should be recommended more often.
- Definition, applications, and datasets: The survey’s value- and impact-oriented definition ties popularity-bias mitigation to whether bias limits recommendation value or harms stakeholders in a specific application.The authors argue that application characteristics should guide metric selection instead of arbitrary thresholds.
- Future directions: Future research should use more public datasets and richer methods, including simulations, human-in-the-loop evaluations, and field studies.Offline experiments remain useful for comparing popularity tendencies and studying longitudinal reinforcement effects.
- Methodological issues: Evaluation research relies heavily on offline experiments, generic metrics, and datasets concentrated in media domains such as MovieLens.These patterns make real-world effects, cross-study comparison, reproducibility, and progress difficult to assess.
8 Summary
Popularity-biased recommender systems may provide limited value to users and providers and may have harmful effects in some application settings. The survey reviews this research area while identifying substantial need to understand its real-world implications.
- Popularity-biased recommender systems may be of limited value to users and providers and harmful in certain application settings.
- The literature still has many future research directions, especially concerning the real-world implications of popularity bias.
Declarations
The authors declare no conflict of interest and provide the article under a Creative Commons Attribution 4.0 International License.
- The authors declare no conflict of interest.
- The article is licensed under a Creative Commons Attribution 4.0 International License, subject to attribution and license-link requirements.