PaperlayerView exact revision on arXiv

Source-linked AI summary

Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Era

Sunhao Dai, Chen Xu, Shicheng Xu, Liang Pang, Zhenhua Dong, Jun Xu

arXiv:2404.11457v2cs.IRcs.AIcs.CL

TL;DR

LLM integration is creating new bias and unfairness challenges in IR, while existing literature remains fragmented. This survey unifies these issues as distribution mismatches, reviews them across three IR stages, and organizes mitigation strategies. It identifies fifteen bias and unfairness types, reviews corresponding mitigations, and highlights open problems including feedback loops and inadequate benchmarks.

  • Problem

    LLM integration has introduced emerging bias and unfairness challenges in IR, while the literature remains fragmented and lacks a unified definition.

  • Method

    The survey reviews bias and unfairness across data collection, model development, and result evaluation, unifying them as distribution mismatch problems and categorizing mitigation strategies.

  • Results

    The survey reviews fifteen types of bias and unfairness with corresponding mitigation strategies and provides a comprehensive overview of current progress.

  • Takeaways & Limitations

    The unified framework and taxonomy are intended to guide future efforts toward eliminating bias and ensuring fairness for more trustworthy IR systems.

  • Takeaways & Limitations

    Existing benchmarks are mostly conducted in simulated environments, creating a need for large-scale real-world datasets and dynamic benchmarks.

Abstract

from arXiv · show

With the rapid advancements of large language models (LLMs), information retrieval (IR) systems, such as search engines and recommender systems, have undergone a significant paradigm shift. This evolution, while heralding new opportunities, introduces emerging challenges, particularly in terms of biases and unfairness, which may threaten the information ecosystem. In this paper, we present a comprehensive survey of existing works on emerging and pressing bias and unfairness issues in IR systems when the integration of LLMs. We first unify bias and unfairness issues as distribution mismatch problems, providing a groundwork for categorizing various mitigation strategies through distribution alignment. Subsequently, we systematically delve into the specific bias and unfairness issues arising from three critical stages of LLMs integration into IR systems: data collection, model development, and result evaluation. In doing so, we meticulously review and analyze recent literature, focusing on the definitions, characteristics, and corresponding mitigation strategies associated with these issues. Finally, we identify and highlight some open problems and challenges for future work, aiming to inspire researchers and stakeholders in the IR field and beyond to better understand and mitigate bias and unfairness issues of IR in this LLM era. We also consistently maintain a GitHub repository for the relevant papers and resources in this rising direction at https://github.com/KID-22/LLM-IR-Bias-Fairness-Survey.

1 INTRODUCTION

LLMs are reshaping IR through new data sources, model paradigms, and evaluation practices, while introducing bias and unfairness challenges. This survey unifies these issues as distribution mismatches and organizes their mitigation strategies.

  • LLM integration has introduced new IR data sources, proactive generation paradigms, and LLM-based result evaluation, alongside bias and unfairness challenges.These challenges may affect IR reliability and contribute to societal issues such as echo chambers and cognitive interference.
  • Existing literature on bias and fairness in IR, LLMs, and LLM-enhanced IR remains fragmented and lacks a unified definition.The survey addresses this ambiguity through a comprehensive perspective focused on the intersection of LLMs and IR.
  • Bias and unfairness are formulated as distribution mismatch problems against objective targets for bias and subjective human-value targets for unfairness.This formulation provides a common basis for understanding the two issues and developing mitigation strategies.
  • The survey examines bias and unfairness across data collection, model development, and result evaluation in LLM-integrated IR systems.It reviews definitions, characteristics, and mitigation strategies across these three stages.
  • Mitigation strategies are organized into data sampling and distribution reconstruction.The categories include data augmentation and filtering, plus rebalancing, regularization, and prompting.
  • The survey differs from prior work by comprehensively covering emerging bias and fairness issues at the intersection of IR and LLMs through a unified perspective on causes and mitigation.

2 A UNIFIED VIEW OF BIAS AND UNFAIRNESS

The paper views bias and unfairness in LLM-integrated IR as mismatches between predicted and target result distributions. It maps mitigation to data modifications and predicted-distribution adjustments across the reshaped IR pipeline.

  • 2.1 LLMs and IR Systems: LLMs reshape the IR pipeline across data collection, model development, and result evaluation.The integration includes LLM-generated content as data, LLM-enhanced or generative IR models, and LLM-based evaluators.
  • 2.2 Distribution Alignment Perspective: The framework models an IR system as producing predicted results bR=f(Q) from requirements, user attributes, and optional interaction history.The input is Q={T,U,H}, and results may come from IR models or direct LLM generation.
  • 2.2 Distribution Alignment Perspective: Bias and unfairness are unified as mismatches between the predicted-result distribution P(bR) and a target distribution P(R).Bias concerns objective and factual realities, whereas unfairness concerns subjective human values and social contracts.
  • 2.3 Taxonomies of Mitigation Strategies: Mitigation aims to align retrieved information with a target distribution defined by objective criteria or subjective social values.The framework groups strategies into data sampling and distribution reconstruction.
  • 2.3 Taxonomies of Mitigation Strategies: Data sampling modifies data through augmentation, which completes distributions, and filtering, which truncates them toward desired outcomes.Examples include counterfactual imputation, external knowledge, re-ranking, and constrained beam search.
  • 2.3 Taxonomies of Mitigation Strategies: Distribution reconstruction adjusts predicted distributions through rebalancing, regularization, and prompting.These strategies respectively modify weights or samples, constrain learning, or guide LLM outputs toward target distributions.

3 CAUSE AND MITIGATION OF BIAS

The survey reviews bias types arising at different stages of LLM–IR integration and pairs them with corresponding mitigation strategies.

  • Table 1 organizes bias types by stage of LLM–IR integration and discusses their corresponding mitigation strategies.

3.1 Bias in Data Collection

Bias introduced during data collection includes source bias from mixed human and LLM-generated corpora and factuality bias from non-factual generated content. The survey reviews causes and mitigation strategies for both.

  • Data-collection bias is categorized into source bias and factuality bias.
  • 3.1.1 Source Bias: Source bias occurs when retrieval models rank LLM-generated content above semantically similar human-authored content.Neural matching models can capture distinctive representations in LLM-generated text and assign it higher relevancy scores; training on such content can amplify the bias.
  • 3.1.1 Source Bias: Debiased training constraints aim to recalibrate relevancy predictions so human-written and LLM-generated content receive fairer treatment.
  • 3.1.2 Factuality Bias: Factuality bias arises when AIGC introduces non-factual or hallucinated content into IR data sources, altering the data distribution and retrieval process.
  • 3.1.2 Factuality Bias: LLMs generate factual errors across question answering, open-ended generation, natural language inference, and multi-task or multi-domain settings.TruthfulQA reports false answers resembling popular misconceptions, while FActScore finds long-form factual consistency substantially behind humans.
  • 3.1.2 Factuality Bias: Flawed or incomplete training data contributes to factuality bias, especially for rare or specialized knowledge.Low-quality errors, repetition, and limited knowledge coverage can harm generated factual correctness.
  • 3.1.2 Factuality Bias: Mitigation uses higher-quality factual training data, external knowledge bases such as retrieval-augmented generation, and methods that improve model capabilities.Examples include Self-Consistency and Dola during inference.

3.2 Bias in Model Development

LLM integration introduces model-development biases involving input position, popularity, instruction adherence, and contextual consistency. The survey organizes mitigation around prompting, data augmentation, and rebalancing, while noting computational and strategic challenges.

  • LLM-based IR models exhibit four model-development biases: position, popularity, instruction-hallucination, and context-hallucination bias.
  • Position Bias: Position bias favors documents or items from specific input positions, with models often preferring list beginnings or ends over middle items.
  • Position Bias: Mitigation for position bias uses prompting, candidate shuffling with aggregation, and rebalancing of position-sensitive prior distributions.
  • Position Bias: Position-bias mitigation can increase computational demand because multiple candidate permutations may require processing.
  • Popularity Bias: LLM-based models may favor items popular in pre-training corpora as well as in fine-tuning data, extending popularity bias beyond conventional data distributions.
  • Hallucination Biases: Instruction-hallucination occurs when generated content deviates from user instructions, while context-hallucination produces content inconsistent with long or complex context.

3.3 Bias in Result Evaluation

Using LLMs as IR evaluators introduces selection, style, and egocentric biases. These biases can affect which responses are preferred, with mitigation relying on prompting, augmentation, rebalancing, or evaluator diversity.

  • LLM-based evaluators exhibit selection, style, and egocentric bias when assessing IR results.
  • Selection Bias: Selection-bias mitigation includes prompting, position or token switching, and rebalancing, but augmentation methods can be time-consuming and costly.
  • Style Bias: Style bias favors presentation features such as longer, fluent, verbose, or visually engaging responses, even when they contain factual errors.
  • Style Bias: Current style-bias mitigation mainly uses prompts, which are often insufficient and may require architectural or training changes.
  • Egocentric Bias: Egocentric bias makes evaluators prefer outputs generated by themselves or models from the same family, creating a risk of self-reinforcement during reward-based training.

4 CAUSE AND MITIGATION OF UNFAIRNESS

The survey frames IR unfairness through differing stakeholder perspectives and fairness concepts. It distinguishes user fairness from item fairness, linking them to equitable treatment and opportunity distribution.

  • Fairness Concepts: Fairness in IR can involve alignment with cultural values such as gender equality, addressing disadvantages, and avoiding discriminatory language.
  • Fairness Concepts: Users and items may have distinct fairness perspectives, associated respectively with equality and distributive justice.
  • User Fairness: User fairness requires equitable, non-discriminatory information services that treat different users similarly.
  • Item Fairness: Item fairness seeks to equalize opportunities across diverse items by providing weaker items more exposure based on need.

4.2 Unfairness in Data Collection

Unfairness can enter IR through data collection when discriminatory content or unbalanced representation shapes training data. Mitigation redistributes, filters, downweights, or generates safer and more equitable data.

  • User Unfairness: User unfairness can arise from taxonomic, discriminatory, and offensive training content that disproportionately affects specific groups.
  • User Unfairness: Data-collection mitigation for user unfairness includes matched pairs, non-toxic examples, downweighting discriminatory samples, and content filtering.
  • Item Unfairness: Item unfairness can result from insufficient representation of certain items in collected data.
  • Item Unfairness: LLM-generated items can introduce new content and perspectives but may still contain discrimination.
  • Item Unfairness: Mitigation for item unfairness generates safe question-answer pairs, filters discriminatory items, and enriches training data with diverse equitable content.

4.3 Unfairness in Model Development

LLM integration creates user- and item-level unfairness during model development, while fine-tuning, prompting, weighting, and decoding offer mitigation strategies.

  • User unfairness: Pre-training knowledge and explicit gender or race attributes can produce discriminatory recommendations or unfair answers.
  • User unfairness: Fine-tuning mitigations include intersectional prompts, fair-aware prompt tuning, and different loss weights for samples containing discriminated content.
  • Item unfairness: LLM-based recommendation models are more prone than traditional models to unfair item outcomes, including unfair job recommendations.
  • Item unfairness: Item unfairness can reinforce item polarization, potentially creating echo chambers that limit exposure to diverse perspectives.
  • Item unfairness: Mitigation strategies for item unfairness include prompt-based learning, decoding strategies, and adding fairness terms to diffusion processes.

4.4 Unfairness in Result Evaluation

LLM-based evaluators help assess fairness objectives more efficiently than human evaluation, but may fail to represent diverse human behaviors and complicate credit attribution for generated items.

  • Evaluation setup: Fairness evaluation requires measuring social-value distributions, while LLM evaluators reduce the labor demands of human evaluation.
  • User unfairness: User unfairness arises when LLM evaluators fail to simulate real human behavior and exhibit group behavior toward particular human groups.
  • User unfairness: Evaluator fairness can be improved through psychologically informed prompts, personality-enriched training data, and personalized lexicons.
  • Item unfairness: Generated-item evaluation must attribute credit back to item providers to assess item fairness comprehensively.
  • Item unfairness: Proposed credit-assessment approaches use world knowledge, MIN-K% Prob, influence functions, embedding similarities, and watermarking for item tracking.

5 CONCLUSION AND FUTURE DIRECTIONS

The survey unifies LLM-era IR bias and unfairness as distribution mismatch problems and organizes mitigation around data sampling and distribution reconstruction. It identifies feedback loops, theoretical gaps, and inadequate benchmarks as priorities for future work.

  • Conclusion: The survey reviews fifteen types of bias and unfairness and categorizes mitigation strategies into data sampling and distribution reconstruction.
  • Biases and Unfairness in IR Feedback Loops: User, model, and information interactions can form feedback loops that reinforce existing biases and unfairness through changing perceptions, preferences, and training data.
  • Unified Mitigation Framework: A unified mitigation framework is needed because different bias and unfairness types are interconnected rather than isolated.
  • Theoretical Analysis and Guarantees: Research remains predominantly empirical, motivating more rigorous theoretical analysis and analytical frameworks.
  • Better Benchmarks and Evaluation: Current benchmarks mainly use simulated environments, creating a need for large-scale real-world datasets, dynamic benchmarks, and systematic evaluation protocols.
Loading 2404.11457v2…