PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Fairness in Large Language Models

Yingji Li, Mengnan Du, Rui Song, Xin Wang, Ying Wang

arXiv:2308.10149v2cs.CLcs.AI

TL;DR

LLMs can encode social biases from unprocessed data and carry them into downstream tasks, while fairness research spans different model sizes and training paradigms. This survey organizes the literature by those differences, reviewing evaluation and debiasing methods across intrinsic and extrinsic bias and prompting-based systems. It highlights that intrinsic and extrinsic bias need not correlate and that incompatible metrics make benchmark reliability difficult to guarantee.

  • Problem

    LLMs can learn social biases from unprocessed training data and propagate them to downstream tasks, while fairness research differs across model sizes and training paradigms.

    Source: §1 · pp. 1–2

  • Method

    The survey classifies fairness research by model size and training paradigm, reviewing intrinsic and extrinsic evaluation and debiasing for medium-sized models and fairness work on prompting-based large models.

    Source: §1 · p. 2, §8 · p. 20

  • Results

    Intrinsic and extrinsic bias are not necessarily correlated, and incompatible metrics make benchmark reliability difficult to guarantee.

    Source: §7.1 · p. 19

  • Takeaways & Limitations

    The authors recommend investing in training-data processing because mitigating bias through training strategy alone cannot fundamentally eliminate model bias.

    Source: §7.4.2 · p. 20

    Intrinsic and extrinsic bias may not correlate, and incompatible metrics make benchmark reliability difficult to guarantee.

    Source: §7.1 · p. 19

Abstract

from arXiv · show

Large Language Models (LLMs) have shown powerful performance and development prospects and are widely deployed in the real world. However, LLMs can capture social biases from unprocessed training data and propagate the biases to downstream tasks. Unfair LLM systems have undesirable social impacts and potential harms. In this paper, we provide a comprehensive review of related research on fairness in LLMs. Considering the influence of parameter magnitude and training paradigm on research strategy, we divide existing fairness research into oriented to medium-sized LLMs under pre-training and fine-tuning paradigms and oriented to large-sized LLMs under prompting paradigms. First, for medium-sized LLMs, we introduce evaluation metrics and debiasing methods from the perspectives of intrinsic bias and extrinsic bias, respectively. Then, for large-sized LLMs, we introduce recent fairness research, including fairness evaluation, reasons for bias, and debiasing methods. Finally, we discuss and provide insight on the challenges and future directions for the development of fairness in LLMs.

1. Introduction

LLMs can capture social biases from unprocessed training data and carry them into downstream decisions, creating potential harms. This survey organizes fairness research by model size and training paradigm, distinguishing its approach from prior surveys.

1. Introduction

  • Biases learned from unprocessed training data can carry into downstream decisions, where unfair systems may harm vulnerable or marginalized groups.The introduction gives examples of biased occupational and healthcare decisions.Source: §1 · pp. 1–2
  • The survey argues that fairness research should be synthesized across model sizes and training paradigms because these differences bring distinct research approaches and challenges.The authors contrast their fine-grained classification with Gallegos et al. [21], which they say does not account for those differences.Source: §1 · p. 2
  • The review separates medium-sized models under pre-training and fine-tuning from large-sized models under prompting, then surveys evaluation, debiasing, and future directions.It covers intrinsic and extrinsic methods for medium-sized models and evaluation, bias sources, and debiasing for large-sized models.Source: §1 · p. 2

2. Fairness in LLMs

This section frames fairness in LLMs computationally, identifies several sources of algorithmic bias, and distinguishes how bias is defined across training paradigms. It separates medium-sized models’ representation and downstream-output biases from bias observed in large models’ generated responses.

2. Fairness in LLMs

  • The section adopts a computational view of fairness and defines language-model social bias as assuming a person has a characteristic based on group membership.It connects such bias in encoding and downstream use with unfairness toward the affected group.Source: §2 · p. 2

2.1. Sources of Algorithmic Bias

  • Algorithmic bias sources include harmful corpora or subjective labels, demographic train–test distribution shifts, unexpected encoding effects, and bias amplified during pre-training or fine-tuning.The section names these label, sampling, semantic, and amplifying biases.Source: §2.1 · p. 3

2.2. Defining Bias and Fairness in LLMs

  • The section distinguishes pre-training and fine-tuning from prompting, noting that model size and training paradigm shape how social bias manifests.Pre-training and fine-tuning are described as the traditional strategy, while prompting is presented as suitable for large models.Source: §2.2 · p. 4, §2.2.1 · p. 4, §2.2.2 · p. 5

2.2.1. Pre-training and Fine-tuning Paradigm

  • For medium-sized models, intrinsic bias is task-independent representation bias, while extrinsic bias concerns outputs on specific downstream tasks.The corresponding evaluation metrics and debiasing strategies are classified as intrinsic or extrinsic.Source: §2.2.1 · p. 4
  • Demographic groups are represented through sensitive attributes, and the section frames intrinsic and extrinsic evaluation around differences between group-representing samples.The definitions introduce intrinsic and extrinsic metrics and fairness thresholds for those comparisons.Source: §2.2.1 · p. 4

2.2.2. Prompting Paradigm

  • For prompting models, social bias can be reflected in generated responses, particularly when model representations are unavailable.The section identifies quantifying generated bias and designing prompts as additional concerns.Source: §2.2.2 · p. 5

3. Evaluation Metrics

The section organizes medium-sized LLM fairness evaluation into intrinsic metrics applied to embeddings and extrinsic metrics applied to downstream-task outputs. It surveys similarity- and probability-based intrinsic measures, alongside NLU- and NLG-based extrinsic benchmarks.

3.1. Intrinsic Bias Evaluation Metrics

  • Intrinsic metrics quantify associations between targets and concepts in embeddings, while extrinsic metrics characterize bias through downstream-task performance gaps.Source: §3.1 · p. 5, §3.2 · p. 8

3.1.1. Similarity-based Metrics

  • WEAT-style similarity measures compare associations across word or contextual embeddings; SEAT’s results can vary with embedding selection and may not reliably indicate stereotypes.CEAT extends WEAT to distributions of effect sizes in contextualized embeddings; lower scores indicate a target group is closer to the negative-polarity attribute.Source: §3.1.1 · pp. 5–7

3.1.2. Probability-based Metrics

  • Probability-based metrics quantify intrinsic bias from candidate-word or candidate-sentence probabilities, using templates or evaluation datasets.Source: §3.1.2 · p. 7
  • LPBS adjusts for target-attribute prior probabilities, and Categorical Bias extends it to multiclass targets using normalized probabilities across sentence templates.Categorical Bias equals LPBS with two sentence templates.Source: §3.1.2 · pp. 7–8
  • StereoSet and CrowS-Pairs assess stereotyped associations; AUL modifies CrowS-Pairs by combining multiple correct predictions rather than testing only the target token.StereoSet offers stereotyped, anti-stereotyped, and meaningless choices; CrowS-Pairs uses pseudolog-likelihood over semantically contrasting sentence pairs.Source: §3.1.2 · p. 8

3.2.1. NLU-based Metrics

  • NLU-based benchmarks measure bias in tasks including coreference, semantic similarity, inference, classification, sentiment, and emotion prediction.WinoBias compares gender-pronoun associations with stereotyped and anti-stereotyped occupations; Winogender includes neutral gender and one occupation per instance.Source: §3.2.1 · pp. 8–9

3.2.2. NLG-based Metrics

  • NLG-based measures evaluate generated completions, conversations, and answers, including sentiment, toxicity, conversational-style, and stereotype-bias measures.BOLD covers five domains, while BBQ tests responses under adequate and insufficient contextual information.Source: §3.2.2 · pp. 10–11
  • For counterfactual sentiment evaluation, CSB measures individual and subgroup fairness using Wasserstein-1 distances between sentiment-score distributions.Source: §3.2.2 · p. 10

4. Intrinsic Debiasing

Intrinsic debiasing mitigates representational bias before downstream use and is task-agnostic. The reviewed preprocessing approaches alter training data through counterfactual balancing or calibration of harmful information.

4.1. Pre-processing

  • Preprocessing methods address deficiencies in training data before model training.Source: §4.1 · p. 11

4.1.1. CDA-based

  • Counterfactual Data Augmentation balances training samples by replacing sensitive attributes with those of another demographic group.For binary gender debiasing, an example replaces “He is a doctor” with “She is a doctor.”Source: §4.1.1 · p. 11

4.1.2. Data Calibration

  • Data calibration creates fairer corpora by removing potentially biased texts or tuning parameters with small intervention-based datasets.For morphologically complex languages, the review describes generating biased text from fair text through machine-translation round trips.Source: §4.1.2 · p. 11

4.2. In-processing

In-processing methods incorporate fairness into model design by tuning parameters. The review groups approaches into retraining optimization, disentanglement, alignment constraints, and contrastive learning.

4.2. In-processing

  • In-processing incorporates fairness into LLM design and obtains a fairer model through parameter tuning.Source: §4.2 · p. 11

4.2.1. Retraining Optimization

  • Retraining can directly reduce bias, but may be resource-intensive and difficult to scale.Approaches include dropout regularization, demographic-perturbation retraining, and fairness-aware distillation.Source: §4.2.1 · p. 12

4.2.2. Disentanglement

  • Disentanglement methods separate biased concepts from useful information, with approaches ranging from projection to filtering bias subspaces while retaining factual gender information.OSCAR disentangles connections between biased concepts rather than removing them all.Source: §4.2.2 · p. 12

4.2.3. Alignment Constraint

  • Alignment methods constrain representations across sensitive attributes; examples include max-min prompt debiasing, cross-lingual embedding alignment, and reducing attention to biased tokens.Source: §4.2.3 · p. 12

4.2.4. Contrastive Learning

  • Contrastive learning narrows distances between positive samples from different populations and pushes negative pairs apart.MABEL applies this objective to gender-balanced entailment pairs, while CCPA learns a continuous biased prompt.Source: §4.2.4 · p. 12

4.3. Post-processing

Post-processing freezes pretrained LLM parameters and debiases output representations. The reviewed methods include projection, parameter-efficient tuning, and additional debiasing modules.

4.3. Post-processing

  • Post-processing methods keep pretrained LLM parameters fixed while debiasing output representations.Source: §4.3 · p. 12

4.3.1. Projection-based

  • Projection-based methods isolate and remove bias-associated components, but removing only useless information is difficult and risks compromising original semantics.Iterative Gradient-Based Projection uses a probe classifier and loss gradients to guide representation projection.Source: §4.3.1 · p. 13

4.3.2. Parameter-e

  • Parameter-efficient approaches freeze LLM parameters while tuning adapters or equality prompts, aiming to preserve useful information and avoid catastrophic forgetting.Some methods also allow fairness-efficiency trade-offs to be controlled on demand.Source: §4.3.2 · p. 13

4.3.3. Additional Debiasing Module

  • Additional debiasing modules after the encoder include contrastive fair filters and variational autoencoders that separate sensitive information in latent space.Source: §4.3.3 · p. 13

5. Extrinsic Debiasing

Extrinsic debiasing targets fairness in downstream tasks through data-centric corrections and model-centric training frameworks. The surveyed approaches balance demographic representation, fairness objectives, and predictive performance in different ways.

5. Extrinsic Debiasing

  • Extrinsic debiasing is task-specific: data-centric methods correct training-data defects, while model-centric methods incorporate fairness objectives or supporting techniques into learning [p09, p10, p17].Source: §5 · p. 13, §5.1 · p. 13, §5.2 · p. 14

5.1.1. Data Augmentation

  • Data augmentation balances corpora with counterfactual examples, while GE-score pruning removes low-contribution pairs and PANDA-based fine-tuning can improve downstream fairness.Mix-Debias uses CDA-expanded gender-balanced downstream data for sentence-embedding mixup.Source: §5.1.1 · pp. 13–14

5.1.2. Data Calibration

  • Data calibration adjusts task-specific corpora through gender labeling, balanced datasets, relabeling, proxy-word removal, or fairness-informed sample selection.Calibration can reduce harmful texts and balance labels before training, improving prediction fairness.Source: §5.1.2 · p. 14

5.1.3. Instance Weighting

  • Instance weighting reduces the influence of potentially biased examples; BLIND detects demographic shortcuts, while self-debiasing anneals weights to preserve in-distribution performance.The approach treats classification bias as selection bias and aims to recover a non-discriminatory distribution.Source: §5.1.3 · p. 14

5.2.1. Regularization Constraint

  • Regularization adds fairness constraints to downstream training, including penalties on spurious features, gender-tuning objectives, embedding similarity, and demographic information in generated text.Gender-tuning jointly predicts class labels and masked gender-associated words; generation debiasing minimizes mutual information between demographic polarity and sentence semantics.Source: §5.2.1 · p. 14

5.2.2. Adversarial Learning

  • Adversarial learning trains an encoder to prevent an attacker from recovering protected attributes, but sensitive information can remain in representations and outputs; orthogonal discriminators are one proposed enhancement.Source: §5.2.2 · p. 15

5.2.3. Auxiliary Classifier

  • Auxiliary-classifier approaches include INLP, which projects representations into target-attribute classifiers’ null spaces, and correction layers that reweight input features before classification.Source: §5.2.3 · p. 15

5.2.4. Contrastive Learning

  • Contrastive learning encourages representations to cluster by task labels rather than sensitive attributes, with related supervised methods constraining representational and experience fairness [136].Source: §5.2.4 · p. 15

6. Fairness of Large-sized LLMs

Research on fairness in large-sized LLMs addresses evaluation, sources of bias, and debiasing. These models are rapidly developing and increasingly deployed in real-world scenarios, alongside growing fairness concerns.

6. Fairness of Large-sized LLMs

  • The section surveys fairness research on large-sized LLMs across evaluation, bias sources, and debiasing methods.Source: §6 · p. 15
  • Large-sized LLMs with billion-level parameters and prompting paradigms are developing rapidly.Source: §6 · p. 15
  • As these models are deployed in more real-world scenarios, fairness concerns are growing.Source: §6 · p. 15

6.1. Evaluating Fairness of Large-sized LLMs

Fairness evaluation of prompted large LLMs examines generated content across several task formats and measurement strategies. The surveyed work assesses demographic representation, stereotypes, counterfactual responses, and group performance disparities.

6.1. Evaluating Fairness of Large-sized LLMs

  • Evaluations analyze generated content from prompted models using demographic representation, stereotypical association, counterfactual fairness, and performance disparities across generative tasks.Table 2 classifies the evaluation methods, and Figure 5 illustrates the strategies.Source: §6.1 · p. 15, Table 2 · p. 16, Figure 5 · p. 16

6.1.1. Demographic Representation

  • Demographic-representation methods count demographic mentions or compare gender-word probabilities; GPT-3 favored men in 83% of 388 occupations in one prompt-completion study.A modified CATs setup prompts stereotypes and measures the probability gap for one generated gender word [140].Source: §6.1.1 · p. 16

6.1.2. Stereotypical Association

  • Stereotypical-association studies probe co-occurrence, completion, stories, and analogy; GPT-3 outputs reflected training-data stereotypes and showed greater violent prejudice against Muslims than other religious groups.Other work measures gender-occupation associations and stereotype-related toxicity in generated content [144].Source: §6.1.2 · pp. 16–17

6.1.3. Counterfactual Fairness

  • Counterfactual-fairness tests alter demographic terms in prompts and check output invariance; ChatGPT showed unfairness in high-stakes domains despite outperforming small models.Source: §6.1.3 · p. 17

6.1.4. Performance Disparities

  • Performance-disparity evaluations compare outcomes across groups: BBQ bias decreases with disambiguated context, while DecodingTrust finds GPT-4 less fair in imbalanced tests despite greater accuracy in balanced tests.Other approaches use downstream classifiers and fairness metrics, automated bias questions covering 8,110 attributes across 841 groups, and 70 generated decision scenarios [149].Source: §6.1.4 · pp. 17–18

6.2. What are the Reasons for Model Bias?

Studies of large-model bias investigate training-data patterns and representation choices. The reported findings link generated bias to corpus frequencies and describe group skews in model and dataset outputs.

6.2. What are the Reasons for Model Bias?

  • Because newer models such as GPT-4 and LLaMA-2 may undergo a capability phase transition, explanations of bias in earlier models may not transfer directly [159].Source: §6.2 · p. 18
  • LLaMA-2 generation bias correlates with training-data frequencies: male pronouns are more frequent, while female identity terms occur in a larger share of documents.Identity terms also skew toward LGBTQ+ sexual orientation and Western groups; similar pronoun patterns appear in other similarly sized models.Source: §6.2 · p. 18
  • GPT-4 occupation-gender selections reflect real-world occupational gender proportions, while NLPositionality finds datasets and models favor advantaged groups and may further marginalize non-binary people and non-native English speakers [159].Source: §6.2 · p. 18

6.3. Debiasing Large-sized LLMs

Large-sized LLMs are harder to debias, with instruction fine-tuning and prompt engineering offering different approaches. Their effects vary: some fine-tuning improves fairness measures, while prompt effectiveness is unstable.

6.3.1. Instruction Fine-tuning

  • InstructGPT fine-tunes GPT-3 with RLHF to follow human instructions and was verified to produce significantly less toxicity, while bias was evaluated on modified Winogender and CrowS-Pairs.Instruction-dataset fine-tuning has been used to mitigate model bias in zero-shot and few-shot tasks.Source: §6.3.1 · p. 18
  • LLaMA-2-chat’s fairness was greatly improved over the pre-trained LLaMA-2 base model after security fine-tuning.The techniques included adversarial prompts and security demonstrations, a security-specific reward model, and security context distillation.Source: §6.3.1 · p. 19

6.3.2. Prompt Engineering

  • Prompt engineering can change model behavior without further training; examples include inclusive pronouns for GPT-4 [159] and fairness-oriented prompts for Claude 2.0 [149].This approach can save computational resources, particularly for very large LLMs.Source: §6.3.2 · p. 19
  • Prompt debiasing is unstable: higher-abstraction prompts helped GPT-3 more [140], while job-ad prompts did not outperform zero-shot generation [165].Fine-tuning on unbiased real ads performed better than those prompts in the ad-generation task [165].Source: §6.3.2 · p. 19

7. Discussions

The discussion identifies gaps in fairness evaluation, bias explanation, and debiasing, especially for large-sized LLMs. It calls for broader measurement and datasets, deeper inquiry into bias sources, and development-stage attention to fairness.

7. Discussions

  • Fairness research on medium-sized LLMs remains limited, while large-sized LLM fairness is still developing and a societal focus.The discussion frames its proposals as responses to shortcomings and challenges in current research.Source: §7 · p. 19

7.1. Unreliable Correlation between Intrinsic and Extrinsic Biases

  • Intrinsic and extrinsic bias metrics may not correlate, and metrics can conflict, limiting benchmark reliability and motivating downstream-task evaluation and new challenge sets.The discussion notes that intrinsic debiasing has benefited downstream fairness in some work, but other studies find no necessary correlation.Source: §7.1 · p. 19

7.2.1. Expand Methods for Quantifying Bias

  • For large-sized LLMs, closed models are often assessed only through responses, while human judgments are resource-intensive and may introduce annotator bias.The survey proposes statistical principles and automated measurement from more perspectives.Source: §7.2.1 · pp. 19–20

7.2.2. Develop More Diverse Datasets

  • Existing datasets may not reflect model performance, while large-LLM-specific BBQ and BiasAsker cover limited ranges of tasks and biases.The survey calls for diverse, comprehensive benchmark datasets specific to large-sized LLMs.Source: §7.2.2 · p. 20

7.3. Further Explore the Reasons for Bias

  • Large-model bias explanations include corpus–stereotype association analyses [169], and the survey calls for inquiry from broader perspectives.It points to psychological, political, and model-factor investigations of medium-sized LLMs as inspiration.Source: §7.3 · p. 20

7.4.1. Improve Current Debiasing Strategies

  • RLHF-based instruction fine-tuning relies heavily on large auxiliary models and heuristics, making implementation costly and difficult to generalize.The survey calls for lower-cost debiasing and more targeted, controllable prompts that generalize across models.Source: §7.4.1 · p. 20
  • Prompt engineering remains in its infancy, while early GPT-4 self-reflection and reasoning about beliefs are described as opportunities for guiding behavior [159].The survey suggests combining prompt tuning with techniques such as interpretability methods.Source: §7.4.1 · p. 20

7.4.2. Consider Fairness During Development

  • Mitigating bias through training strategy alone cannot fundamentally eliminate model bias, so the survey urges data processing during development.It particularly encourages investing in processing training data rather than ingesting everything on the network.Source: §7.4.2 · p. 20

8. Conclusions

The conclusion organizes fairness research by model scale and training paradigm, then finds that evaluation, bias-source analysis, and debiasing strategies still need strengthening. It particularly identifies fairness in large-sized LLMs as early-stage research.

8. Conclusions

  • The survey divides fairness research into medium-sized LLMs under pre-training and fine-tuning and large-sized LLMs under prompting.It reviews intrinsic and extrinsic evaluation and debiasing for medium-sized models, and evaluation, bias reasons, and debiasing for large-sized models.Source: §8 · p. 20
  • The survey concludes that LLM fairness research needs stronger evaluation, bias-source investigation, and debiasing strategies.It discusses challenges and future research directions for fairness development.Source: §8 · pp. 20–21
  • Fairness research on large-sized LLMs remains at an early stage, and practitioners should combine more techniques to build comprehensive and safe language-model systems.Source: §8 · p. 21
Loading 2308.10149v2…