PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Understanding and Mitigating Social Biases in Language Models

Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, Ruslan Salakhutdinov

arXiv:2106.13219v1cs.CLcs.AIcs.CYcs.LG

TL;DR

Language models deployed in real-world applications can reproduce social biases, while existing bias tests do not fully capture token-level, sentence-level, and contextual effects. The paper defines these bias sources, introduces benchmarks and A-INLP for post-hoc debiasing, and reports reduced gender and religious stereotypes while retaining contextual information. Its conclusion is that these methods advance the performance-fairness frontier, though diverse real-world contexts remain difficult and other bias types are outside scope.

  • Problem

    Text-generation bias can arise at individual tokens or across whole sentences, and mitigation must preserve legitimate context associations across diverse contexts.

  • Method

    The paper defines local and global representational biases, develops benchmarks and metrics, and applies A-INLP to dynamically detect bias-sensitive tokens for post-hoc debiasing.

  • Results

    The empirical results and human evaluation show effective mitigation of gender and religious stereotypes while retaining contextual information for text generation.

  • Takeaways & Limitations

    The proposed evaluation and debiasing approach pushes the performance-fairness frontier for large pretrained language models.

  • Takeaways & Limitations

    Diverse real-world contexts remain significantly harder to detect and mitigate, and allocational harms and questionable correlations are left for future work.

Abstract

from arXiv · show

As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes. Among such real-world deployments are large-scale pretrained language models (LMs) that can be potentially dangerous in manifesting undesirable representational biases - harmful biases resulting from stereotyping that propagate negative generalizations involving gender, race, religion, and other social constructs. As a step towards improving the fairness of LMs, we carefully define several sources of representational biases before proposing new benchmarks and metrics to measure them. With these tools, we propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information for high-fidelity text generation, thereby pushing forward the performance-fairness Pareto frontier.

1. Introduction

The paper frames representational bias in language models as a real-world fairness concern and separates local, global, and contextual dimensions for evaluation and mitigation. It introduces benchmarks, metrics, and A-INLP to reduce bias while preserving useful context.

  • Language models used in applications such as generation, dialogue, recommendation, and search can shape social biases and stereotypes.
  • Representational biases include stereotyping, unequal performance across groups, population misrepresentation, and denigrating language.
  • Text-generation bias requires finer analysis than embedding association tests because it can arise at individual tokens or across whole sentences.
  • Bias mitigation must preserve legitimate contextual associations, such as the relationship between surgery and doctor, while reducing gendered associations.
  • The paper distinguishes local biases in context-conditioned token predictions from global biases spanning multiple phrases in generated sentences.
  • A-INLP performs post-hoc debiasing by dynamically identifying bias-sensitive tokens and generalizing beyond predefined word lists across gender and religion contexts.

2. Related Work

Related work measures and mitigates social biases in generated text and embeddings, while this paper supplements it by separating bias sources and evaluating them with benchmarks. The paper also distinguishes representational bias from other harms left for future work.

  • Prior work evaluates social bias, human-aligned values, social-bias implications, and toxic speech in generated text.
  • Existing generation-debiasing approaches include adversarial trigger prompts, data augmentation, and data collection, whereas this paper proposes complementary target methods.
  • Embedding research shows undesirable social associations can be propagated, motivating binary and multiclass debiasing methods across gender, race, and religion.
  • Table 1 organizes benchmarks and metrics for local bias, global bias, and language-model performance, including diverse real-world contexts.
  • Allocational harms and questionable correlations are important bias perspectives that the paper leaves for future work.

3. Defining Sources of Biases in LMs

The paper defines language-model prediction and separates representational bias into local token-level and global sentence-level forms. It proposes counterfactual metrics that assess bias while preserving context associations using diverse real-world contexts.

  • A language model predicts a distribution over next vocabulary tokens from prior context across successive time steps.
  • The next-token probability is implemented from token embeddings and an encoded context, using a softmax over their distances.
  • The paper focuses exposition on binary gender but states that its approach generalizes to multiclass social groups.
  • Local bias: Local bias occurs when counterfactual edits to a social-group term change the next-token distribution, measured with KL divergence and Hellinger distance.
  • Local bias: These local metrics extend association tests by comparing all tokens while weighting more likely tokens more heavily.
  • Global bias: Global bias compares sentiment and regard differences between complete sentences generated from counterfactual contexts using absolute classifier-score differences.
  • Context and benchmarks: The evaluation separates biased associations from legitimate context associations by testing both counterfactual bias differences and probability assigned to the ground-truth next word.
  • Context and benchmarks: The benchmark collects 16,338 diverse contexts from five real-world text corpora spanning spoken and written English in formal and informal settings.

4. Mitigating Biases

The paper mitigates local and global language-model bias by learning bias-sensitive tokens and applying autoregressive iterative nullspace projection during generation. A tunable mixture with the original model controls the fairness–performance trade-off, while an automatically learned weight adapts debiasing to each timestep.

  • 4. Mitigating Biases: A-INLP learns bias-sensitive tokens and mitigates their effects through autoregressive iterative nullspace projection.The method extends INLP from embeddings to autoregressive generation using context-dependent bias classifiers.
  • 4.1. Identifying Bias-Sensitive Tokens: Bias-sensitive tokens are identified by projecting candidate token embeddings onto a learned bias subspace and selecting tokens with high projection values.The subspace is derived from definitional bias pairs, whose embedding differences are decomposed with SVD.
  • 4.2. Mitigating Bias via Nullspace Projection: At each generation step, INLP projects the context embedding into the nullspace of a trained bias classifier before computing the debiased token distribution.The resulting distribution is sampled to generate the next token.
  • 4.2. Mitigating Bias via Nullspace Projection: The final next-token distribution interpolates between the debiased and original language-model distributions using α.α = 0 recovers the original predictions, whereas α = 1 fully applies INLP at all time steps.
  • 4.2. Mitigating Bias via Nullspace Projection: The automatically learned α_t increases when likely next tokens overlap with the bias-sensitive-token set and decreases otherwise.Candidate tokens are selected by intersecting likely model predictions with the learned sensitive-token set.
  • 4.2. Mitigating Bias via Nullspace Projection: The approach can also use token-level subspace debiasing instead of INLP with little difference in language-model performance.This alternative is described as another instantiation of the token-level debiasing step.

5. Experiments

Experiments evaluate bias detection and mitigation on pretrained GPT-2 using simple and diverse contexts, multiple bias metrics, and human judgments. Classifier generalization improves with diverse contexts and subsequences, while the experiments compare A-INLP variants across fairness and performance measures.

  • 5.2. Results on Learning a Bias Classifier: Automatically detected bias-sensitive tokens often align with human perceptions of harmful gender and religious stereotypes.The analysis reports especially negative stereotyping for female-gender and Muslim-religion terms.
  • 5.3. Results on Bias Mitigation: A-INLP TUNE α exposes a performance–fairness trade-off, A-INLP LEARN α automatically balances it, and A-SUBSPACE provides another effective variant.Figure 3 reports bias metrics separately for gender and religion contexts.
  • 5.2. Results on Learning a Bias Classifier: Classifiers trained only on simple contexts cannot generalize to diverse contexts, whereas adding diverse contexts improves generalization to both context types.Adding subsequences further improves detection of bias in partial input contexts.
  • 5. Experiments: Experiments apply the proposed approach to pretrained GPT-2 and compare it with established and newly proposed benchmarks and metrics.The study analyzes intermediate bias-token and classifier objectives before testing mitigation.
  • 5.1. Experimental Setup: The evaluation spans three datasets, including simple contexts and proposed diverse contexts for measuring local and global bias.Simple contexts use templates, while diverse contexts target fairness and language-model performance in real-world settings.

3. StereoSet (Nadeem et al., 2020) is a recently proposed

The paper evaluates A-INLP against GPT-2 and related baselines on StereoSet and generation-based fairness measures. Results indicate improved fairness, including reduced stereotype scores and better regard equality, while preserving language-modeling or text-quality performance, though diverse contexts remain harder to debias.

  • A-INLP LEARN α consistently outperforms other approaches on global performance and fairness, pushing the performance-fairness Pareto frontier outward.
  • A-INLP equalizes global regard scores across male and female contexts, especially for sentences receiving negative regard.
  • Debiasing is significantly harder in diverse real-world contexts; classifiers trained on simple, diverse, and subsequence data perform best but leave room for improvement.
  • StereoSet evaluation shows A-INLP improves GPT-2's stereotype score while maintaining its language-modeling score.
  • 19%: A-INLP improves overall ICAT performance on StereoSet tasks testing religion-associated bias while retaining language-modeling performance.
  • Human evaluation finds A-INLP retains clarity and content near 5 while improving fairness from 3.93 to 4.00.

2. Our approach depends on carefully crafted bias defini-

The approach has limitations involving bias generalization, computational preprocessing, and imperfect fairness-performance tradeoffs. These constraints require careful interpretation and evaluation across diverse real-world settings.

  • Bias definitions, subspaces, and classifiers may reflect one perception of bias that does not generalize across cultures, regions, or time periods.
  • A-INLP adds time and space complexity during preprocessing, although that cost can be amortized across inference runs.
  • During inference, A-INLP is as fast as GPT-2, supporting the feasibility of deploying the debiasing method.

6. Conclusion

The paper frames representational bias in pretrained language models as a measurement and mitigation problem distinct from earlier embedding and template-based evaluations. It concludes that A-INLP and new metrics improve fairness while retaining contextual text-generation quality.

  • The paper proposes evaluation metrics for measuring sources of representational bias in large pretrained language models.
  • A-INLP automatically detects bias-sensitive tokens before applying debiasing approaches to mitigate them.
  • Unlike SEAT, the paper's representational-bias evaluation extends to language generation rather than only contextual embeddings.
  • StereoSet provides tests for intrasentence and intersentence stereotypical associations, but its contexts lack sufficient diversity for clearly measuring fine-grained context-bias associations.
  • Earlier template-based evaluations use small sets of bias terms and simple placeholders, limiting tests of diverse templates and disentanglement of context from bias associations.

B. Benchmarks for Measuring Bias

The benchmark construction expands bias evaluation beyond simple templates by collecting diverse real-world contexts, while the implementation uses cached Transformer context representations for efficient generation. The paper also tunes debiasing hyperparameters using performance-fairness tradeoffs.

  • The benchmark collects 16,338 diverse contexts from five real-world text corpora to evaluate bias and context associations beyond simple templates.
  • The datasets vary in topics, formality, and spoken or written text, providing diversity for context-based bias measurement.
  • Cached Transformer context embeddings generate each next token without repeated forward passes over prior conditioning tokens.
  • The implementation samples the next token from Softmax(Wo_t) after applying a vocabulary-sized linear transformation to the logit vector.
  • Hyperparameters are selected using validation performance and the largest area under the performance-fairness curve during debiasing.

C.5. Training Resources and Time

The paper constructs bias-sensitive token and context resources, trains classifiers for nullspace projection, and evaluates local and global fairness alongside language-model performance. Its preprocessing takes 740 seconds and 1470 MiB, while generation takes 1.21 seconds and 1231 MiB per sentence.

  • Bias-sensitive token identification: Bias subspaces are estimated from predefined gender and religion word pairs or triplets using PCA, then token projections identify bias-sensitive vocabulary items.The projection value measures a token embedding’s extent of bias and supports token selection.
  • Bias-sensitive token identification: The method compares GloVe and GPT-2 embedding subspaces, finding GloVe substantially more accurate, especially for religion, while GPT-2 surfaces several stop words as highly projected tokens.The stop words are removed from the bias-sensitive token set.
  • Training resources: Training data combines simple templates with diverse corpus sentences and partial subsequences to obtain contexts for bias-classifier training.The resulting dataset contains 2,940 training, 1,260 validation, and 1,800 test samples.
  • Training resources: The bias classifier is a linear SVM with ℓ2 penalty and squared hinge loss, trained iteratively until its accuracy reaches approximately 33% for both gender and religion.The experiments use 80 classifiers for gender and 50 for religion.
  • Evaluation metrics: Fairness evaluation uses counterfactual context edits and compares next-token distributions with KL divergence and Hellinger distance locally, plus sentiment and regard differences globally.Lower scores indicate better fairness for these divergence- and difference-based measures.
  • Evaluation metrics: Performance metrics assess whether the model preserves probability for the correct next token and maintains the original distribution over all next tokens after debiasing.The former uses higher-is-better measures, while distributional difference uses a lower-is-better measure.

D.4. Ablation Studies

Ablations show that stronger nullspace projection can improve fairness while reducing language-model performance, whereas the first sensitive-token correction may reduce bias throughout the remaining generation. Across hyperparameter settings, A-INLP can keep LM scores near the original while improving fairness, with StereoSet gains for gender and religion.

  • Nullspace projection: Better-trained bias classifiers remove more bias through nullspace projection but also discard context information, producing better fairness at the expense of lower LM performance.This tradeoff becomes stronger as the number of nullspace-projection iterations increases.
  • Autoregressive effects: Fixing the first occurrence of a sensitive token often leads to significantly less bias in the remaining generated sentence without further debiasing.The paper illustrates this propagation effect with generation examples.
  • Autoregressive effects: Global fairness plots are less smooth than local plots because autoregressive token-level debiasing introduces stochasticity during generation.The reported explanation concerns the generation process rather than a smooth deterministic tradeoff.
  • Context diversity: Only classifiers trained on simple, diverse, and subsequence contexts perform well enough on diverse contexts, which remain significantly harder to detect and mitigate.The authors state that substantial room for improvement remains.
  • StereoSet comparison: A-INLP improves gender StereoSet stereotype scores while keeping LM scores within 1.5% of the original and raises overall ICAT by 3%.The reported result also preserves coherent text despite a slight performance decrease at higher fairness.
  • StereoSet comparison: For religion contexts, A-INLP improves the overall ICAT metric by 7%, with better fairness accompanied by a slight decrease in language-model performance.The paper reports this as a performance-fairness tradeoff across the evaluated settings.
  • Hyperparameter tradeoff: Hyperparameter sweeps over α and bias-classifier training epochs identify multiple settings that retain LM scores near GPT-2’s 88.5 while reducing stereotype scores from 58.0 to approximately 50.Figure 5 measures fairness as 100-SS and performance as LM score.

E. Limitations and Attempts that Failed

The approach is constrained by imperfect bias-classifier data, unstable GPT-2 logits, embedding-subspace inaccuracies, and subword coverage gaps. These issues can remove contextual information, distort token probabilities, or impair bias-sensitive-word discovery.

  • Data and projection: Imperfect classifier datasets may encode information beyond the target bias attribute, so nullspace projection can remove contextual information and reduce LM performance.The limitation applies especially to context embeddings across different bias classes.
  • Data and projection: With α = 1, small GPT-2 context-embedding changes can cause large softmax-logit variance, making rare or irrelevant tokens suddenly probable while meaningful-token probabilities fall.The authors introduce learnable α to mitigate this instability.
  • Alternative debiasing: A-SUBSPACE changes logits only slightly and preserves LM performance, but it is less effective at improving fairness than A-INLP.The paper characterizes it as a more conservative debiasing algorithm.
  • Embedding limitations: An inaccurate GPT-2 embedding subspace can misestimate q(w), causing stop words to receive large α values and motivating use of a GloVe-estimated subspace.The stop-word issue is presented as an example of difficulty learning the debiasing parameter.
  • Embedding limitations: GPT-2 subwords absent from pretrained GloVe vocabulary reduce the accuracy of bias-sensitive-word discovery and the debiasing algorithm.This creates a coverage limitation when GloVe supplies the estimating subspace.
Loading 2106.13219v1…