Source-linked AI summary
Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP
Timo Schick, Sahana Udupa, Hinrich Schütze
TL;DR
Large language models trained on unfiltered internet data can reproduce harmful biases, motivating methods that address them without relying solely on data curation. This paper tests whether models can recognize undesirable attributes in their own outputs and proposes self-debiasing, which lowers the probability of biased text using textual behavior descriptions. The results show substantial but insufficient self-diagnosis and self-debiasing capability.
Problem
Language models trained on large internet crawls can reproduce or amplify harmful biases present in those data.
Method
The paper evaluates self-diagnosis and proposes a decoding algorithm that compares ordinary and bias-encouraged token probabilities to reduce biased generation.
Results
GPT2-XL reaches 72.7% average self-diagnosis accuracy and T5 reaches 87.3% across the evaluated attributes, while template changes can substantially affect performance.
Takeaways & Limitations
Self-debiasing can reduce the probability of biased text without training data, parameter changes, or manually curated word lists.
Takeaways & Limitations
The evaluation relies heavily on imperfect Perspective API scores, while templates and attribute descriptions can change performance and filtering can be overly aggressive.
Abstract
from arXiv · showhide
When trained on large, unfiltered crawls from the internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: they often generate racist, sexist, violent or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model's parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.
1 Introduction
Internet-scale pretraining exposes language models to harmful biases that they may reproduce, while word bans and dataset curation have important limitations. The paper therefore studies self-diagnosis and proposes self-debiasing as a test-time decoding approach, although current capabilities do not eliminate corpus-based bias.
- Internet crawls contain biased or harmful text, which pretrained language models can reproduce or amplify.
- Banned-word lists can miss biased text expressed with innocuous words and can remove knowledge needed for meaningful generation.
- Dataset curation is valuable but resource intensive for the large datasets and multiple models required across common global languages.
- Self-diagnosis asks whether models can detect undesirable attributes in their own outputs using internal knowledge and textual descriptions.
- Self-debiasing uses a decoding algorithm to reduce biased generations from a textual description of undesired behavior without training-data or parameter changes.
- Current large models show some self-diagnosis and self-debiasing ability, but it is insufficient to eliminate corpus-based bias.
2 Related Work
Prior work documents bias in pretrained representations and proposes debiasing, toxicity mitigation, and controllable-generation methods. The paper’s approach instead uses task descriptions and internal model knowledge to control behavior without task-specific training data or external resources.
- Pretrained static and contextualized embeddings exhibit unfair, discriminative, and other undesirable biases.
- Static-embedding debiasing methods often use predefined word lists or external resources, although dictionary definitions can replace such lists.
- Methods for contextualized embeddings and text generation include bias mitigation, non-toxic pretraining, and plug-and-play language models.
- Self-diagnosis builds on zero-shot learning with task descriptions, while self-debiasing resembles prefix-constrained decoding and prompt-based controllable generation.
- Compared with related controllable-generation approaches, the proposed method does not require customized pretraining objectives or labeled training data.
3 Self-Diagnosis
Self-diagnosis tests whether pretrained language models can detect undesirable attributes in their own outputs from short textual descriptions, then examines robustness across models, attributes, and templates. Larger models perform substantially better, but template choices still affect smaller models and some language-understanding settings.
- 3.1 Experimental Setup: Self-diagnosis asks a pretrained model to detect socially undesirable attributes in its output using only a short textual attribute description.The method relies on internal knowledge acquired during pretraining, without training data or external resources.
- 3.1 Experimental Setup: The model estimates whether generated text exhibits an attribute from its probabilities for “Yes” and “No” in a prompted question-answering template.The generated text and attribute description form the self-diagnosis input.
- 3.2 Results: GPT2-XL reaches 72.7% average accuracy and ρ = 0.51 PCC, while the largest T5 model reaches 87.3% average accuracy and ρ = 0.74 PCC across six attributes.The smallest model performs no better than chance for any of the six attributes, and self-diagnosis ability strongly correlates with model size.
- 3.3 Template Sensitivity: Output alternatives “yes/no” and “true/false” perform similarly, with the default “Yes/No” choice slightly better for bigger models.This comparison evaluates the output vocabulary used for the self-diagnosis answer.
- 3.3 Template Sensitivity: Removing quotes causes a slight performance drop, whereas removing “Question:” and “Answer:” severely hurts performance for almost all models.The latter change does not alter the overall trend that bigger models show better self-diagnosis.
- 3.3 Template Sensitivity: Template and description sensitivity is lower for larger models, but language-understanding cases such as “toxic” can still affect large models.Alternative descriptions improve smaller models; GPT2-XL performs similarly across definitions, while T5 models favor the default and ORIGINAL descriptions.
4 Self-Debiasing
Self-debiasing modifies a pretrained language model’s decoding process using textual attribute descriptions to reduce undesirable generations without changing model parameters. Across toxicity and social-bias evaluations, it reduces bias, though effectiveness depends on the model, task, and decoding strength, with some quality trade-offs and remaining failures.
- Method: Self-debiasing uses internal model knowledge and textual attribute descriptions to penalize continuations associated with undesirable attributes during generation.The method computes a difference between ordinary and self-debiasing next-token probabilities, then applies a scaling function to alter the output distribution.
- Method: A soft decay controlled by λ reduces biased-token probabilities without forcing them to zero, preserving the ability to compute perplexity.The hard indicator-based variant can assign zero probability to the correct next token, producing infinite perplexity; the proposed soft variant instead scales probabilities according to the difference magnitude.
- RealToxicityPrompts: About 25% lower attribute-generation probability was achieved with λ = 10 for GPT2-XL across six attributes without negatively affecting perplexity.Higher λ values further improved self-debiasing but slightly increased perplexity; λ = 100 reduced behavior by more than half across all attributes.
- RealToxicityPrompts: Keyword-based self-debiasing performed worse than the original attribute descriptions and produced higher Wikitext-2 perplexity.The keyword condition used λ = 100, whereas the original descriptions used λ = 50.
- RealToxicityPrompts: Human evaluation confirmed that λ = 100 considerably reduced undesirable attributes without harming fluency or coherence, despite increasing perplexity on Wikitext-2.Human annotators found the self-debiased continuations slightly more fluent and coherent than regular outputs.
- Limitations: Self-debiasing is not uniformly effective: it can fail on some examples, and its CrowS-Pairs template may be poorly matched when both sentences in a pair are biased.RoBERTa’s self-debiased variants also increased pseudo-perplexity from 8.6 to an average of 9.7 ± 0.1, while λ = 10 yielded 8.8 ± 0.0.
5 Discussion
The discussion explains why self-debiasing is preferable to direct prompting or banned-word lists, while emphasizing limitations in evaluation, generality, and reliability. It also identifies user-configurable behavior as a practical advantage, but concludes that the method should not be used alone.
- Approach: Self-debiasing can simultaneously address multiple undesired attributes without requiring negation, which current language models often misunderstand.The paper used this indirect setup for multiple attributes in its RealToxicityPrompts experiments.
- Approach: Directly prompting models not to express a bias can make them generate text related to that bias, influencing every word's probability.Self-debiasing instead modifies probabilities only for words explicitly considered biased.
- Limitations: Perspective API-based evaluation cannot thoroughly test biases it does not measure, and its labels may be error-prone because it can miss subtle bias and overrely on lexical cues.Human evaluation partly mitigates this issue, but crowdworkers bring their own biases and perceptions.
- Limitations: The evaluation is limited to English, two datasets, and a small subset of language models, restricting conclusions about other languages, models, and behaviors.The authors call for broader datasets and more reliable measures of bias.
- Limitations: Template and attribute-description changes can substantially alter self-diagnosis performance, while self-debiasing may aggressively filter harmless words.Large λ values increase Wikitext-2 perplexity, although human evaluation found no loss of fluency or coherence.
- Implications: Current self-debiasing only reduces, rather than reliably prevents or eliminates, bias and toxic behavior, so it should not be the sole real-world mitigation.The method provides users an interface for explicitly setting desired behavior in concrete use cases.
6 Conclusion
The conclusion reports that language models can diagnose undesirable attributes in their own outputs and that this capability supports a decoding method for reducing biased text. However, the evidence remains limited, and self-debiasing reduces rather than eliminates corpus-based bias.
- Findings: Language models can perform self-diagnosis by examining their outputs for undesirable attributes using internal knowledge and textual descriptions.The conclusion presents this as the paper's central empirical finding.
- Method: The proposed decoding algorithm reduces biased-token probability by comparing its original probability with its probability under explicitly encouraged undesired behavior.This comparison supplies the signal used for self-debiasing.
- Limitations: The evaluation covers only two English datasets and a small portion of potentially undesirable behaviors, motivating tests across more behaviors, languages, benchmarks, and models.The conclusion explicitly describes these as important extensions.
- Implications: Self-diagnosis and self-debiasing reduce but do not eliminate corpus-based bias and are not viable as an isolated route to bias-free models.The authors suggest combining them with complementary models or stronger debiasing solutions.