Source-linked AI summary
Detoxifying Language Models Risks Marginalizing Minority Voices
Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, Dan Klein
TL;DR
The paper asks whether detoxification can improve LM safety without reducing equitable utility on language used by marginalized groups. It evaluates multiple detoxification methods with automatic and human assessments, finding disproportionately worse performance on AAE and minority identity language and attributing the failures to biased toxicity data. The authors discuss representational harms and possible data- and model-level remedies.
Problem
Detoxification seeks to reduce toxic generations without harming LM utility on nontoxic inputs, but its effects on marginalized-group language require evaluation.
Method
The paper evaluates controllable detoxification techniques using automatic perplexity tests and human judgments of generation quality across WAE, AAE, and minority identity language.
Results
Detoxification disproportionately worsens perplexity and generation quality on AAE and minority identity language, with stronger detoxification amplifying the bias.
Takeaways & Limitations
Biased toxicity data can make controllable detoxification steer away from AAE and minority identity mentions, creating representational harms for minority groups.
Takeaways & Limitations
The identified harms arise largely from spurious correlations in toxicity datasets, while improving annotation quality can make data collection more expensive and current debiasing methods remain imperfect.
Abstract
from arXiv · showhide
Language models (LMs) must be both safe and equitable to be responsibly deployed in practice. With safety in mind, numerous detoxification techniques (e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to mitigate toxic LM generations. In this work, we show that current detoxification techniques hurt equity: they decrease the utility of LMs on language used by marginalized groups (e.g., African-American English and minority identity mentions). In particular, we perform automatic and human evaluations of text generation quality when LMs are conditioned on inputs with different dialects and group identifiers. We find that detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups. We identify that these failures stem from detoxification methods exploiting spurious correlations in toxicity datasets. Overall, our results highlight the tension between the controllability and distributional robustness of LMs.
1 Introduction
Detoxification reduces toxic generations but can degrade LM utility disproportionately on language associated with marginalized groups. The paper links these harms to biased toxicity data and discusses resulting social risks and possible remedies.
- Detoxification methods mitigate toxic generations, but they also increase perplexity on nontoxic inputs, creating a utility trade-off.The intended goal is to reduce toxicity without affecting nontoxic-input quality.
- Detoxification disproportionately increases perplexity on AAE and minority identity mention text compared with WAE, reducing utility for marginalized-group language.The paper evaluates language containing African-American English markers and mentions such as “gay” or “Muslim.”
- Human evaluations likewise find that detoxified models produce lower-quality generations from AAE prompts, especially when preserving topicality.On WAE prompts, models roughly maintain topic, fluency, and style; these qualities deteriorate on AAE prompts.
- Spurious correlations between toxicity labels and AAE or minority identity mentions cause detoxification systems to steer away from those language patterns.The paper attributes the failures to biased toxic-classification data that treats these features as signals of toxicity.
- The authors argue that these biases can force marginalized users to code-switch or hide identity, contribute to stigma, and motivate better annotation and training procedures.The proposed remedies include improved data annotation and model-training procedures for debiasing detoxification.
2 Methods and Experimental Setup
The study evaluates four controllable detoxification techniques using GPT-2 and toxicity-labeled data. It compares their effects on toxicity reduction and language-model behavior, including sensitivity to detoxification strength.
- The study evaluates DAPT, PPLM, GeDi, and output filtering as state-of-the-art techniques for steering generations away from toxicity.DAPT finetunes on nontoxic data; PPLM uses classifier gradients; GeDi combines LM probabilities; filtering selects the least toxic of 10 generations.
- Stronger GeDi detoxification sharply increases the AAE-to-WAE perplexity ratio, rising from approximately three times at ω = 0 to almost 400 times under strong detoxification.The ratio measures relative perplexity on AAE versus WAE text as detoxification strength changes.
- GPT-2 medium serves as the base LM, with original-method hyperparameters and top-k sampling fixed at k = 50 for fair comparison.The toxicity training data comes from the Jigsaw Civil Comments dataset after removing examples with low inter-annotator agreement.
3 Detoxifying LMs Introduces Biases
Detoxification reduces toxic generations but introduces disproportionate utility and quality costs for AAE and minority identity language, with stronger detoxification amplifying these biases.
- 3.1 Automatic Evaluation Using Perplexity: Perplexity also rises on toxic WAE data and, more modestly, on nontoxic WAE data, indicating toxicity reduction trades off against general utility.The authors note that some quality degradation may reflect domain shift between comment-based training data and Twitter evaluation data.
- 3.1 Automatic Evaluation Using Perplexity: Detoxification increases perplexity disproportionately on AAE and minority identity mentions, with PPLM raising perplexity 2.1× on WAE versus 4.3× on identity-mention data.These evaluations use nontoxic tweets and compare detoxified models with baseline behavior.
- 3.1 Automatic Evaluation Using Perplexity: Increasing detoxification strength widens the AAE-to-WAE perplexity gap across techniques, reaching ratios above 400 with stronger GeDi detoxification.The GeDi parameter ω controls detoxification strength in this analysis.
- 3.2 Human Evaluation of Generation Quality: All detoxification methods reduce toxicity relative to baseline GPT-2, but typically degrade generation quality, including topicality, fluency, and style.More than 80% of annotators rated GeDi less topical, while every technique except DAPT was rated less fluent.
- 3.2 Human Evaluation of Generation Quality: Generation quality is consistently lower on AAE prompts across metrics, with detoxified models especially likely to change the topic.GeDi was preferred only half as often for topicality on AAE prompts as on WAE prompts.
4 Why Detoxification Introduces Biases
Detoxification inherits biases from toxicity datasets whose labels spuriously correlate with AAE and minority identity mentions, steering models away from that language.
- 4 Why Detoxification Introduces Biases: All evaluated detoxification techniques rely on labeled toxic or nontoxic data, either directly through finetuning or indirectly through classifiers and auxiliary language models.DAPT uses the data directly, while PPLM, GeDi, and Filtering incorporate models trained on toxicity data during decoding.
- 4 Why Detoxification Introduces Biases: Annotation and sampling biases create spurious correlations between toxicity labels and AAE or minority identity mentions.Crowdworkers may misjudge unfamiliar AAE as toxic, while toxic comments are often directed toward marginalized groups.
- 4 Why Detoxification Introduces Biases: These correlations cause DAPT to forget AAE and identity mentions, while PPLM, GeDi, and Filtering steer generations away from them as if they were toxic.The resulting behavior explains the observed utility degradation on marginalized-group language.
- 4 Why Detoxification Introduces Biases: Increasing detoxification strength exacerbates these problems, including through longer DAPT finetuning or higher GeDi ω.The bias is therefore linked to detoxification with biased supervised data rather than to one specific technique.
5 Harms of Detoxification
Detoxified language models create representational harms for minority groups, affecting both their language understanding and the identities and topics they discuss. These failures may extend beyond detoxification to other controllable-generation methods.
- Detoxified language models struggle to understand and generate minority dialects and identity mentions, creating representational harms for marginalized groups.AAE speakers may need to code-switch to WAE for NLP systems to work effectively, which can discourage engagement.
- Avoiding minority identity terms and topics can silence users’ identities and reinforce existing societal stigmas.The paper links WAE-only alignment with stigmatizing AAE as incorrect or “bad” English.
- Similar biases may arise whenever controllable generation is applied to other attributes, such as sentiment, because biased datasets can shape which racial groups are discussed.
6 Future Work: Towards Bias-Free Detoxification
Future work targets the spurious correlations underlying detoxification harms through better datasets and debiasing methods. However, annotation improvements may cost more, and current debiasing methods remain imperfect.
- Improving annotation procedures or labeling schemes could reduce spurious correlations in toxicity datasets, but collecting higher-quality annotations may be more expensive.
- Training models to handle biased data offers an alternative or complement to improving datasets, although state-of-the-art debiasing methods remain imperfect.
A Minority Identity Mention Word List
The study identifies minority identity mentions using a specified word list spanning sexual orientation, gender, racial, ethnic, religious, and related identities.
- The minority-identity detector uses terms covering lesbian, gay, bisexual, transgender, queer, racial, ethnic, non-binary, Latinx, Jewish, Arab, and Muslim identities.
B Amazon Mechanical Turk Details
The Mechanical Turk appendix documents the crowdworker materials and test interface used in the human evaluation.
- Figures 4 and 5 present the instructions and examples provided to Amazon Mechanical Turk crowdworkers.
- Figure 6 presents an example of the crowdworker test interface.