Source-linked AI summary
Challenges in Detoxifying Language Models
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, Po-Sen Huang
TL;DR
The paper asks how LM toxicity should be evaluated and mitigated without overlooking quality and social-bias consequences. It combines automatic toxicity scores with human annotation across several mitigation strategies, finding lower automatic toxicity alongside reduced marginalized-group coverage, increased biases, and weaker agreement with human judgments.
Problem
Safe LM deployment requires reliable toxicity measurement and mitigation, but automatic toxicity evaluation may not capture the full consequences for LM quality and social bias.
Method
The paper critically evaluates several toxicity-reduction methods using both classifier-based toxicity scores and human judgments, while examining LM quality and social-bias trade-offs.
Results
Detoxification lowers automatic toxicity scores but increases LM loss, amplifies topic- and dialect-related biases, reduces marginalized-group coverage, and increases disagreement with human annotators.
Takeaways & Limitations
Toxicity evaluation should better align with human perceptions, distinguish toxicity subtypes, and account for classifier bias and marginalized-group coverage.
Takeaways & Limitations
Likelihood-based metrics can show high bias when loss gaps are large, but small loss gaps do not imply low bias.
Abstract
from arXiv · showhide
Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of LM toxicity. We critically discuss this approach, evaluate several toxicity mitigation strategies with respect to both automatic and human evaluation, and analyze consequences of toxicity mitigation in terms of model bias and LM quality. We demonstrate that while basic intervention strategies can effectively optimize previously established automatic metrics on the RealToxicityPrompts dataset, this comes at the cost of reduced LM coverage for both texts about, and dialects of, marginalized groups. Additionally, we find that human raters often disagree with high automatic toxicity scores after strong toxicity reduction interventions -- highlighting further the nuances involved in careful evaluation of LM toxicity.
1 Introduction
The paper examines how toxicity in language-model generation should be evaluated and mitigated, using both automatic and human assessment. It finds that simple interventions can improve automatic toxicity metrics but may reduce coverage of marginalized groups and create evaluation discrepancies.
- Language models can generate harmful language, including hate speech, insults, profanities, and threats, commonly grouped as toxicity.
- Safe LM deployment requires measuring toxicity, understanding its origins, and taking effective steps to mitigate toxic generation.
- The paper evaluates toxicity mitigation using both automatic toxicity scores and human judgments, while examining trade-offs with LM quality and social bias.
- Combinations of simple methods effectively optimize automatic toxicity metrics but are prone to overfiltering texts related to marginalized groups.
- Strong toxicity reduction interventions increase disagreement between high automatic toxicity scores and human annotators, limiting those scores’ usefulness for further mitigation.
- Reducing automatic toxicity scores trades off against LM evaluation loss and disproportionately affects texts about and by marginalized groups, increasing topic- and dialect-related biases.
2 Related Work
Prior research studied toxic language detection, generation, and mitigation, while this paper extends that work by comparing classifier-based toxicity scoring with human evaluation and examining social-bias effects.
- Toxic-language research has largely focused on online-community moderation, whereas toxic text generated by language models is a more recent research direction.
- Earlier studies showed that prompts can induce racist GPT-2 continuations and examined mitigation approaches for toxicity in non-synthetic prompts.
- Generation steering typically uses an external toxicity classifier, though a language model’s own toxicity self-diagnosis can also guide reduction.
- Toxicity classifiers can exhibit topic-related false positives for identity terms and disparate performance across dialects, including higher scores for African-American English.
- Classifier-based mitigation may transfer these social biases to language models, and prior work reports that detoxification can amplify social biases.
- The paper broadens earlier analyses across models, demographics, and datasets, and uses Wikipedia metadata rather than keyword matching for topic-related bias measurement.
3 Toxic Language and LMs
The paper treats toxicity as one important but limited dimension of LM harm and evaluates it through both automatic measures and human judgment. It cautions against overgeneralizing results across applications and audiences.
- Under the adopted definition, an utterance is toxic when it is rude, disrespectful, or unreasonable and likely to make someone leave a discussion.
- Toxicity judgments are subjective and depend on raters, cultural background, and inferred context, so fair application across contexts remains an open issue.
- The toxicity notion covers only one aspect of possible LM harms and does not address harmful stereotypes or biases that emerge statistically across many samples.
- LM safety criteria are application- and audience-specific, limiting generalization about the absolute or relative efficacy of particular techniques.
- Automatic evaluation provides a low-cost initial indication and supports narrowly focused steering research, but human judgment ultimately defines impacts on people.
4 Model and Methods
The study evaluates a 1.4B-parameter TransformerXL language model using data filtering, test-time filtering, and PPLM-based controllable generation. The interventions rely on toxicity classifiers and are compared across automatic evaluation settings.
- Methods: The experiments cover data-based, controllable-generation, and direct output-filtering approaches for reducing LM toxicity.
- Model: The standard model is a 1.4B-parameter TransformerXL trained on C4, using 24 layers, 16 heads, dmodel = 2048, and dff = 8192.
- Training Set Filtering: Training-set filtering removes C4 documents above toxicity thresholds of 0.2, 0.1, or 0.05, leaving 85%, 57%, or 22% of the original documents, respectively.
- Decoder / Test-Time Filtering: Test-time filtering generates up to K samples and returns the first continuation below BERT toxicity score τreject = 0.01, or the lowest-scoring sample if none qualifies.
- PPLM: PPLM steers hidden representations toward low predicted toxicity while constraining divergence from the original LM prediction.
- Evaluation: Table 1 compares three mitigation approaches and their combinations using expected maximum toxicity and the probability of generating toxic text at least once over 25 generations.
5 Classifier-Based Toxicity Evaluation
The paper evaluates toxicity mitigation with automatic classifier-based metrics on RealToxicityPrompts, while examining limitations of the evaluation protocol. Mitigation methods substantially reduce automatic toxicity scores, but the metrics have important contextual and domain limitations.
- Evaluation setup: RealToxicityPrompts measures toxicity in 20-token continuations using PERSPECTIVE API scores in prompt-conditional and unconditional settings.The benchmark contains 100K English web-language prompts for the conditional setting, labelled toxic or nontoxic.
- Evaluation setup: Expected Maximum Toxicity averages the maximum score across 25 continuations, while Probability of Toxicity measures the frequency of any score above 0.5.
- Results: Decoder filtering and PPLM substantially reduce automatic toxicity metrics across generation settings, with PPLM combinations producing the largest overall reductions.Decoder filtering further improves scores reduced by training-set filtering alone.
- Results: Automatic toxicity reductions reach 6-fold and 17-fold relative to the reported prior state of the art in toxic- and nontoxic-prompt settings, respectively.In the unprompted setting, Probability of Toxicity decreases from 0.05 to 0.00.
- Limitations: The evaluation protocol can lack sufficient context, truncate words, and apply PERSPECTIVE API outside its intended forum-comment domain.Using longer samples and truncating incomplete sentences produced broadly similar automated observations.
6 Evaluation via Human Annotation
The paper compares automatic toxicity scores with human judgments of LM continuations, using trained annotators and upsampled high-score samples. Human ratings generally improve after mitigation and correlate with classifier scores, but strong interventions produce many apparent automatic false positives.
- Methodology: Human evaluation asks three annotators to judge whether each continuation adds toxicity and to rate grammaticality, comprehensibility, and prompt consistency.
- Methodology: High classifier-score continuations are upsampled so toxicity ranges can be compared efficiently; 300 samples are prepared for each setting.Samples are rated by at least three annotators, with NOT SURE annotations discarded and ratings weighted to compensate for upsampling.
- Results: Human toxicity ratings improve across the evaluated mitigation methods and show a direct, largely monotonic relation with PERSPECTIVE API scores.
- Annotation quality: Krippendorff’s alpha is 0.49 for the standard LM and 0.48 across all LM annotations, indicating fair overall agreement.About 1% of cases receive the NOT SURE label.
- False positives: Among continuations with PERSPECTIVE API scores above 0.75, human raters identify fewer toxic samples, increasingly so as interventions strengthen or combine.After strong interventions, most samples predicted as likely toxic are false positives.
7 Consequences on LM Quality
The paper examines how toxicity interventions affect language-model quality, finding that stronger training-data filtering worsens LM loss. The degradation is especially pronounced for documents classified as highly toxic.
- Effect on Language Modeling Loss: Training-set filtering moderately increases LM loss, with stronger filtering producing larger increases.
- Effect on Language Modeling Loss: Train-filter@0.05 reaches the loss level of a 417M-parameter model trained on C4 without interventions, despite the filtered model’s larger original scale.
- Effect on Language Modeling Loss: LAMBADA accuracy decreases from 50.1% to 34.9% with train-filter@0.05.
- Effect on Language Modeling Loss: For train-filter@0.05, loss increases by 0.17 (+7%) on documents scoring below 0.1 and by 0.9 (+34%) on documents scoring at least 0.5.
- Text Quality: The interventions show no strong differences from the standard LM in comprehensibility, grammaticality, or consistency with the prompt.
8 Social Bias Amplification
Detoxification can reduce automatic toxicity while worsening language-model coverage for marginalized topics and dialects. Likelihood-based loss gaps reveal amplified social biases, but small gaps do not establish that bias is absent.
- Topic-related biases: Training-time filtering introduces gender bias by increasing loss more for text about female actors than male actors.The standard LM losses are similar for female and male actors: 3.414 vs. 3.412.
- Overall effects: Detoxification increases loss across identity-related subsets, with the largest impact on marginalized groups.The comparison uses loss gaps relative to a standard LM to reduce confounding from pre-existing differences in baseline loss.
- Topic-related biases: Detoxification more strongly worsens loss for marginalized ethnic groups, amplifying bias even when their standard-model loss is lower.Hispanic-American-related text has standard-model loss 3.46 versus 3.68 for European-American text but receives the largest negative impact from detoxification.
- Dialect-related biases: Toxicity classifiers are more likely to label TWITTERAAE tweets toxic, suggesting detoxification can harm dialect coverage.The analysis identifies bias against African-American English in toxicity classification.
- Dialect-related biases: Detoxification has a larger impact on African-American English coverage than White Aligned English coverage, further amplifying existing bias.Under the standard LM, AAE loss is already higher than WAE loss: 5.53 vs. 4.77; AAE comprises 0.07% of C4 documents.
- Limitations of likelihood for bias evaluation: Small loss gaps can miss important biases, so low likelihood-based bias metrics should not be interpreted as evidence of unbiased behavior.Large loss gaps can indicate bias, but the converse implication does not hold.
9 Conclusion
The paper finds that automatic toxicity reduction can diverge from human judgments and impose quality and fairness costs. It therefore calls for toxicity evaluation that better reflects human perceptions and distinct toxicity types.
- Conclusion: Human annotators increasingly disagree with high automatic toxicity scores after strong toxicity reduction measures.This limits the usefulness of automatic scores for further toxicity mitigation.
- Future directions: The paper highlights defining human-aligned toxicity metrics, distinguishing toxicity subtypes, and using separate test sets for each subtype.It also emphasizes debiasing toxicity classifiers because their bias can transfer to language models.
- Conclusion: Reducing automatic toxicity scores trades off against language-model loss and disproportionately harms texts about and by marginalized groups.The identified effects include increased topic-related and dialect-related biases.
10 Ethical Considerations
The ethical discussion emphasizes that toxicity is subjective, deployment contexts differ, and English-only experiments limit generalization. It also addresses annotation risks and broader harms beyond toxicity.
- Subjectivity and context: Toxicity can differ across social groups and cultures, so methods may mitigate toxicity for some people without sufficing for others.The paper describes toxicity as subjective and ill-defined.
- Scope: The experiments consider only English language models, so their findings should not be generalized uncritically to non-English models.The paper notes that ethical deployment also requires equitable safety for non-English speakers.
- Broader harms: The paper addresses one dimension of LM harm while recognizing additional concerns including statistical bias, privacy, and environmental impact.These other harms are identified as relevant to LM development and deployment.
- Human annotation: Human toxicity annotation exposes participants to toxic language and therefore requires ethical review and wellbeing safeguards.The study was reviewed by DeepMind’s Human Behavioural Research Ethics Committee, and annotators could opt out or use wellbeing resources.
- Human annotation: The annotation instructions omitted race and dialect priming, so some annotators may have been unaware of safe reclamation by African-Americans.The authors distinguish this issue from their focus on generated language, whose model has no identity.
A.1 Training Set Filtering
The paper trains language models on C4 variants filtered by PERSPECTIVE toxicity thresholds and evaluates toxicity, language-model quality, and coverage-related outcomes. Stronger filtering reduces the training corpus and worsens quality metrics, while longer continuations preserve the main automatic-evaluation pattern.
- Training Set Filtering: The PERSPECTIVE thresholds are lower than the advisable 0.7–0.9 range for content moderation and exclude uncertain mid-range scores.This choice can affect bias-mitigation efforts optimized toward higher score ranges.
- Additional evaluation: Longer continuations of up to 100 tokens, truncated at sentence boundaries, produce overall observations similar to shorter automatic evaluations.The longer-continuation analysis also reports improved correlation between human annotators and automated toxicity scores.
- Human evaluation: Human evaluation instructions rate only machine continuations in prompt context and include comprehensibility, consistency, and grammaticality questions.The study also notes ambiguity for semicomprehensible text and the need to consider harms such as medical misinformation or voter disinformation.
F Automatic and Human Toxicity Evaluation: Additional Results
Additional analyses show that automatic toxicity metrics can diverge from human judgments, particularly after filtering and for identity-related content, while filtering also affects likelihood-based quality measures.
- Filtering interventions tend to reduce correlations with human annotations, whereas PPLM tends to increase them relative to the standard LM.
- Correlations between human annotations and automatic toxicity metrics vary across prompt types, toxicity bins, continuation lengths, and mitigation methods.Short toxic prompts show low correlation, while PPLM tends to correlate more highly than training-set or test-time filtering.
- 32% of 50 inspected high-scoring continuations were judged false positives, while 34% were unclear and 34% toxic.
- AAE tweets receive higher toxicity scores than WAE tweets: 0.36 versus 0.26 on average, with 27.9% versus 15.4% above 0.5.
- Threshold-based rejection sampling increases loss-per-token by at most −n−1 ln ϵ relative to the base LM, with only logarithmic dependence on ϵ.Using ϵ = 10−8 increases the bound by a factor of 2 relative to ϵ = 10−4.
H.4 Likelihood for General Filtering Methods
Likelihood-based metrics may fail to reveal important bias from filtering because filtering can impose document-level effects while producing small per-token loss gaps. The authors therefore recommend restraint when generalizing these results across filtering methods and applications.
- Small loss gaps do not necessarily imply small bias, because likelihood-based metrics can miss important biases.
- Filtering can increase loss at the document level while having only a small impact on loss-per-token.
- Perfect distillation of a test-time filtered distribution could yield small loss gaps while preserving the filtered model’s biases.
- Toxicity judgments can be ambiguous or subjective, including for political discontent, sarcasm, and differences in perceived discomfort.
- High-scoring continuations after filtering include false positives, including identity-group examples documented in the appendix.