PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro

arXiv:2202.04173v3cs.CLcs.AIcs.CYcs.LG

TL;DR

Large language models can generate toxic language, creating a need for detoxification that preserves model quality. The paper studies domain-adaptive training across data, scale, and parameter efficiency, finding that self-generated data improves data efficiency and adapter training offers a better toxicity–perplexity trade-off for large models.

  • Problem

    Large language models generate toxic language, while detoxification must also preserve language-model quality across scales and training settings.

    Source: §1 · pp. 1–2

  • Method

    The paper evaluates domain-adaptive detoxification across training corpus, model size, and parameter-efficient methods, including SGEAT, adapters, and prefix-tuning.

    Source: §1 · p. 2, §4 · p. 5

  • Results

    SGEAT consistently outperforms curated-corpus baselines with one-third less data, while adapters achieve a better toxicity–perplexity trade-off than whole-model adaptation for large models.

    Source: §1 · p. 2, §4.2 · p. 6, §9 · p. 10

  • Takeaways & Limitations

    Self-generated training data and adapter-only layers provide data- and parameter-efficient routes to detoxification while maintaining language-model quality.

    Source: §1 · p. 2, §9 · p. 10

    SGEAT may filter minority identity mentions because Perspective API can amplify social bias, and self-generated data may inherit pre-trained-model bias.

    Source: §8.2 · p. 10

Abstract

from arXiv · show

Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.

1 Introduction

Large language models can generate toxic and biased language, motivating detoxification methods that reduce toxicity without sacrificing model quality. This paper explores domain-adaptive training across training data, model size, and parameter efficiency.

Contributions

  • The paper studies detoxification for models from 126M to 530B parameters, a scale not previously examined in this domain.Source: §1 · p. 2
  • SGEAT uses language models to generate nontoxic training data, mitigating exposure bias and outperforming curated-corpus DAPT with one-third less data.The method filters self-generated text with Perspective API before domain-adaptive training.Source: §1 · p. 2

Motivation

  • Domain-adaptive training targets a trade-off between detoxification effectiveness and language-model quality, measured by toxicity, perplexity, and downstream accuracy.Source: §1 · p. 2

Contributions

  • Large models have similar toxicity to smaller models under the same pre-training corpus but require more effort to detoxify.Source: §1 · p. 2
  • Adapter-based training achieves a better toxicity–perplexity trade-off than whole-model adaptation for large models, while prefix-tuning is less effective.Source: §1 · p. 2

2 Related Work

Prior detoxification work modifies decoding, pre-training data, or model parameters, but these approaches differ in computational cost, accessibility, and preservation of language-model quality. This paper positions domain-adaptive training as a flexible alternative and compares it with decoding-time and reinforcement-learning methods.

Decoding-time methods

  • Decoding-time methods avoid parameter updates but may increase inference cost, require auxiliary models, or sacrifice perplexity and downstream accuracy.Source: §2 · p. 3

Domain-adaptive training

  • Domain-adaptive training fine-tunes models on curated nontoxic data, preserving fast inference while targeting toxicity and language-model quality.Source: §2 · p. 3

Reinforcement learning methods

  • On REALTOXICITYPROMPTS, SGEAT reduces toxic outputs by 27% for a 530B model and 37% for an 8B model, compared with 25% fewer toxic outputs for InstructGPT at 175B.Source: §2 · p. 3

Domain-adaptive training

  • Figure 1 depicts SGEAT as prompt construction, self-generation, Perspective API filtering, and domain-adaptive training.Source: Figure 1 · p. 4

3 Evaluation Protocols

The evaluation protocol measures detoxification alongside language-model quality across models, prompts, and downstream tasks. Toxicity is assessed automatically and by humans, while quality is assessed with perplexity and zero-shot utility.

3.2 LM Quality Evaluation

  • The protocol emphasizes reducing toxicity while preserving perplexity and downstream utility, while leaving social-bias analysis to a later section.Source: §3 · p. 4, §3.2 · p. 5

3 Evaluation Protocols

  • The study evaluates GPT-3-like models from 126M to 530B parameters, all pretrained on the same English corpus constructed from 15 high-quality datasets.Source: §3 · p. 4

3.1 Toxicity Evaluation

  • Automatic toxicity evaluation uses Perspective API on roughly 100k REALTOXICITYPROMPTS prompts, reporting Expected Maximum Toxicity and Toxicity Probability over 25 generations.Source: §3.1 · p. 4
  • Human evaluation complements automatic scoring because toxicity judgments are subjective and ultimately human-centric.Source: §3.1 · p. 4

3.2 LM Quality Evaluation

  • Language-model quality is measured with validation perplexity and zero-shot accuracy averaged across nine tasks spanning question answering, understanding, and commonsense reasoning.Source: §3.2 · pp. 4–5

4 Impact of Training Corpus

This section develops SGEAT, which uses self-generated, filtered text for domain-adaptive detoxification, and evaluates it against curated-corpus and decoding-time alternatives. SGEAT achieves stronger toxicity reduction with better data efficiency, while combined methods and model-quality trade-offs define the broader comparison.

4.1 SGEAT

  • SGEAT generates nontoxic training data through prompt construction, self-generation, toxicity filtering, and domain-adaptive training.Its augmented variant derives prompts from the most nontoxic unconditional generations, encouraging further nontoxic continuations.Source: §4.1 · pp. 5–6

4.2 Evaluation Results of Domain-Adaptive Training

  • SGEAT (augmented) consistently achieves lower toxicity than DAPT despite using a smaller training corpus, while preserving downstream utility with only a slight perplexity increase.For the 1.3B model, it reduces toxicity by 0.14; DAPT uses three times more data, yet achieves higher toxicity.Source: §4 · p. 5, §4.2 · p. 6, Table 3 · p. 7
  • Jigsaw’s human-annotated nontoxic subset fails to detoxify the model and can increase toxicity, likely because its average toxicity is higher than SGEAT’s data.This contrasts with the curated DAPT corpus and the self-generated corpus evaluated in the same setup.Source: §4.2 · p. 6
  • The augmented prompt design is the strongest SGEAT variant because it retains generation diversity while producing prompts likely to yield nontoxic continuations.Heuristic prompting is less effective, while unconditional generation remains diverse and performs well.Source: §4.1 · p. 5, §4.2 · p. 6

4.3 Evaluation Results of Decoding-time Methods

  • Rejection sampling adds 4× decoding overhead yet is less effective than SGEAT, whereas DEXPERTS lowers toxicity more but increases PPL by 9.47 and reduces downstream accuracy by 9.4%.Combining decoding-time methods with SGEAT produces the lowest toxicity among the compared methods.Source: §4.3 · pp. 6–7

5 Impact of Model Size

Across model sizes, toxicity is similar when pre-training data is shared, but larger models require more data or training effort to detoxify without harming quality.

Standard Model Toxicity

  • Models trained on the same pre-training corpus show similar toxicity across parameter sizes, suggesting toxicity comes from the dataset rather than model size.Source: §5 · p. 7

Detoxification Effectiveness

  • SGEAT consistently outperforms DAPT across model sizes, reducing 126M toxicity probability from 57% to 30% with a 1/3 smaller corpus.The results identify self-generated data as more efficient than curated pre-training data for detoxification.Source: §5 · p. 7

Larger-scale LMs

  • For the 530B model, increasing data from 50k to 100k samples improves the detoxification-quality trade-off more than increasing training epochs from 3 to 5.More epochs can cause overfitting and substantially hurt perplexity and downstream accuracy.Source: §5 · p. 7

LM Quality Evaluation

  • After 3 epochs, perplexity rises slightly and utility drops modestly in most cases, without significant overfitting or quality degradation.Source: §5 · p. 8

Domain Adaptation v.s. Overfitting

  • For 530B, toxicity initially drops steeply with little perplexity loss, but later flattening—especially for DAPT—signals overfitting.SGEAT reaches substantially lower toxicity than DAPT at the same perplexity after one epoch.Source: §5 · p. 8, Figure 2 · p. 8

Training Schedule

  • Three epochs is a good whole-model adaptation cutoff across model sizes and methods, balancing toxicity reduction against perplexity.Source: §5 · p. 8

6 Parameter-efficient Training

The paper compares adapter and prefix tuning for parameter-efficient detoxification, finding adapters offer the better trade-off and preserve quality especially for large models.

6 Parameter-efficient Training

  • Both methods train only added parameters, allowing domain adaptation without perturbing the original LM parameters or maintaining multiple full model copies.Source: §6.1 · pp. 8–9

6.1 Comparison between Adapter and Prefix Tuning

  • Adapters add bottleneck projection layers within transformer layers, whereas prefix tuning appends continuous prefix vectors to steer generation.Source: §6.1 · p. 9
  • Adapters provide a better detoxification-quality trade-off than prefix tuning, while longer prefixes can hurt perplexity and decoding speed.The proposed explanation is that adapters provide more additional capacity, whereas long prefixes add irrelevant context.Source: §6.1 · p. 9
  • On the 1.3B model, adapters show no significant detoxification or quality advantage over whole-model adaptation, unlike their benefits at larger scales.Source: §6.1 · p. 9

6.2 Apply Adapter to larger-scale Models

  • Adapter training improves parameter efficiency and mitigates perplexity and quality loss versus whole-model adaptation for larger LMs.For 530B, adapters reduce the perplexity drop by at most 0.64 and improve average downstream accuracy by 0.7%.Source: §6.2 · p. 9

7 Human Evaluation

Human evaluation across five model sizes broadly agrees with Perspective API toxicity scores and confirms that larger models are more difficult to detoxify.

Human Evaluation

  • Human toxicity scores mostly align with Perspective API scores, with a Pearson correlation coefficient of 0.9661.Source: Figure 3 · p. 9, §7 · p. 10
  • The evaluation averages scores from five annotators across 300 prompts for each method and model size.Source: §7 · p. 9
  • Human evaluation confirms that models of different sizes have similar toxicity levels.Source: §7 · p. 10
  • Human evaluation also confirms that larger models present greater detoxification challenges.Source: §7 · p. 10

8 Discussion

Effective detoxification can alter perplexity unevenly across demographic groups, while SGEAT remains limited by bias in both its detector and pre-trained model.

8.1 Bias against Marginalized Groups

  • SGEAT increases perplexity moderately on BOLD after detoxification, without noticeable gender-group discrepancy but with a larger increase for the Hispanic group.The authors attribute this mainly to bias already present in the pre-trained model and corpus.Source: §8.1 · p. 10

8.2 Limitation of SGEAT

  • Although SGEAT achieves a strong toxicity–perplexity trade-off, the authors note that its effectiveness can still be improved.Source: §8.2 · p. 10
  • Perspective API may amplify social bias by filtering toxic text together with minority identity mentions, limiting SGEAT’s fairness.The authors identify more robust, unbiased, and fair hate-speech detection as a way to improve the filtered corpus.Source: §8.2 · p. 10
  • Bias in the pre-trained model can be inherited by self-generated data, harming detoxification coverage for certain demographic groups.The paper leaves building a bias-free pre-trained LM as future work benefiting SGEAT and other detoxification methods.Source: §8.2 · p. 10

9 Conclusion

The paper studies detoxification across corpus design, model scale, and parameter efficiency, concluding that self-generated data and adapters can reduce toxicity while preserving language-model quality.

9 Conclusion

  • The study spans training corpus, model size, and parameter-efficient training, covering models from 126M to 530B parameters.It reports that adapters achieve a better toxicity–language-quality trade-off than whole-model adaptation.Source: §9 · p. 10
  • Adapter-based training provides a parameter-efficient route to detoxification with a better trade-off between toxicity and language-model quality.Source: §9 · p. 10

A.1 Details of Pre-trained LMs

  • The evaluated model suite includes pre-trained language models with architecture details summarized in Table 7.Source: §A.1 · p. 15, Table 7 · p. 15

A.2 Training Details

  • Experiments fine-tune models for 3 epochs with batch size 512 and sequence length 2048, using Adam and model-size-dependent learning rates.Source: §A.2 · p. 15

A.3 Details of Evaluation Protocols

The appendix specifies human and automatic evaluation protocols, then analyzes prompt design, toxicity, and diversity for self-generated detoxification data.

A.3 Details of Evaluation Protocols

  • Human annotators from four English-speaking regions evaluated toxicity through Amazon MTurk under strict prior-approval and HIT-quality requirements.Each batch contained 10 continuations and paid $0.7.Source: §A.3 · p. 15
  • Human toxicity scores were mostly aligned with Perspective API scores, supporting the automated evaluation as an approximate measure.Source: §A.3 · p. 15, Table 8 · p. 17

B.1 Unconditional Generation

  • SGEAT’s unconditional generation uses only the start-of-sentence token and collects 100,000 samples across random seeds.Source: §B.1 · p. 17

B.2 Conditional Generation

  • Automatically constructed prompts select the least-toxic 25% of unconditional documents, split them into prompts and continuations, and generate four continuations per prompt.Source: §B.2 · p. 18

B.3 Unprompted Toxicity

  • Automatically constructed prompts reduce generated-data toxicity relative to unconditional generation.Source: §B.3 · p. 18, §B.4 · p. 18

B.5 Data Diversity Evaluation

  • SGEAT (standard) produces diversity closer to OWTC than heuristic prompting, whose lower diversity limits detoxification effectiveness.Source: §B.5 · pp. 18, 20, Table 10 · p. 19

B.6 Benchmark and Analysis of Prompt Design

The appendix shows that prompt diversity and toxicity jointly shape SGEAT, while validation choices, model transfer, and corpus mixing affect the toxicity–quality trade-off.

B.6 Benchmark and Analysis of Prompt Design

  • Using all automatically constructed prompts gives the best toxicity reduction, with average generated-data toxicity of 0.07.Source: §B.6 · pp. 20–21, Table 13 · p. 21
  • A dataset with average toxicity as low as 4e-4 is less effective than the diverse automatic-prompt set, showing that low data toxicity alone is insufficient.The analysis attributes effectiveness to both lower toxicity and broader coverage of the generation distribution.Source: §B.6 · p. 21
  • Manual prompts perform worse than unconditional generation; negative prompts may increase model toxicity, whereas repeated positive prompts can reduce continuation toxicity.Source: §B.6 · p. 21
  • The nontoxic Jigsaw subset fails to detoxify and can increase toxicity, plausibly because of higher residual toxicity and domain shift from pre-training data.Source: §B.6 · p. 21

C.2 Perplexity Evaluation on Nontoxic Validation Set

  • On nontoxic validation subsets, perplexity increases less than on the full validation set, implicating toxic documents in part of the observed degradation.Source: §C.2 · p. 22

C.3 Perplexity Evaluation on Self-Generated Data v.s. Pre-training Data

  • SGEAT (augmented) reaches PPL 5.98 versus 7.93 for OWTC, indicating closer alignment with high-density regions of the pre-trained LM’s output space.Source: §C.3 · pp. 22–24

C.4 Transferring Self-Generated Dataset from Larger Models to Smaller Models

  • Transferring 1.3B-generated data to a 126M model is less effective for toxicity reduction than self-generated data, though it yields better validation PPL.Source: §C.4 · p. 24, Table 21 · p. 24

C.5 Mixing Nontoxic Pre-training Corpus and Self-Generated Data

D.2 Error Analysis of Perspective API

The analysis finds that Perspective API can substantially overestimate toxicity, including misclassifying nontoxic cleaning-related text, while SGEAT detoxification can still produce occasional failures.

D.2 Error Analysis of Perspective API

  • 34% of samples classified as toxic by Perspective API were judged nontoxic by a majority of five human annotators.This quantifies disagreement between the automatic detector and human evaluation.Source: §D.2 · p. 25

“Let us help you with your cleaning!

  • Perspective API assigned a toxicity score of 0.9930 to a cleaning-services paragraph that was identified as a surprisingly toxic document.The example motivates more robust and unbiased hate-speech detection models.Source: “Let us help you with your cleaning! · p. 25

D.3 Case Studies of SGEAT

  • The appendix presents both successful and failure cases of SGEAT-based detoxification for a 530B model.The supplied tables include three successful examples and one prompt whose 25 generations still contained toxic language.Source: §D.3 · p. 25, Table 23 · p. 26, Table 24 · p. 27

Related papers

Loading 2202.04173v3…