Source-linked AI summary
Mitigating harm in language models with conditional-likelihood filtration
Helen Ngo, Cooper Raterink, João G. M. Araújo, Ivan Zhang, Carol Chen, Adrien Morisot, Nicholas Frosst
TL;DR
Open-web training corpora can contain harmful content that is difficult to review at scale. The paper uses a pretrained language model and researcher-written triggers to filter documents, and reports lower maximum toxicity alongside a slight decrease on standard language-modeling benchmarks. It also identifies limits tied to model size, learned data distributions, and trigger phrasing.
Problem
Open-web training data can contain harmful views and toxicity, but its scale makes human evaluation and filtration impractical.
Source: §1 · p. 1Method
The method scores researcher-written trigger phrases conditioned on document excerpts with a pretrained language model, then filters documents with high conditional likelihood.
Source: §3 · p. 2, §3.1 · p. 3Results
Models trained on likelihood-filtered data consistently show lower maximum toxicity than unfiltered and blocklist-filtered baselines, with a slight decrease on standard language-modeling benchmarks.
Source: §5.2 · p. 6, §8 · p. 7Takeaways & Limitations
Undesirable examples in standard language-modeling benchmarks highlight the need to identify and remove harmful data before releasing evaluation benchmarks built from internet text.
Source: §1 · p. 2, §5.1 · p. 5Filtration depends on a large pretrained model, may miss undesirable text poorly represented in its training distribution, and can flag counterspeech or expository text that overlaps with trigger language.
Source: §6 · p. 6
Abstract
from arXiv · showhide
Language models trained on large-scale unfiltered datasets curated from the open web acquire systemic biases, prejudices, and harmful views from their training data. We present a methodology for programmatically identifying and removing harmful text from web-scale datasets. A pretrained language model is used to calculate the log-likelihood of researcher-written trigger phrases conditioned on a specific document, which is used to identify and filter documents from the dataset. We demonstrate that models trained on this filtered dataset exhibit lower propensity to generate harmful text, with a marginal decrease in performance on standard language modeling benchmarks compared to unfiltered baselines. We provide a partial explanation for this performance gap by surfacing examples of hate speech and other undesirable content from standard language modeling benchmarks. Finally, we discuss the generalization of this method and how trigger phrases which reflect specific values can be used by researchers to build language models which are more closely aligned with their values.
1 Introduction
Web-scraped language-model data can contain harmful views and toxic content, while its scale makes human review and filtering impractical. The paper proposes programmatic identification and removal, tests it with human labels and model toxicity, and surfaces undesirable content in existing benchmarks.
1 Introduction
- Web-scale training corpora may contain systemic bias, prejudice, hate speech, abuse, and other toxicity, making effective removal important for both training data and internet-derived benchmarks.The size of these datasets makes reading them in their entirety impractical.Source: §1 · p. 1
- The paper contributes a programmatic method for identifying and removing undesirable text, with human-labelled experiments validating that it consistently identifies non-value-aligned content.Source: §1 · p. 1
- Experiments report lower maximum toxicity for models trained on filtered data, while benchmark analysis surfaces undesirable examples and highlights the need to identify harmful data in evaluation corpora.Source: §1 · p. 2
2 Related Work
Prior work addresses toxicity through token-level filtering or generation-time interventions, and value alignment through small-scale fine-tuning. This paper’s related-work contrast is between those approaches and filtering non-value-aligned documents at dataset scale.
2 Related Work
- Word-level blocklists remove simple hateful text but miss harmful pages without listed words, can flag benign academic or expository uses, and may disproportionately filter text associated with minority identities.Source: §2 · p. 2
- Vocabulary shifting learns toxicity representations for individual tokens and boosts non-toxic token likelihoods, but assigns token connotations without regard to context.Source: §2 · p. 2
- Self-debiasing uses a language model’s learned knowledge to identify bias during generation, whereas this work intervenes during dataset curation.Source: §2 · p. 2
- Prior fine-tuning with 80 handwritten question-answer pairs can reduce non-value-aligned generation, while this work filters large-scale data and lets researchers specify what to remove [Solaiman and Dennison, 2021].Source: §2 · p. 2
3 Conditional-Likelihood Filtration
Conditional-likelihood filtration uses a pretrained model’s learned distribution and researcher-written triggers to identify documents expressing similar rhetoric for removal. Its trigger-based design can be combined with a narrower blocklist and repeated with new triggers.
3 Conditional-Likelihood Filtration
- A pretrained language model scores researcher-written trigger phrases conditioned on document excerpts, and documents with high conditional likelihood are removed from the training data.The method measures the maximum toxicity of generations from models trained on the filtered dataset using the PERSPECTIVE API.Source: §3 · p. 2
- Conditional-likelihood filtration can be paired with a narrower blocklist to retain expository context and counterspeech while minimizing undesirable content.Source: §3 · p. 2
- The method can be run iteratively with new trigger phrases to capture emergent forms of unwanted rhetoric.Source: §3 · p. 2
3.1 Methodology
- The methodology appends triggers to excerpts from the beginning of each document, ranks documents by conditional likelihood, and selects a removal threshold to minimize removal of neutral or value-aligned text.Because of memory constraints, concatenated sequences are truncated to 384 tokens; the method assumes a webpage heading reflects the content that follows.Source: §3.1 · p. 3
4 Dataset creation & human evaluation
The study uses human evaluators to label sampled documents and assesses likelihood-based ranking, removal thresholds, and filtration overlap. High-likelihood documents were more often judged harmful, though some removed documents were expository or counterspeech.
4 Dataset creation & human evaluation
- 9.43% of filtered documents were labeled harmful, versus 0.66% of retained documents; 5.86% of filtered data was counterspeech or expository text, versus 0.80% of retained data.The figures summarize human-label composition in Table 4.Source: §4 · p. 3, Table 4 · p. 3
- Documents in the top 10% of conditional log-likelihood under the baseline-1.5B model were more likely to be classified as harmful than documents with low likelihood.Source: §4 · p. 3
- A threshold above -4.0 produced a post-filtration dataset 3.7% smaller than the original after word-level blocklist filtering.Thresholds were selected to minimize removal of non-harmful data.Source: §4 · p. 4
- Conditional-likelihood filtration and PERSPECTIVE capture distinct subsets, and PERSPECTIVE does not capture the majority of data removed by conditional-likelihood filtration.Source: Figure 3 · p. 4
5 Results
Across four 355M model setups, likelihood-filtered training lowered maximum toxicity compared with unfiltered and blocklist-filtered baselines, while unfiltered models performed best on both language-modeling benchmarks.
5 Results
- The comparison covers four 355M setups; likelihood filtration follows blocklist filtration, and one setup finetunes an existing blocklist-filtered model for 30k steps.The four setups use unfiltered data, blocklist-filtered data, finetuning on likelihood-filtered data, or training on likelihood-filtered data.Source: §5 · p. 4
5.1 Language modeling benchmarks
- Unfiltered models perform best on LAMBADA and lm1b, while likelihood-filtered models significantly outperform blocklist-filtered models on both.The likelihood-filtered dataset retains 96.3% of the original dataset.Source: Figure 4 · p. 5, §5.1 · p. 5
- 2.3% of lm1b evaluation examples contain blocklisted words, and the authors warn that toxic benchmark content may incentivize models more likely to generate harmful text.Cross-referencing also surfaced examples of hate speech and other problematic content in lm1b.Source: §5.1 · p. 5
5.2 Maximum toxicity scores
- Likelihood-filtered models consistently exhibit lower maximum toxicity than unfiltered and blocklist-filtered baselines, measured with PERSPECTIVE ratings of 5,000 generations.The evaluation uses prompts from REALTOXICITYPROMPTS and scores generations along the TOXICITY axis.Source: Figure 5 · p. 6, §5.2 · p. 6
5.3 Ablation by Trigger Phrase
- Across trigger phrases addressing different harm axes, models consistently show lower maximum toxicity, including phrases focused on racism and nationalism.The trigger-phrase ablation trained separate 355M models on datasets filtered with a single trigger phrase.Source: §5.3 · p. 6
6 Limitations & Future Work
Conditional-likelihood filtration is computationally intensive, varies in effectiveness across trigger topics, and can remove counterspeech; it also has temporal and comparison limitations.
6 Limitations & Future Work
- Conditional-likelihood filtration identifies undesirable text for removal from a pretraining corpus but depends on a computationally intensive large language model pretrained on unfiltered data; 128M- and 355M-parameter models did not reliably surface trigger-related text.Source: §6 · p. 6
- Political triggers surfaced harmful text more effectively than sexism or homophobia triggers, and overlapping language sometimes led to counterspeech and expository text being flagged.The authors suggest the pattern may reflect overrepresentation of 2011–2020 news domains in their corpus.Source: §6 · p. 6
- Conditional-likelihood filtration removes material distinct from blocklist-filtered text, but large-scale comparisons of the approaches remain future work.The proposed direction is to move from overly broad blocklists toward more nuanced, model-based approaches.Source: §6 · pp. 6–7
- Programmatically generated adversarial triggers did not successfully surface candidate text, likely because they do not fit natural-language distributions.The authors suggest future work on programmatically writing natural-language trigger phrases.Source: §6 · p. 7
- Filtration may miss undesirable text about recent events when those events fall outside the pretrained model's corpus-curation window.The authors recommend documenting the temporal coverage of the baseline model's pretraining corpus.Source: §6 · p. 7
7 Risks and Social Impact
Value-based filtration is context-dependent and adaptable, but that adaptability also creates potential for misuse; human evaluation is advised before deployment.
7 Risks and Social Impact
- Because triggers can be adapted to different social contexts, malicious users could also remove oppositional speech and promote their own ideas.The authors say adverse uses of value-alignment methods need further study.Source: §7 · p. 7
- The authors used human annotators to reduce the risk of removing beneficial speech and say human evaluation should precede deployment.They call for further research into the method's limitations.Source: §7 · p. 7
- What counts as undesirable text is difficult and context-dependent, adding challenges to automated flagging.The paper ties removal decisions to the social and ethical contexts of deployment.Source: §7 · p. 7
8 Conclusion
The authors use pretrained-model knowledge and researcher-written triggers to filter web-scale text, reporting lower maximum toxicity alongside a slight benchmark-performance decrease. They frame the method as scalable partial mitigation that can be adapted to different values.
8 Conclusion
- Filtered-data models exhibit lower maximum toxicity and a slight decrease on standard language-modeling benchmarks, while benchmark examples offer a partial explanation for the performance gap.The authors note that lm1b contains examples using words found in standard word-level blocklists and urge benchmark-content review before public release.Source: §8 · p. 7
- Conditional-likelihood filtration offers a scalable way to identify and remove undesirable text from web-scale corpora, partially mitigating toxicity learned during pretraining.The authors explicitly state that the method does not solve corpus-based harm in natural language processing.Source: §8 · p. 7
- Researchers can adapt the trigger phrases to reflect other value sets and curate language models more closely aligned with their values.The paper's method uses researcher-written triggers to specify the content targeted for removal.Source: §8 · p. 8
A.1 Pretrained filtration model
Table A.1 is identified as detailing the pretrained 1517M model used for filtration.
A.1 Pretrained filtration model
- Table A.1 is identified as detailing the pretrained 1517M model used for filtration.Source: Table A.1 · p. 10
A.2 Ablations
The appendix reports LAMBADA accuracy for likelihood-filtered models under pretraining and finetuning, then gives examples from the benchmark and of webpages removed by filtering.
A.2.1 Trained on likelihood-filtered
- LAMBADA last-token accuracy was 67.99% for the unfiltered baseline, 62.19% for blocklist-only filtering, and 67.54% for the union of triggers.Single-trigger models scored 68.00% (working-class trigger), 67.10% (Black-people trigger), 68.82% (Antifa trigger), and 69.13% (social-justice-warrior trigger); models were pretrained for 360k steps.Source: §A.2.1 · p. 10
A.2.2 Finetuned on likelihood-filtered
- Finetuned models had LAMBADA accuracy of 61.92% at 10k and 62.19% at 20k steps for the working-class trigger, and 60.42% at 30k steps for the Antifa trigger.At 30k steps, the Black-people, social-justice-warrior, and union-of-triggers variants scored 61.99%, 61.98%, and 61.89%, respectively.Source: §A.2.2 · p. 10
A.3 One Billion Word Benchmark samples with frequently blocklisted words
- The lm1b benchmark examples contain words found in commonly used word-level blocklists.Source: Table 1 · p. 11
A.4 Likelihood-filtered examples
- Table 2 shows the first 384 tokens of webpages filtered using the Antifa trigger at a threshold above -4.00.Source: Table 2 · p. 12