PaperlayerView exact revision on arXiv

Source-linked AI summary

When Bad Data Leads to Good Models

Kenneth Li, Yida Chen, Fernanda Viégas, Martin Wattenberg

arXiv:2505.04741v1cs.LGcs.AIcs.CL

TL;DR

Toxic-data filtering may reduce data diversity and downstream capabilities, while base-model toxicity alone does not capture behavior after post-training. The paper studies pretraining and detoxification together, finding that toxic data can raise base-model toxicity yet make it easier to reduce and improve the detoxification–capability trade-off.

  • Problem

    The paper asks whether toxic pretraining data should be filtered when models are later customized through post-training, given the potential costs to data diversity and downstream capability.

    Source: §1 · p. 1

  • Method

    The authors combine a toy representation experiment with controlled Olmo-1B pretraining on clean and toxic data, then evaluate prompting and inference-time intervention for detoxification.

    Source: §1 · p. 1, §2.1 · p. 2, §5.3 · p. 6

  • Results

    Toxic data increases base-model toxicity but makes toxicity easier to mitigate, producing a better detoxification–capability trade-off with post-training.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Pretraining data selection should be treated empirically, with pretraining and post-training evaluated as an end-to-end system.

    Source: §7 · p. 9

    The experiments focus on toxicity, and the optimal amount of toxic pretraining data remains undetermined; too much may reduce steerability.

    Source: §7 · p. 9

Abstract

from arXiv · show

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

1. Introduction

The paper treats pretraining and post-training as a co-designed system, asking whether toxic data can make models easier to detoxify. Its experiments find that toxic pretraining raises base-model toxicity but can improve detoxification while preserving general capability.

1. Introduction

  • Filtering toxic data can reduce data diversity; Longpre et al. (2023) found it also reduced toxicity identification and downstream performance on most QA domains.Source: §1 · p. 1
  • The paper evaluates models after prompting and activation steering, framing pretraining and post-training as a unified system rather than judging base-model behavior alone.The authors hypothesize that increasing toxic data can improve base-model alignability up to a threshold.Source: §1 · p. 1
  • The authors’ motivation is that alignment methods may bypass, rather than unlearn, mechanisms that produce toxic generations, which can later be brought back into use.Source: §1 · p. 1
  • In a toy setting, increasing a feature’s training-data share tends to give it a less entangled representation.This motivates testing whether adding toxic data can make toxicity easier to identify and control.Source: §1 · p. 1
  • Olmo-1B models trained with more 4chan data show higher toxicity-probe accuracy across layers, which the authors interpret as stronger internal knowledge of toxicity.The authors propose that this knowledge may support detoxification during post-training.Source: §1 · p. 2
  • Toxic pretraining raises base-model toxicity, but post-training makes those models less toxic and yields a better detoxification–capability trade-off than the compared baselines.The comparison includes prompting and steering, as well as supervised finetuning, DPO, MEDA, and INST.Source: §1 · p. 2

2. A Motivating Experiment

The motivating experiment tests how a feature’s representation changes when that feature is rare or more prevalent in training data. In the toy transformer, increasing a feature’s share reduces its entanglement, motivating the paper’s proposal to add toxic data before detoxification.

2. A Motivating Experiment

  • The experiment studies how missing or scarce data affects representations, using superposition theory to motivate an entanglement measure.When features outnumber neurons, multiple feature representations must share an activation-space dimension.Source: §2 · p. 2

2.1. Entanglement of Features

  • Entanglement uses absolute cosine similarity between feature directions; superposition makes editing one feature project onto others, creating side effects.A feature with lower entanglement can stand out more, while other features become more entangled.Source: §2.1 · p. 2

2.2. Toy Experiment Setup

  • The toy setup treats each unique sequence from a cyclic Markov chain as a feature and varies the underrepresented chain’s sample share across datasets.It uses a 4-layer transformer with a 4-dimensional residual stream and trains each dataset configuration with 10 random seeds.Source: §2.2 · pp. 3–4

2.3. Experimental Results

  • As underrepresented features receive more training data, their entanglement drops toward the average entanglement of other features.Figure 3 illustrates the sharp drop; the reported natural entanglement for compressing features into a 4-dimensional residual stream is around 0.8.Source: Figure 3 · p. 4, §2.3 · p. 4
  • The authors connect the toy result to toxicity: because C4 contains little toxic data, toxicity may be entangled with common concepts, so aggressive steering could harm general capabilities.This motivates their proposal to add toxic data to pretraining.Source: §2.3 · p. 4

3. Pretraining with Toxic Data

The section varies toxic-data exposure during pretraining and evaluates base-model capability and toxicity detection. More toxic data consistently improves toxicity detection, while a moderate amount can enhance general capability.

3. Pretraining with Toxic Data

  • Moderate toxic-data exposure can enhance MMLU capability, while toxicity detection on ToxiGen improves consistently as toxic data increases.The figure tracks both measures as toxic data increases in the pretraining dataset.Source: Figure 4 · p. 4, §3 · p. 5
  • The controlled data spectrum combines near-clean C4 with near-completely toxic 4chan, increasing toxic data from 0% to 25% in 5% increments.Clean-data quantity stays constant; 25% is an exaggerated ratio chosen to capture the sweet spot, not a recommendation.Source: §3 · pp. 4–5
  • With 4chan added, the probe-accuracy distribution shows more attention heads specializing in toxicity, forming a fatter right tail.Source: Figure 5 · p. 5

4. Toxic Data Improves Concept Building

The authors probe toxicity representations in Olmo-1B by training linear classifiers on model activations. Models trained with 25% 4chan data show higher probe accuracy and more heads that specialize in toxicity.

4. Toxic Data Improves Concept Building

  • Linear probes classify toxic versus non-toxic ToxiGen examples from each attention head’s last-token activations, using a 4:1 train-validation split.Validation accuracy measures how separable the toxicity representation is for each head.Source: §4 · p. 5
  • Probe validation accuracy was significantly higher for the toxicity-trained model with 25% 4chan than the C4-only model (p = 0.0002), with a 95% confidence interval for the difference of [0.67, 1.18].Source: §4 · p. 5
  • For inference-time intervention, targeting high-accuracy heads is important for changing behavior while minimizing damage to overall capabilities.Source: §4 · p. 5
  • The verbalization experiment identified approximately 6 toxic tokens for the C4-only model and 11 for the model trained with toxicity data.The tokens were selected by proximity of their unembedding vectors to the probe direction of the most accurate layer.Source: §4 · p. 5

5. Toxic Data Improves Alignability

This section tests whether toxic pretraining data improves how effectively post-training can reduce toxicity, using ITI and other detoxification methods. Although untreated toxicity rises with toxic-data share, ITI and other post-training techniques can make toxicity easier to reduce.

5.3. Experimental Results

  • Without intervention, toxicity rises with 4chan share, but under ITI it falls through 10% and rebounds beyond that while remaining below the clean model.The reported 10% sweet spot is for the relatively small model and data scale used; practitioners should determine the proportion empirically.Source: Figure 6 · p. 6, §5.3 · pp. 6–7

5.1. Background on Inference-Time Intervention

  • ITI shifts attention-head activations along attribute directions during decoding; experiments intervene on 30 heads at weak (4), medium (8), or strong (12) strength.The intervention strength and number of heads are tuned to balance general capability with the target attribute.Source: §5.1 · p. 6

5.2. Experimental Baseline Comparisons

  • The baselines include a detoxifying instruction prompt, MEDA and INST toxicity annotations, and SFT and DPO; the latter use Tulu V2 and preference or instruction datasets.MEDA appends a Perspective API toxicity score, while INST prepends a natural-language toxicity label.Source: §5.2 · p. 6

5.3. Experimental Results

  • At 10% 4chan, prompting and steering show better alignability than clean-data models; weak steering beats all baselines in detoxification with the lowest cross-entropy loss.The comparison uses the 10%-toxic-data model selected from Figure 6; stronger intervention remains available when greater detoxification is needed.Source: Table 1 · p. 7, §5.3 · p. 7
  • Adding toxic pretraining data also boosts the detoxification effectiveness of SFT and DPO, following a trend similar to ITI’s smile-shaped curve.This suggests the finding may extend beyond linear steering.Source: §5.3 · p. 7, Table 2 · p. 8

5.4. Red-teaming Experiments

  • Without ITI, both models have GCG attack success rates above 80%; with strong ITI, rates fall for both, and the model trained with 10% toxic data achieves the lowest rate, 38.5%.Source: §5.4 · p. 7, Table 3 · p. 8

6. Related work

Prior detoxification approaches either finetune models or control generation at decoding time, with reported trade-offs in capability preservation and robustness. This work situates itself in the less-studied co-design of pre- and post-training, exploring how pretraining data frequency relates to linear representations and detoxification trade-offs.

Finetuning-based Detoxification

  • Finetuning-based detoxification, including RLHF and DPO, can degrade original capabilities; DPO defenses may be overridden, and even unintentional finetuning can weaken alignment.The proposed approach merges the training stages to a degree, with the hypothesis that better toxicity representations can be more effectively suppressed at deployment.Source: §6 · p. 8

Detoxification with Controlled Generation

  • Controlled-generation methods can impose inference costs or hurt fluency and capability; activation-based methods such as ITI rely on well-developed linear representations.This paper investigates conditions under which such linearity emerges more effectively.Source: §6 · p. 8

Co-design of Pre- and Post-training

  • Co-design work is relatively scarce; this study extends work on pretraining frequency and linear representations, and reports a better toxicity–fluency trade-off than textual toxicity annotations.Unlike that annotation approach, it adds toxic data directly to pretraining.Source: §6 · p. 8

7. Conclusion and Future Work

The paper’s toxicity case study finds that more toxic pretraining data improves toxicity representation and makes toxicity easier to mitigate after training. It argues for evaluating pre- and post-training together while identifying unresolved questions about scope, data quantity, and mechanisms.

7. Conclusion and Future Work

  • More toxic pretraining data improves toxic-content classification and yields less-entangled toxicity representations, while making toxicity easier to mitigate during post-training.The paper frames this as a case study of traditionally unwanted pretraining data.Source: §7 · p. 9
  • Pretraining data selection should be treated as an empirical question, with pre- and post-training assessed as one system against overall goals.The paper presents toxicity as a case study of the possible synergy between the two stages.Source: §7 · p. 9
  • Whether the findings generalize from toxicity to other alignment-related features remains a future research question.Source: §7 · p. 9
  • The optimal amount of toxic pretraining data is undetermined, and steerability may decrease when too much is included.The paper calls for a precise relationship between feature frequency and post-training steerability.Source: §7 · p. 9
  • The mechanisms behind the findings remain underexplored, including how feature count and hidden-space size relate to entanglement reduction.Source: §7 · p. 9

Impact Statement

The paper studies methods for reducing toxic content generation, which it identifies as having natural societal benefits.

Impact Statement

  • Reducing the likelihood that models generate toxic content has natural societal benefits.Source: Impact Statement · p. 9

A. Capability Evaluation of Pretrained Models

Capability evaluations found little overall degradation as toxic-data proportions increased. On MMLU, accuracy varied only slightly across checkpoints trained with different toxic-data proportions.

A. Capability Evaluation of Pretrained Models

B. Attention Head Accuracy Heatmaps

The appendix compares attention-head validation accuracies for Olmo-1B trained with C4 alone and with C4 plus 4chan. Its heatmaps show a slight overall accuracy increase with 4chan, while the accompanying bar plot presents Figure 5 in a different style.

B. Attention Head Accuracy Heatmaps

C. Verbalization of Toxicity Directions

The section lists the 50 tokens closest to the toxicity direction for models trained on C4 only and on C4 with 4chan. The two lists contain different tokens, including terms such as “hate” in the C4-only list and “Jew” and “Jews” in the C4-and-4chan list.

C. Verbalization of Toxicity Directions

Loading 2505.04741v1…