PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs

Kyle O'Brien, Stephen Casper, Quentin Anthony, Tomek Korbak, Robert Kirk, Xander Davies, Ishan Mishra, Geoffrey Irving, Yarin Gal, Stella Biderman

arXiv:2508.06601v2cs.LGcs.AI

TL;DR

Open-weight models can be modified downstream, while the science of their safety remains nascent, motivating safeguards that remain effective after tampering. The paper filters dual-use content during pretraining and finds stronger resistance to adversarial fine-tuning, while showing that in-context harmful information still requires complementary defenses.

  • Problem

    Open-weight models can be modified arbitrarily, and the science of open-weight model safety remains nascent.

  • Method

    The paper trains language models from scratch using a multi-stage pipeline that filters dual-use content from pretraining data, alongside post-training safeguard baselines and complementary defenses.

  • Results

    The filtered models resist up to 10,000 steps and 300M tokens of adversarial fine-tuning on biothreat-related text, improving by more than an order of magnitude over post-training baselines.

  • Takeaways & Limitations

    Data filtering is a promising safeguard layer, but in-context harmful knowledge and staged attacks motivate defense in depth.

  • Takeaways & Limitations

    The experiments cover unimodal 6.9B-parameter models without instruction fine-tuning and focus on biothreat proxy knowledge evaluated with multiple-choice questions.

Abstract

from arXiv · show

Open-weight AI systems offer unique benefits, including enhanced transparency, open research, and decentralized access. However, they are vulnerable to tampering attacks which can efficiently elicit harmful behaviors by modifying weights or activations. Currently, there is not yet a robust science of open-weight model risk management. Existing safety fine-tuning methods and other post-training techniques have struggled to make LLMs resistant to more than a few dozen steps of adversarial fine-tuning. In this paper, we investigate whether filtering text about dual-use topics from training data can prevent unwanted capabilities and serve as a more tamper-resistant safeguard. We introduce a multi-stage pipeline for scalable data filtering and show that it offers a tractable and effective method for minimizing biothreat proxy knowledge in LLMs. We pretrain multiple 6.9B-parameter models from scratch and find that they exhibit substantial resistance to adversarial fine-tuning attacks on up to 10,000 steps and 300M tokens of biothreat-related text -- outperforming existing post-training baselines by over an order of magnitude -- with no observed degradation to unrelated capabilities. However, while filtered models lack internalized dangerous knowledge, we find that they can still leverage such information when it is provided in context (e.g., via search tool augmentation), demonstrating a need for a defense-in-depth approach. Overall, these findings help to establish pretraining data curation as a promising layer of defense for open-weight AI systems.

1 INTRODUCTION

Open-weight models provide transparency, research access, and decentralized benefits, but their weights can be modified arbitrarily and existing safeguards can be undone. This paper investigates whether filtering dual-use content during pretraining can produce tamper-resistant ignorance of biothreat proxy knowledge while preserving unrelated capabilities.

  • Open-weight models offer transparency, research access, and deconcentration of power, but downstream modifications create distinctive safety risks.
  • The paper targets robust ignorance of biothreat proxy knowledge by filtering dual-use content from pretraining data.The authors hypothesize that knowledge not learned during pretraining will be harder for attackers to elicit.
  • A multi-stage filtering pipeline uses less than 1% of total training FLOPS and prevents biothreat proxy capabilities without observed degradation to unrelated capabilities.
  • 10,000 steps and 300M tokens of adversarial fine-tuning mark the reported tamper-resistance limit, exceeding post-training baselines by more than an order of magnitude.
  • Filtered models remain vulnerable when harmful knowledge is supplied in context, motivating complementary defenses and a defense-in-depth strategy.The paper reports that none of the tested defenses resist staged attacks combining fine-tuning with in-context retrieval.
  • The released suite contains 6.9B-parameter models trained with combinations of data filtering and post-training safeguards for studying training-data removal.The paper also studies multiple-choice safeguard evaluation, hybrid unlearning, and synthetic document training challenges.

2 FILTERING PREVENTS TARGET CAPABILITIES

The paper evaluates scalable pretraining-data filtering as a way to prevent biothreat proxy knowledge while preserving unrelated capabilities. A multi-stage pipeline combines inexpensive keyword screening with classifier review, and the resulting models show strong resistance to input-space attacks.

  • Research design: The study trains models from scratch on filtered and unfiltered datasets to test whether removing proxy-biothreat documents durably prevents target knowledge.The evaluation uses filtered versions of a large pretraining corpus and compares their capabilities with a baseline trained on unfiltered data.
  • Stage 1 – Keyword Blocklist: 91.58% of pretraining documents and 90.64% of annealing documents bypass keyword escalation because they contain no more than two blocklist terms.Documents meeting the keyword criterion require no further review, enabling the pipeline to process most data with simple string lookups.
  • Stage 2 – ModernBERT Classifier: The pipeline escalates documents containing multiple blocked terms to a fine-tuned ModernBERT classifier, which reduces false positives before filtering.The classifier is trained on expert-labeled proxy and general-biology examples and reviews documents flagged by the blocklist.
  • Evaluation results: Filtering substantially inhibits biothreat proxy knowledge acquisition, performing best on cloze-style evaluations while preserving general knowledge with no apparent net degradation of non-bio capabilities.On robust multiple-choice evaluations, filtering outperforms the baseline but underperforms Circuit-Breaking methods.
  • Evaluation results: Data filtering improves robustness to input-space attacks, outperforming Circuit-Breaking under GCG-U attacks but underperforming it on few-shot attacks.The comparison is based on repeated black-box evaluations of adversarial attack settings.
  • Takeaway: The evaluations support pretraining-data curation as an effective technique for mitigating unwanted knowledge in LLMs.The paper presents filtering as a basis for significant safety improvements through careful curation of pretraining data.

3 FILTERING ACHIEVES STATE-OF-THE-ART TAMPER-RESISTANCE

The paper evaluates pretraining-data filtering against latent-space, adversarial fine-tuning, and benign fine-tuning attacks. Filtered models show strong tamper resistance, including sustained resistance to adversarial fine-tuning and robustness to benign adaptation.

  • Attack evaluation: The evaluation covers latent-space attacks, adversarial fine-tuning, and benign fine-tuning.These threats target model behavior through activation-space perturbations, harmful-data updates, or legitimate domain adaptation.
  • Adversarial fine-tuning: 10,000 steps and 305M tokens: filtered models remain resistant to adversarial fine-tuning on biothreat proxy text.The attacks use WMDP-Bio Forget data and include both full-parameter and LoRA runs.
  • Latent-space attacks: Filtering performs better on cloze evaluations, while CB+LAT performs better on multiple-choice evaluations under latent-space attacks.The two safeguards therefore show different relative strengths across evaluation formats.
  • Overall tamper resistance: Filtered models are the most resistant to attacks in all cases but one.Latent-space attacks are effective in multiple-choice evaluations but not in cloze evaluations, possibly reflecting learned multiple-choice heuristics.
  • Benign fine-tuning: Filtered models’ biothreat-proxy performance remains unchanged under benign WikiText fine-tuning.Circuit-Breaking and CB+LAT, by contrast, become ineffective under this benign intervention.
  • Comparison with prior work: Filtered models appear resistant to greater amounts of adversarial fine-tuning than related works have tested.The authors caution that fine-tuning attack configurations are difficult to compare directly.

4 DEFENSE-IN-DEPTH

The defense-in-depth evaluation combines pretraining-data filtering with Circuit-Breaking and examines retrieval augmentation and synthetic-document training. Combined safeguards improve coverage across several attacks, but retrieval and staged attacks expose important limits.

  • Retrieval augmentation: Filtering cannot prevent in-context retrieval of harmful information, whereas Circuit-Breaking can impair that retrieval.Filtered and baseline models perform well when answer-bearing passages are supplied in open-book tests.
  • Complementary safeguards: Combining filtering and Circuit-Breaking increases resistance to few-shot and latent-space attacks while preserving comparable performance on other evaluations.The results suggest improved coverage across diverse attacks when both interventions are applied.
  • Synthetic-document training: Synthetic biothreat-misinformation documents failed to substantially suppress biothreat proxy capability.They sometimes increased filtered models’ biothreat proxy knowledge and offered no compelling improvement over filtering alone.
  • Synthetic-document training: The authors conclude that scalable synthetic-document training is challenging and can be confounded by simple proxy evaluations.They suggest that future work may require carefully designed synthetic datasets.
  • Deployment scope: In closed-weight settings, contextual access to dual-use knowledge may remain useful for trusted users while being restricted for untrusted users.The authors identify this contextual capability as a limitation for open-weight safety but potentially useful in controlled deployments.

5 RELATED WORK

Related work frames open-weight safety through data curation, capability suppression, attack-based elicitation, and safety cases. Prior studies document vulnerabilities to prompt, latent-space, and fine-tuning attacks, motivating defenses that support inability-based safety arguments.

  • Data filtering and curation: Web-scale data curation is difficult because of direct costs, filtering errors, dataset-quality degradation, and multilingual content.These challenges complicate filtering harmful content before pretraining.
  • Capability suppression: Capability-suppression research includes refusal fine-tuning, machine unlearning, and other methods targeting harmful LLM capabilities.The cited work spans post-training safeguards and direct capability suppression techniques.
  • Black-box elicitation: Modern safety-fine-tuned LLMs remain vulnerable to prompt engineering and jailbreaking attacks that elicit harmful knowledge or behaviors.These black-box elicitation methods take multiple forms but share the goal of bypassing safety behavior.
  • White-box elicitation: White-box elicitation includes embedding-space, latent-space, and weight-space attacks using benign or adversarial fine-tuning.The literature studies attacks that directly manipulate internal representations or model weights.
  • Tamper resistance: Prior work finds few-shot fine-tuning attacks especially effective, while existing tamper-resistance methods often withstand only dozens or hundreds of examples.This establishes the prior robustness range against which stronger safeguards can be assessed.
  • LLM safety cases: Safety cases are structured, evidence-based arguments that a system poses an acceptable level of risk.The paper connects tamper robustness to evidence for inability-based safety cases, while noting that filtering and Circuit-Breaking are not unbreakable.

6 DISCUSSION

The discussion argues that pretraining data filtering improves tamper resistance but cannot address all attack paths or deployment risks. The authors therefore frame filtering as one component of broader, defense-in-depth risk management.

  • Implications for open- and closed-weight model risk management: 10,000 steps and 300M tokens mark state-of-the-art tamper resistance, but the authors caution that this remains a limited absolute amount of adversarial fine-tuning.They suggest high-quality adversarial data, rather than compute, may become the main bottleneck.
  • Comprehensive defense strategies: A comprehensive open-weight risk-management toolkit may combine data filtering and Circuit-Breaking with additional monitoring and mitigation strategies.The authors call for continued development of assessment and ecosystem-wide standards.
  • Implications for open- and closed-weight model risk management: Open-weight model risk depends partly on external actors’ resources and goals, which developers cannot control.The authors conclude that model safeguards alone cannot provide comprehensive risk management.
  • Limitations: The experiments cover unimodal 6.9B-parameter models without instruction fine-tuning, trained on 550B tokens and evaluated mainly through biothreat multiple-choice questions.The implemented attacks provide only a lower bound on worst-case behavior.
  • Limitations: The study uses WMDP-Bio as its target and does not address the broader challenge of defining acceptable versus unacceptable dual-use behavior.The authors identify contextual harm and exploitable grey areas as deployment-relevant specification challenges.
  • Limitations: Filtering does not prevent retrieval augmentation attacks, and it appears insufficient to suppress harmful propensities such as toxic text generation.These shortcomings motivate combining filtering with other defenses.
  • Impact and risk statement: The paper focuses on biothreat proxy knowledge, uses existing proxy datasets and benchmarks, introduces no novel attacks, and withholds its filtered pretraining dataset.The released models have weaker capabilities than existing open-weight peers.
  • Future research: The authors expect training larger, more capable, or multimodal model organisms to clarify how filtering scales across model sizes and modalities.They also present released models as testbeds for studying deep versus shallow ignorance and interpretability.

ACKNOWLDGEMENTS

The acknowledgements credit collaborators and infrastructure providers for discussions, methodological input, compute, cluster support, and maintenance enabling the filtering and tampering experiments.

  • ACKNOWLDGEMENTS: The authors thank numerous researchers for helpful discussions, methodological input, advising, and collaboration logistics.Several contributors also supported the paper’s writing and technical development.
  • ACKNOWLDGEMENTS: Donated GPUs and support from CoreWeave, Prime Intellect, GW4, and the UL Met Office enabled filter development, pretraining, and tampering experiments.The acknowledgements specifically connect infrastructure support to the research’s computational work.

CONTRIBUTIONS

The contributions section assigns leadership for pretraining, filtering, evaluation, safeguard baselines, adversarial experiments, writing, and broader technical and advisory support.

  • CONTRIBUTIONS: Kyle O’Brien led pretraining and data filtering, resolved proxy-knowledge evaluation challenges, and wrote much of the paper.His work produced the Robust MCQA and Verified Cloze WMDP-Bio splits.
  • CONTRIBUTIONS: Stephen Casper led baseline-safeguard experiments and adversarial evaluations, and wrote much of the paper.
  • CONTRIBUTIONS: Quentin Anthony, Tomek Korbak, Robert Kirk, Xander Davies, and Ishan Mishra provided technical, logistical, and writing support.Geoffrey Irving, Yarin Gal, and Stella Biderman advised the project, with Biderman responsible for its original idea.

B EXPERIMENTS WITH MULTI-STAGE PRETRAINING FILTERING

The appendix compares single-stage and multi-stage filtering pipelines for pretraining. Their trade-off is between reducing target knowledge and preserving general knowledge.

  • Filtering approaches: Multi-stage filtering escalates documents containing at least two blocked terms to a ModernBERT classifier for review.This reduces false positives relative to outright rejection, at the expense of increased false negatives.
  • Filtering approaches: Single-stage filtering rejects documents containing at least two blocked terms instead of escalating them for review.
  • Results: Single-stage filtering produces the lowest WMDP performance but regresses average general-knowledge benchmark performance by less than one percentage point.
  • Results: Multi-stage pretraining also significantly regresses WMDP-Bio performance, while filtering combinations differ in their balance between target reduction and general-knowledge preservation.The appendix concludes that the optimal configuration depends on practitioners’ priorities.

C FILTERING’S IMPACT ON TOTAL FLOPS

Filtering adds a small amount of computation beyond model training, with the increase depending on whether filtering covers both pretraining and annealing or only annealing.

  • 8.32PD FLOPs estimates the pretraining computation, accounting for activation checkpointing and additional forward passes.The paper uses C = 8.32PD for its pretraining runs.
  • 2.62e20 FLOPS is the estimated cost of end-to-end multi-stage filtering across pretraining and annealing.The weak-filter setup costs 1.92e20 FLOPS because it performs no GPU filtering during pretraining.
  • Less than 1% is the total-compute increase from the most comprehensive filtering approach.The weak-filter configuration increases total computation by 0.61%.
  • Filtering overhead is therefore small relative to model-training costs in both evaluated configurations.The comparison concerns end-to-end weak filtering and weak filtering only.

D.1.1 DATASETS

The filtering datasets combine labeled proxy and general-biology papers with synthetic augmentations and classifier-based filtering, while evaluation must account for multiple-choice shortcut behavior and uneven post-training effects.

  • Datasets: 24,453 proxy papers and 66,360 general-biology papers form the WMDP-Bio Forget and bio Retain corpora.The corpora are sourced from PubMed and provide labeled examples for filtering.
  • Dataset limitation: The labeled corpus is dominated by scientific papers, so classifiers may generalize poorly to lectures, news, and exam questions.The authors use data augmentation and other countermeasures to increase diversity.
  • Datasets: Three augmentations per proxy document recast examples as lectures, exams, or articles to diversify the filtering data.Augmentations from each source document remain in the same split.
  • Classifier filter: ModernBERT-Large is the second-stage classifier, applying chunked inference and filtering approximately 5% of the annealing dataset at threshold 0.0105.The classifier was chosen over earlier BERT variants because of its larger context window and observed performance.
  • Evaluation: Multiple-choice benchmarks can reward heuristic shortcuts, with 51.38% and 50.12% accuracy substantially exceeding the 25% random baseline.This complicates interpretation of whether accuracy reflects retained knowledge.

E COMPARING OUR ADVERSARIAL FINE-TUNING ATTACKS TO PRIOR WORKS

The paper compares its adversarial fine-tuning attacks with prior work using differing configurations and reports resistance tested at larger attack scales than earlier studies.

  • Comparison scope: Comparisons across adversarial fine-tuning studies are difficult because models, data, context sizes, batch sizes, and hyperparameters differ.The paper therefore reports configuration quantities separately and treats the comparison as rough.
  • Comparison scope: The comparison reports full fine-tuning runs rather than the minimum fine-tuning needed to produce successful attacks.Some prior runs were substantially longer than necessary for successful attacks.
  • Comparison result: The authors report the greatest numbers of unique examples, total steps, and step × batch size product among the compared works.A concurrent work reported extensive attacks without quantitative configuration details, so it could not be compared on those measures.

F SYNTHETIC DOCUMENT TRAINING EXPERIMENTS

Synthetic document training did not improve resistance beyond filtering alone and sometimes made biothreat-proxy performance worse.

  • Synthetic document training showed limited evidence of inhibiting biothreat-proxy learning during fine-tuning attacks.A learning-rate warmup stabilized training and slightly improved attack success in these experiments.
  • SDT produced no evidence of improvement over filtering alone across the evaluated attacks.It mildly impeded attack effectiveness under multiple-choice evaluation but failed to improve resistance under other attacks and sometimes degraded it.
  • SDT was more likely to increase than decrease model performance on biothreat-proxy evaluations.This result concerns the synthetic-document experiments described for the annealing phase.

G EXPERIMENTS ON MODELS FROM MAINI ET AL. (2025)

Experiments on small instruction-tuned models found that filtering offered only slight resistance to fine-tuning attacks and was particularly vulnerable to few-shot attacks, while CB methods were effective.

  • Evaluation: General capability was measured by average performance on MMLU, Lambada, PIQA, and HellaSwag, alongside jailbreak robustness on 250 held-out harmful-request examples.GPT-4o and a modified StrongReject prompt graded successful jailbreak compliance.
  • Results: 1.7B-parameter filtered models were only slightly more resilient to fine-tuning attacks and particularly vulnerable to few-shot attacks.The authors caution that these relatively small models may not generalize to large, state-of-the-art LLMs.
  • Results: Filtered models were fairly resistant to universal GCG-U and latent-space attacks after attack-tuning efforts.The evaluation used tuned few-shot, GCG-U, latent-space, full fine-tuning, and LoRA attacks.
  • Results: CB and CB+LAT were effective, whereas synthetic document training was largely ineffective.Synthetic training used incompetent compliances to harmful requests; the tested defenses included CB, CB+LAT, and SDT.
  • Interpretation: The results support possible limits of filtering for behaviors that do not require conveying precise information.The authors contrast these findings with their hypothesis that filtering can provide durable safeguards.

J FILTERED DOCUMENTS ANALYSIS

The filtered-document analysis indicates that scientific sources supplied many likely proxy-knowledge documents, while false positives were especially common in StackExchange and other nontechnical content.

  • Filtering quality: The filters were designed for high recall, but the authors did not measure classification performance on the actual training datasets.Near-random WMDP-Bio performance suggested that enough proxy-knowledge examples were classified, but precision remained unmeasured.
  • Sources: Most filtered annealing documents came from Semantic Scholar and DCLM, with scientific papers a major source of likely proxy knowledge.The source distribution was reported on a log scale in Figure 11.
  • Sources: Semantic Scholar removals commonly included biomedical, public-health, and virology papers.These categories were identified in a qualitative study of randomly sampled documents.
  • False positives: DCLM and Wikipedia also produced likely false positives, including nontechnical pandemic, drug, antimicrobial-resistance, and deceased-person discussions.The analysis identified scientific papers and technical proxy documents alongside these false-positive patterns.
  • False positives: Most filtered StackExchange documents were likely false positives, explaining differences between strong and weak filtering counts.FLAN also showed false positives involving Ebola summarization, drought monitoring, and translation tasks.
  • Sources: Camel documents commonly contained biology knowledge related to the proxy targets.The corresponding table presents two randomly sampled Camel documents removed by the weak filter.
  • Limitations: The authors conclude that the filters likely have high false-positive rates and could be made more precise.They report that fewer false positives could improve filtering while preserving the observed benchmark effects.
Loading 2508.06601v2…