PaperlayerView exact revision on arXiv

Source-linked AI summary

The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Adam Khoja, Zhenqi Zhao, Ariel Herbert-Voss, Cort B. Breuer, Samuel Marks, Oam Patel, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Russell Kaplan, Ian Steneker, David Campbell, Brad Jokubaitis, Alex Levinson, Jean Wang, William Qian, Kallol Krishna Karmakar, Steven Basart, Stephen Fitz, Mindy Levine, Ponnurangam Kumaraguru, Uday Tupakula, Vijay Varadharajan, Ruoyu Wang, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, Dan Hendrycks

arXiv:2403.03218v7cs.LGcs.AIcs.CLcs.CY

TL;DR

LLM hazardous-capability evaluations are often private and narrowly focused, while existing safeguards and unlearning methods leave gaps in mitigating malicious use. The paper introduces the public WMDP benchmark and RMU, which reduces hazardous-knowledge performance while largely preserving general capabilities, though closely related capabilities can also decline.

  • Problem

    Current hazardous-capability evaluations are private and highly specific, limiting open research and broad measurement of malicious-use risks.

  • Method

    The paper develops WMDP, a filtered 3,668-question proxy benchmark across biosecurity, cybersecurity, and chemical security, and RMU, which perturbs representations on hazardous data while preserving benign-data activations.

  • Results

    RMU reduces performance on WMDP while mostly retaining general capabilities, but accuracy drops on closely related fields such as virology and computer security.

  • Takeaways & Limitations

    WMDP enables public evaluation of hazardous knowledge and unlearning methods, supporting unlearning as one component of broader malicious-use mitigation.

  • Takeaways & Limitations

    WMDP unlearning does not address relearning in open-source models, and unlearning can reduce capabilities needed by defenders in dual-use domains.

Abstract

from arXiv · show

The White House Executive Order on Artificial Intelligence highlights the risks of large language models (LLMs) empowering malicious actors in developing biological, cyber, and chemical weapons. To measure these risks of malicious use, government institutions and major AI labs are developing evaluations for hazardous capabilities in LLMs. However, current evaluations are private, preventing further research into mitigating risk. Furthermore, they focus on only a few, highly specific pathways for malicious use. To fill these gaps, we publicly release the Weapons of Mass Destruction Proxy (WMDP) benchmark, a dataset of 3,668 multiple-choice questions that serve as a proxy measurement of hazardous knowledge in biosecurity, cybersecurity, and chemical security. WMDP was developed by a consortium of academics and technical consultants, and was stringently filtered to eliminate sensitive information prior to public release. WMDP serves two roles: first, as an evaluation for hazardous knowledge in LLMs, and second, as a benchmark for unlearning methods to remove such hazardous knowledge. To guide progress on unlearning, we develop RMU, a state-of-the-art unlearning method based on controlling model representations. RMU reduces model performance on WMDP while maintaining general capabilities in areas such as biology and computer science, suggesting that unlearning may be a concrete path towards reducing malicious use from LLMs. We release our benchmark and code publicly at https://wmdp.ai

1 Introduction

The paper addresses gaps in evaluating and mitigating malicious use of LLMs by introducing WMDP and RMU. WMDP measures hazardous knowledge, while RMU removes such knowledge while preserving general capabilities, subject to capability trade-offs and broader sociotechnical safeguards.

  • Motivation: Current hazardous-capability evaluations are private, manual, and highly specific, limiting open research and broad coverage of malicious-use risks.Existing evaluations test only a few pathways, so low performance does not guarantee security across the broader distribution of risks.
  • Motivation: Existing safeguards based on refusal training or pretraining-data filtering can be bypassed through adversarial attacks, jailbreaks, or later finetuning.The paper motivates unlearning as a safeguard for closed-source models because hazardous knowledge can be removed before serving.
  • Contributions: WMDP is a benchmark of 3,668 multiple-choice questions measuring hazardous knowledge in biosecurity, cybersecurity, and chemical security.Academics and technical consultants developed threat models and questions, while a stringent filtering process removed sensitive information before release.
  • Contributions: RMU perturbs activations on hazardous data while preserving activations on benign data, reducing WMDP performance while mostly retaining general capabilities.The method is designed to remove hazardous knowledge while preserving capabilities measured by MMLU and MT-Bench.
  • Limitations and scope: Unlearning inherently reduces capabilities and may harm defenders when scientific knowledge is dual-use, motivating structured access and complementary interventions.The authors present unlearning as one component of a larger sociotechnical solution rather than a complete mitigation.

2 Related Work

Related work documents growing concern about LLM-assisted malicious use and limitations in existing safety evaluations, safeguards, and unlearning benchmarks. The paper positions WMDP as an open evaluation for hazardous knowledge and unlearning research.

  • Evaluating risk from LLMs: LLMs may assist with planning bioattacks, procuring pathogens, synthesizing dangerous chemicals, and conducting cyberattacks.Major AI labs have developed frameworks for measuring and mitigating biological, cybersecurity, and chemical hazards.
  • Evaluating risk from LLMs: Many hazardous-capability evaluations remain private to the labs that developed them, motivating an open-source evaluation for broader benchmarking and unlearning research.WMDP is presented as the paper’s open alternative.
  • Hazard levels of knowledge: WMDP organizes hazard mitigation around measuring and removing yellow-category knowledge while retaining as much green-category knowledge as possible.The figure distinguishes the hazardous red category from the proxy knowledge targeted by WMDP.
  • Mitigating risk from LLMs: Input filtering and preference-based safety methods can be vulnerable to jailbreaks and adversarial attacks, while removing hazardous pretraining data may not prevent later reintroduction through finetuning.These limitations motivate interventions that reduce inherent hazardous knowledge in models.
  • Machine unlearning: Earlier machine-unlearning methods mainly erased specific samples or facts, and existing methods struggle to remove knowledge without access to all relevant training data.Recent work broadens targets to concepts such as gender, harmful behaviors, or fictional universes.
  • Machine unlearning: Prior unlearning work has not proven elimination of scientific knowledge that enables malicious use, while many benchmarks use specific samples or artificially chosen deletion sets.This gap motivates a benchmark focused on hazardous scientific knowledge.

3 The WMDP Benchmark

WMDP is a public, expert-written benchmark of 3,668 multiple-choice questions designed to measure hazardous knowledge while avoiding release of sensitive information. Its questions use threat models across biosecurity and cybersecurity, covering hazardous-information precursors, neighbors, components, and attack stages.

  • The WMDP Benchmark: 3,668 expert-written multiple-choice questions comprise WMDP across biosecurity, cybersecurity, and chemistry.The benchmark cost over $200K to develop and aims to reduce WMDP accuracy while preserving performance on general benchmarks.
  • The WMDP Benchmark: WMDP is an automatic, public benchmark intended to guide measurement and reduction of models’ hazardous capabilities.The dataset also includes additional MMLU-based unlearning benchmarks.
  • Dataset Form: Four-choice questions provide an automatic measure that the broader research community can readily iterate on.Multiple-choice questions are a common paradigm for testing language-model knowledge.
  • Dataset Function: Because WMDP is a proxy, low scores likely indicate missing knowledge needed for malicious assistance, whereas high scores do not necessarily establish unsafe behavior.A high score may not imply that a model can combine knowledge into the sequence of steps needed to create a weapon.
  • Dataset Function: Questions approximate hazardous knowledge through precursors, neighbors, or components rather than directly collecting sensitive information.Examples include precursor research for enhanced pathogens, chemistry questions near hazard boundaries, and primarily offensive cyberweapon components.
  • Biosecurity Threat Model: The biosecurity threat model maps questions onto ideation and the design-build-test-learn cycle, while covering selected dual-use areas with reduced disclosure risk.Coverage includes historical bioweapons, enhanced potential pandemic pathogens, viral reverse genetics, selected synthetic-biology topics, and assay-based testing and learning.
  • Cybersecurity Threat Model: The cybersecurity threat model covers reconnaissance, weaponization, exploitation, and post-exploitation stages of cyberattacks.Reconnaissance questions address foundational concepts and techniques for uncovering information about targets.

4 RMU: Unlearning Inspired By Representation Engineering

RMU is a representation-based finetuning method that perturbs activations on hazardous data while preserving activations on benign data. It uses a two-part loss and targeted updates to reduce hazardous knowledge while retaining general capabilities.

  • Method scope: RMU finetunes model weights to unlearn hazardous biosecurity and cybersecurity knowledge, but does not apply unlearning to chemistry.The authors treat WMDP-Chem as a hazard-measurement tool while remaining uncertain whether its mitigation benefits outweigh capability costs.
  • Forget loss: The forget loss changes hazardous-data representations by increasing activation norms, making them difficult for later layers to process.The loss uses the original frozen model’s hidden states and a fixed random unit vector; c controls activation scaling.
  • Retain loss: The retain loss uses an ℓ2 penalty to keep benign-data activations close to those of the original model.This regularization limits losses in general capabilities during unlearning.
  • Full loss: RMU combines forget and retain losses as L = Lforget + α · Lretain, then minimizes the combined objective.α weights the retain term in the full loss.
  • Training data and updates: RMU approximates hazardous knowledge with forget datasets and preserves general language modeling with a qualitatively distinct Wikitext retain set.The method interleaves updates across hazardous distributions and updates only selected layers to save memory.

5 Experimental Results

The experiments evaluate whether RMU removes hazardous knowledge while preserving general knowledge, fluency, and resistance to recovery. RMU reduces WMDP performance to near random, largely retains broad capabilities, but loses performance on closely related benign topics and some conversational ability.

  • Forget performance: Nearly random accuracy on WMDP-Bio and WMDP-Cyber is achieved by RMU on ZEPHYR-7B while MMLU accuracy is maintained.WMDP and MMLU scores are percentages, with 25% representing random performance.
  • Overall results: RMU unlearns WMDP-Bio and WMDP-Cyber while maintaining performance on MMLU, outperforming baselines that fail on either forgetting or retention.The evaluation uses WMDP for hazardous knowledge, MMLU for general knowledge, and MT-Bench for conversational fluency.
  • Probing: Linear probes achieve only slightly better than random accuracy, suggesting RMU substantially alters rather than merely masks unlearned hazardous information.The probing evaluation tests whether hidden knowledge remains recoverable from model representations.
  • MMLU retention: RMU preserves overall MMLU, college biology, and college computer science performance but greatly reduces scores on virology and computer security.The authors attribute this precision gap partly to using Wikitext rather than subject-specific retain data.
  • Conversational ability: MT-Bench decreases by 0.23 on ZEPHYR-7B, 0.06 on YI-34B, and 0.13 on MIXTRAL-8X7B after RMU.The scores are measured out of 9, and the ZEPHYR-7B degradation is the largest reported decline.
  • Robustness: After GCG optimization, unlearned YI-34B models output gibberish after 2,500 steps, whereas base models answer in fewer than 50 steps.This qualitative experiment provides a signal that RMU remains resilient under optimization pressure.

6 Discussion

WMDP-based unlearning is presented as a risk-mitigation component for both closed- and open-source models, with structured access addressing some remaining access and dual-use concerns.

  • 6.1 How WMDP Mitigates Risk: Unlearning on WMDP mitigates risk for both closed-source and open-source models.
  • 6.1 How WMDP Mitigates Risk: For closed-source models, unlearning removes hazardous knowledge before serving and can limit the value of jailbreaks or malicious API finetuning.Even if a model is jailbroken, the authors state that an unlearned model lacks the knowledge needed to empower malicious users.
  • 6.1 How WMDP Mitigates Risk: For open-source models, unlearning before release can limit adversaries’ ability to repurpose models out of the box.
  • 6.1 How WMDP Mitigates Risk: WMDP complements structured API access, which can provide ordinary users with a safety-mechanism-protected model while reserving the base model for approved users.Structured access requires verifying customers’ identities and intentions before privileged interactions.

7 Conclusion

The paper introduces WMDP as a filtered benchmark for evaluating hazardous knowledge and RMU as a machine-unlearning method. RMU reduces hazardous knowledge while preserving general performance, but its effects on closely related fields show that greater precision remains necessary.

  • 7 Conclusion: WMDP evaluates potential malicious use in LLMs using expert-developed biology, cybersecurity, and chemistry questions filtered for sensitive information.
  • 7 Conclusion: RMU removes hazardous knowledge without significantly compromising general model performance and also generalizes to a private sensitive dataset.
  • 7 Conclusion: RMU reduces accuracy on closely related fields, including introductory virology and computer security, demonstrating the need for more precise unlearning.

A Dataset

The dataset section describes WMDP’s threat-model-based categories, conservative inclusion policy, and filtering procedures for biosecurity, cybersecurity, and chemical security content.

  • A Dataset: The biosecurity questions primarily test knowledge for developing or using existing resources to acquire bioweapons, including tacit laboratory troubleshooting knowledge.
  • A Dataset: WMDP-Chem categories cover general knowledge, synthesis, sourcing or procurement, purification, analysis or verification, deployment mechanisms, and bypass mechanisms.
  • A Dataset: Retain corpora are constructed separately for general biology and computer-security topics rather than simply duplicating the forget corpora.The biology retain set samples general-biology papers while excluding forget-set papers and WMDP-related keywords.
  • A Dataset: The cybersecurity forget corpus is assembled from GitHub passages selected with topic keywords and further filtered by MIXTRAL using a score threshold of 9 or higher.
  • A Dataset: The cybersecurity filter distinguishes offensive procedural material sufficient to perform operations from material lacking enough clear technical detail for execution.

B Experiments

The experiments compare base models and unlearning methods using WMDP and MMLU question-answer accuracy plus MT-Bench fluency, under specified evaluation procedures.

  • B Experiments: The experiments also cover implementation details, robustness and relearning evaluations, unlearning baselines, and how RMU manipulates representations.
  • B Experiments: Table 2 compares base models and unlearning methods on WMDP and MMLU question-answer evaluation and MT-Bench fluency.WMDP and MMLU scores are reported in percentage points, and methods remove WMDP-Bio and WMDP-Cyber.
  • B Experiments: MMLU and WMDP use the default lm-evaluation-harness zero-shot format, selecting the highest logit among answer choices A, B, C, and D.HuggingFace models use lm-evaluation-harness v0.4.2, while GPT-4 is manually evaluated with the same prompt template.
  • B Experiments: The section provides an example biology multiple-choice question about a poliovirus IRES mutation.
  • B Experiments: MT-Bench is evaluated in single-evaluation mode with gpt-4-0613 as the judge.

B.3 Robustness Evaluation

The robustness evaluation tests whether adversarial finetuning or jailbreak attacks can recover hazardous knowledge after RMU unlearning. RMU blocks the tested direct prompts but does not prevent hazardous performance from being recovered through finetuning.

  • Jailbreak robustness: RMU was evaluated against jailbreak attacks on unlearned and base YI-34B models using GCG with a 20-token adversarial suffix.For cybersecurity, the evaluation used free-form generation rather than multiple-choice answers.
  • Jailbreak robustness: After 2,500 optimization steps on RMU-unlearned YI-34B, cybersecurity and biosecurity outputs remained malformed or incomplete rather than providing the requested hazardous instructions.The reported cybersecurity completion was nonsensical, while the biosecurity output repeated fragments without supplying a complete answer.
  • Jailbreak robustness: 50 GCG steps successfully jailbroke the base model on both cybersecurity and biosecurity prompts, producing hazardous completions after direct refusals.The base model initially refused both prompts when asked directly.
  • Representation effects: RMU increases activation norms on hazardous data while preserving norms on benign data, consistent with its representation-based unlearning mechanism.The forget loss causes hazardous-data activations to blow up after about 200 steps, while the retain loss keeps benign-data activations roughly aligned with the frozen model.
  • Finetuning robustness: Finetuning on the cybersecurity forget set recovers WMDP-Cyber performance, showing that RMU does not mitigate this open-source relearning threat.The experiment considers publicly released weights that adversaries can finetune.

C MMLU Subset Unlearning Benchmark

The auxiliary benchmark evaluates whether unlearning selected MMLU subjects reduces performance on forgotten topics while preserving unrelated capabilities. Physics and Economics show relatively targeted degradation, whereas Law causes substantial retain-set degradation.

  • Benchmark settings: The benchmark provides Economics, Law, and Physics settings that forget selected MMLU subjects while retaining the remaining categories.The forgotten subjects are macroeconomics and microeconomics; international and professional law; and three physics categories.
  • Benchmark design: The settings target relatively separate topics with more than 1,000 forget-set questions, enabling evaluation against related retain categories and aggregate MMLU performance.Table 3 reports forget, retain, and overall MMLU accuracy before and after RMU.
  • Results: Physics unlearning significantly reduces College Physics and High School Physics performance while causing only small variation on aggregate MMLU and related mathematics areas.The reported pattern is also described as holding similarly for the Economics setting.
  • Results: Law unlearning significantly degrades retain-set performance, demonstrating the need for methods with greater unlearning precision.The table caption likewise identifies retain-performance decline for some categories.
  • Implications: The broader discussion notes that unlearning can reduce useful defensive capabilities, especially when knowledge is dual-use in cybersecurity.Biosecurity may contain more primarily offensive knowledge that can be removed with less effect on defensive capabilities.

D.1 Limitations

The benchmark’s multiple-choice format, static content, and API-focused threat model limit what its scores establish about malicious-use risk. The authors therefore frame WMDP as requiring complementary evaluations and note that open-weight relearning remains unaddressed.

  • Measurement limits: Four-way multiple-choice questions may miss hazards that emerge only in larger end-to-end evaluations or distinguish memorization from genuine understanding.This concern is especially relevant for biological questions drawn from open-access papers likely present in training data.
  • Measurement limits: WMDP tests retained hazardous knowledge but not whether a model will reveal it helpfully and promptly during planning or execution of a nefarious attack.The authors suggest post-cutoff or difficult-to-search question sources as possible future remedies.
  • Benchmark scope: WMDP is static and cannot anticipate changing cyber and biological threats, so using it alone after unlearning may create a false sense of model safety.The authors recommend complementary benchmarks, including open-ended conversation evaluations.
  • Threat-model boundary: WMDP focuses on API-access models, while malicious actors can trivially reintroduce unlearned information into publicly downloadable model weights.The work would leave this risk unaddressed if open-source models reach capabilities comparable to closed-source models.

E.2 Safety-Capabilities Balance

The paper presents unlearning as a safety intervention that removes specific hazardous capabilities rather than improving general capabilities. It also warns that the intervention can reduce beneficial defensive uses and should be combined with other safeguards.

  • Safety-capabilities balance: Unlearning removes specific model capabilities while improving inherent model safety, rather than improving general capabilities.The paper frames this as the relevant safety-capabilities balance.
  • Trade-offs: WMDP may inadvertently provide a roadmap for malicious use, so the authors apply extensive sensitive-information mitigation to reduce that risk.The benchmark’s measurement purpose can therefore create a tension between evaluation utility and information hazard.
  • Trade-offs: Unlearning may reduce defensive or beneficial applications, so it should be complemented by other interventions that reduce risk.This concern is especially salient for dual-use cybersecurity knowledge.
Loading 2403.03218v7…