PaperlayerView exact revision on arXiv

Source-linked AI summary

Tamper-Resistant Safeguards for Open-Weight LLMs

Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Andy Zou, Dawn Song, Bo Li, Dan Hendrycks, Mantas Mazeika

arXiv:2408.00761v4cs.LGcs.AIcs.CL

TL;DR

Open-weight LLM safeguards can be removed by fine-tuning model weights, creating a need for tamper-resistant protection. The paper introduces TAR, an adversarial-training method that embeds such safeguards, and finds stronger resistance to extensive attacks while largely preserving benign capabilities.

  • Problem

    Open-weight safeguards are vulnerable to weight-tampering attacks that can remove protections after a few fine-tuning steps, threatening safer deployment.

    Source: §1 · pp. 1–2

  • Method

    TAR combines initial safeguarding with adversarial training inspired by meta-learning to optimize tamper resistance while retaining capabilities.

    Source: §4 · p. 4, §4.2 · p. 6

  • Results

    TAR is more robust than prior methods under extensive red teaming, resisting hundreds of fine-tuning steps while preserving benign capabilities.

    Source: §1 · p. 2, §5.1 · p. 8, §5.2 · p. 9

  • Takeaways & Limitations

    The results show that robust tamper-resistant safeguards for open-weight LLMs are tractable and can support safer deployment.

    Source: §6 · p. 9

    TAR evaluates SFT weight-tampering adversaries rather than explicitly optimizing against input-space jailbreaks.

    Source: §3.1 · p. 3, §5.3 · p. 9

Abstract

from arXiv · show

Rapid advances in the capabilities of large language models (LLMs) have raised widespread concerns regarding their potential for malicious use. Open-weight LLMs present unique challenges, as existing safeguards lack robustness to tampering attacks that modify model weights. For example, recent works have demonstrated that refusal and unlearning safeguards can be trivially removed with a few steps of fine-tuning. These vulnerabilities necessitate new approaches for enabling the safe release of open-weight LLMs. We develop a method, called TAR, for building tamper-resistant safeguards into open-weight LLMs such that adversaries cannot remove the safeguards even after hundreds of steps of fine-tuning. In extensive evaluations and red teaming analyses, we find that our method greatly improves tamper-resistance while preserving benign capabilities. Our results demonstrate that progress on tamper-resistance is possible, opening up a promising new avenue to improve the safety and security of open-weight LLMs.

1 Introduction

Open-weight LLM safeguards can be removed by fine-tuning model weights, creating safety and liability concerns. TAR uses adversarial training to build safeguards that resist diverse tampering attacks while preserving general capabilities.

1 Introduction

  • Open-weight LLMs offer broad access and research benefits, but their growing capabilities raise concerns about malicious repurposing.The paper frames tamper-resistant safeguards as relevant to model developers and regulators addressing the dual-use dilemma.Source: §1 · pp. 1–2
  • Weight-editing attacks can defeat refusal and preference-based safeguards after only a handful of fine-tuning steps.Because open-weight adversaries have direct access to model weights, safeguards designed for API-served models are especially vulnerable.Source: §1 · p. 1
  • Tampering vulnerabilities may expose developers to reasonable-care concerns when malicious actors can customize models for critical harm.The paper therefore describes stronger safeguards against weight tampering as urgent.Source: §1 · p. 2
  • TAR applies adversarial training and meta-learning to make safeguards robust against tampering while retaining general capabilities.The method identifies the tamper-resistance loss, train-time adversaries, and two-stage safeguarding procedure as crucial factors.Source: §1 · p. 2
  • TAR safeguards for unlearning and refusal resist fine-tuning attacks of hundreds of steps in extensive red-team evaluations.The evaluation compares TAR with prior methods and includes 26 test-time adversaries.Source: §1 · p. 2

2 Related Work

Prior safeguards often fail against jailbreaks, adaptive attacks, or weight-editing fine-tuning. Existing robust-safeguard work has not established tamper-resistant methods for autoregressive LLMs under broad evaluations.

Adversarial attacks on LLMs

  • Fine-tuning-based safeguards can break under jailbreak attacks or only a handful of steps on uncensored data.This motivates security concerns about post-deployment alignment.Source: §2 · p. 3

LLM safeguards

  • Existing unlearning methods are not robust to adversaries with model-weight access, while jailbreak defenses remain vulnerable to adaptive attackers.Prior approaches include system-level defenses, adversarial training, and several unlearning techniques.Source: §2 · p. 3

Robust safeguards

  • TAR is compared with 12 baseline safeguards using tamper-resistance across up to 26 fine-tuning attacks and general capability on MMLU.Tamper-resistance averages normalized WMDP Biosecurity, Chemical Security, and Cybersecurity error.Source: Figure 2 · p. 3
  • No prior method had been proposed for autoregressive LLMs that is robust to tampering attacks.Earlier robust-unlearning work focused on image classification, bidirectional BERT-style models, vision classifiers, and diffusion models.Source: §2 · p. 3
  • Concurrent refusal-defense methods use perturbation losses, harmful-generation losses, or randomized harmful representations, but evaluations report limited robustness.The paper finds that two compared approaches lack robustness to its tampering attacks.Source: §2 · p. 3

3 Tamper-Resistant Safeguards

The paper defines tampering as compute-bounded weight modification and evaluates safeguards by post-attack safety together with retained capabilities. Its framework covers knowledge restriction and harmful-request refusal under extensive red teaming.

3.1 Threat Model

  • The threat model gives adversaries unrestricted access to safeguarded weights but bounds their compute, such as 500 fine-tuning steps.The adversary minimizes post-attack safety, while the defender seeks high safety and capability before and after the attack.Source: §3.1 · p. 3

3.2 Problem Definition and Metrics

  • Tamper-resistance requires high post-attack safety across a broad range of strong test-time attacks while preserving general capabilities.The framework rejects safeguards that achieve robustness by overwriting the model with unusable noise.Source: §3.2 · p. 4
  • Weaponization knowledge restriction uses WMDP hazardous-domain questions as a forget set and complementary MMLU tasks as a retain set.WMDP contains 3,668 questions spanning biosecurity, chemical security, and cybersecurity.Source: §3.2 · p. 4
  • Harmful-request refusal measures safety through the complement of jailbreak attack success rate and capabilities through MT-Bench conversational performance.HarmBench supplies static red-team cases, while MT-Bench uses an LLM judge for multi-turn evaluation.Source: §3.2 · p. 4

3.3 Red Teaming

  • Red teaming uses up to 26 adversaries with varied compute budgets, datasets, hyperparameters, and optimization settings, including unseen attacks.This stress testing supports robustness measurement and incremental evaluation of defenses.Source: §3.3 · p. 4

4 Safeguard Tamper-Resistance Training

TAR combines initial safeguarding with adversarial tamper-resistance training inspired by meta-learning. Its outer-loop objectives counter fine-tuning attacks while a retain loss preserves benign representations and capabilities.

4 Safeguard Tamper-Resistance Training

  • TAR directly trains safeguards against tampering attacks using an adversarial-training and meta-learning procedure.The method is designed for a train-time attack set and aims to strengthen safeguards against weight modification.Source: §4 · p. 4, §4.2 · p. 5
  • The training algorithm samples attacks and proxy data, accumulates tamper-resistance gradients, and combines them with a retain gradient over outer steps.Its inputs include train-time adversaries, capability and safety proxy datasets, attack samples, and loss scales.Source: §4 · p. 5

4.1 Model Safeguarding

  • The two-stage method first applies an existing safeguard, then trains its parameters for tamper resistance.Initial safeguards may use circuit breaking, constrained gradient ascent, RLHF, or DPO and need not themselves be tamper-resistant.Source: §4 · p. 5, §4.1 · p. 5

4.2 Tamper-Resistance Training

  • TAR approximates gradients through several-step SFT attacks with a straight-through estimator, adapting first-order MAML to seek poor adversary optimality.Unlike standard meta-learning, the outer loop seeks an initialization far from optimality for adversaries’ test distributions.Source: §4.2 · p. 5
  • Negative entropy outlasts negative cross-entropy as the weaponization tamper-resistance loss, thwarting the inner-loop adversary across its trajectory.Cross-entropy maximization affects the adversary early but permits later recovery, whereas entropy maximization eventually satisfies the objective at all inner-loop steps.Source: Figure 3 · p. 6, §4.2 · p. 6
  • The outer objective minimizes a tamper-resistance loss after attack while retaining capability performance through a separate retain loss.The retain loss uses residual-stream representations and is weighted alongside the tamper-resistance loss.Source: §4.2 · p. 6

5 Experiments

The experiments evaluate TAR for weaponization knowledge restriction and harmful request refusal against fine-tuning tampering attacks. Across both settings and broad red-team analyses, TAR improves post-attack robustness while largely preserving benign capabilities, with important sensitivity to attack distributions and some capability trade-offs.

5 Experiments

  • The experiments use separate train-time and test-time adversaries, with weaponization evaluations measuring WMDP recovery and benign MMLU preservation and refusal evaluations measuring HarmBench ASR and MT-Bench.Weaponization uses 64-step train-time attacks; refusal uses 64-step SFT attacks and evaluates post-attack jailbreak success.Source: §5 · pp. 6–7, §5.1 · p. 7, §5.2 · p. 8

5.1 Weaponization Knowledge Restriction

  • TAR maintains low post-attack forget accuracy across biosecurity, chemical security, and cybersecurity, while retaining high benign accuracy; it reduces retain accuracy by 10.6% on average.The comparison covers 26 fine-tuning attacks and finds prior safeguards substantially more brittle under tampering.Source: Table 1 · p. 7, §5.1 · p. 8

5.2 Harmful Request Refusal

  • TAR’s harmful-request refusal safeguard reaches 61.7% post-tampering attack success, versus above 77% for several baselines, while reducing MT-Bench by 1.7.The reported result compares Refusal Training, Representation Rerouting, and R2D2 baselines that resisted jailbreaks before tampering but degraded after attack.Source: §5.2 · p. 9, Table 2 · p. 9

5.3 Analysis

  • Red teaming with 26 biosecurity adversaries and 24 chemical-security and cybersecurity adversaries shows TAR greatly increases tamper-resistance for nearly all adversaries compared with prior safeguards.Attacks vary optimizer, optimization steps, learning rate and schedule, fine-tuning dataset, batch size, and overall fine-tuning method.Source: Figure 4 · p. 8, §5.3 · p. 9
  • TAR’s robustness varies under parameter-efficient fine-tuning and some out-of-distribution learning-rate attacks, showing sensitivity to the adversary distributions sampled during training.The analysis emphasizes extensive red teaming and reports that targeted patching of vulnerabilities is possible.Source: §5.3 · p. 9

6 Conclusion

The paper introduces TAR for tamper-resistant safeguards and applies it to weaponization knowledge restriction and harmful refusal training. Its comparisons and rigorous red-teaming evaluation support the conclusion that open-weight tamper resistance is tractable.

6 Conclusion

  • TAR is presented as the first method in this study to remain robust under its rigorous red-teaming evaluation across weaponization restriction and harmful refusal settings.The paper frames this result as evidence that progress on open-weight tamper resistance is possible.Source: §6 · p. 9
  • The authors argue that tamper-resistant safeguards could support continued deployment of robust open-weight LLMs while addressing malicious-use risks and regulatory alignment.This conclusion connects the method’s demonstrated robustness to the broader deployment and governance context described by the authors.Source: §6 · p. 9

A Limitations

The method remains robust across many tampering attacks, but its scope and practical usability are limited by attack coverage, scaling costs, retain-performance trade-offs, and the possibility of eventual circumvention.

A Limitations

  • The evaluation focuses on supervised fine-tuning attacks, leaving broader open-weight tampering techniques for future red-teaming.The authors identify diverse tampering methods beyond supervised fine-tuning as an important unresolved scope boundary.Source: A Limitations · p. 14
  • Scaling the method to larger models poses computational challenges requiring further optimization.Source: A Limitations · p. 14
  • Low post-attack forget accuracy can coincide with low post-attack retain accuracy, creating a benign fine-tuning usability concern.Benign users must avoid contaminating fine-tuning data with forget-set content, and preserving high retain accuracy remains future work.Source: A Limitations · p. 14
  • Tamper-resistance raises adversary costs but can eventually be circumvented, so it cannot fully mitigate malicious AI-use risks alone.Released open-weight models cannot be unreleased, leaving compromised defenses permanently vulnerable.Source: A Limitations · p. 14

B Author Contributions

  • The author-contribution passage records project leadership, implementation, experimentation, dataset curation, evaluation, infrastructure, and advising roles.Source: B Author Contributions · p. 14

C.1 Note on Updated Implementation

The updated implementation corrects data and sampler issues while showing that TAR's robustness persists across many shuffling orders and stronger attacks. The method combines Random Mapping, task-specific tamper-resistance losses, efficient trajectory sampling, and benign fine-tuning experiments with targeted vulnerability patching.

C.1 Note on Updated Implementation

  • Corrected data contamination and sampler issues remove an observed input-space vulnerability and clarify variability in long attack runs.After correction, robustness remains consistent across multiple shuffling orders for 500 fine-tuning steps, while 1,000-step outcomes can vary.Source: §C.1 · p. 15

C.2 Initial Weaponization Knowledge Restriction Safeguard

  • Random Mapping restricts hazardous-domain knowledge by mapping target residual-stream activations to hashed random vectors while preserving retain-set performance.The raw safeguard is evaluated as an ablation without adversarial training.Source: §C.2 · pp. 15–16

C.3 Designing the Tamper-resistance Loss

  • Entropy-based tamper-resistance loss is designed to flatten adversary loss at a high value, and DPO loss preserves refusal preferences after harmful fine-tuning.Larger simulated inner-loop lengths improve robustness to longer attacks, motivating efficient sampling.Source: §C.3 · p. 16

C.4 Efficiently Sampling Fine-tuning Attacks

  • A single fine-tuning trajectory supplies multiple increasing-length attack trajectories, with subsampling and shared held-out batches reducing computational cost and gradient variance.Source: §C.4 · p. 16

C.5 Implementation Details and Resource Requirements

  • TAR training uses distributed sharding and gradient accumulation to make billion-parameter inner-loop optimization feasible on multi-GPU hardware.The implementation uses FSDP, ZeRO Stage 3, CPU offloading, and sharded-gradient accumulation.Source: §C.5 · pp. 16–17

D.1 Benign Fine-Tuning

  • 5.0 percentage points: benign economics accuracy improves without recovering significant hazardous knowledge after fine-tuning TAR models.The evaluation uses economics data and corresponding hazardous-domain tests before and after benign fine-tuning.Source: §D.1 · p. 17, Table 3 · p. 17

D.2 Targeted Patching of Vulnerabilities

  • Nearly 10.0 percentage points: including the retain-to-weaponization adversary improves average tamper-resistance across 26 adversaries.The same train-time adversary improves robustness to two 500-step test-time variants by nearly 30.0 percentage points on average.Source: Table 4 · p. 17, §D.2 · p. 18

D.3 Test-time Generalization Experiments

TAR generalizes beyond its 64-step training attacks: against a 1,000-step biosecurity attack, its adversary loss stays high and plateaus, while LLMU recovers quickly. Longer inner loops further extend this robustness, with K = 64 balancing robustness and efficiency.

D.3 Test-time Generalization Experiments

  • Loss remains above 7 for all 1,000 test-time steps and plateaus after 200 steps under TAR, demonstrating generalization beyond 64-step training attacks.The adversary entropy approaches maximum during the initial attack steps, matching the negative-entropy training objective.Source: §D.3 · p. 18, Figure 6 · p. 18
  • LLMU loss reaches the recovery region in under 20 steps, corresponding to recovered forget-set performance.Source: §D.3 · p. 18, Figure 6 · p. 18
  • Longer TAR inner-loop lengths produce later loss drop-offs and greater robustness during 1,000-step attacks.The observed trend is consistent with increasing test-time generalization as inner-loop length increases.Source: §D.3 · p. 18, Figure 7 · p. 18
  • K = 64 provides significant robustness across the evaluated adversaries while balancing computational efficiency.This creates a trade-off in which additional training compute can be exchanged for robustness.Source: Figure 7 · p. 18, §D.3 · p. 19

D.4 Ablations

TAR’s initial safeguard and adversarial training are both critical to high tamper-resistance, while increasing the tamper-resistance loss improves robustness at some capability cost. During training, TAR also learns to satisfy complex tamper-resistance objectives after fine-tuning.

D.4 Ablations

  • TAR’s initial safeguard and adversarial training are both critical components for achieving high tamper-resistance.Source: Table 5 · p. 19
  • Nearly 26 percentage points: adding TAR adversarial training reduces Post-Attack Forget accuracy from 61.6 to approximately 35.6 after the initial safeguard alone.The initial Random Mapping safeguard has near-random Pre-Attack Forget accuracy of 27.3 and remains vulnerable to fine-tuning attacks.Source: §D.4 · p. 19, Table 5 · p. 19
  • 19.2 percentage points: including the initial safeguard lowers Pre-Attack Forget-set accuracy compared with adversarial training without that safeguard.Source: §D.4 · p. 19
  • Nearly 10.0%: increasing λTR from 1.0 to 4.0 improves tamper-resistance while reducing Retain MMLU accuracy from 62.5 to 54.7.Post-Attack Forget accuracy decreases from 39.9 to 35.2 as λTR increases, with λretain fixed at 1.0.Source: Table 6 · p. 19, §D.4 · p. 19

D.5 DPO Tamper-Resistance during TAR

  • The outer-loop DPO loss steadily decreases while average inner-loop refusal win-rate increases over 100 outer-loop steps, showing TAR can satisfy complex tamper-resistance losses after fine-tuning.Source: §D.5 · p. 20

E.1 Weaponization Domain Proxy Dataset Details

The experiments use domain-specific proxy datasets and sampled fine-tuning adversaries for biosecurity, chemical security, cybersecurity, and harmful-request refusal. Held-out data supplies tamper-resistance losses, while additional benign fine-tuning improves refusal-model capabilities.

E.1 Weaponization Domain Proxy Dataset Details

  • Weaponization-domain experiments use Biosecurity, Chemical Security, and Cybersecurity forget datasets alongside a mixed Pile-bio and Magpie-Align retain set.Biosecurity includes Pile-bio, Camel-bio, and held-out hazardous-biology data; Chemical Security uses a private hazardous-chemistry dataset; Cybersecurity uses filtered CTF writeups.Source: §E.2 · p. 20, §E.1 · p. 20

E.2 Train-time Settings and Adversaries

  • Train-time weaponization adversaries simulate forget-set, retain-set, and retain-to-forget SFT trajectories across domains, with six sampled adversaries per domain.Learning rates are sampled from {2 × 10^-5, 4 × 10^-5}; Chemical Security uses λTR = 3.0.Source: §E.2 · pp. 20–21
  • Held-out 20% splits compute tamper-resistance losses, while adversary 80% splits generate sampled attacks; minibatches prevent reuse of a single held-out dataset.Source: §E.2 · p. 21
  • Harmful-refusal adversaries perform 64 SFT steps on rejected Anthropic-HH-RLHF completions while TAR computes a DPO tamper-resistance loss on modified refusal completions.Learning rates vary across {2 × 10^-6, 2 × 10^-5, 4 × 10^-5}, and base-model DPO log-probabilities are precomputed.Source: §E.2 · pp. 21–22
  • An additional 100 Magpie-Align SFT steps improve benign-capability performance for the TAR refusal model.Source: §E.2 · p. 22

F.1 Weaponization Knowledge Restriction

Weaponization safeguards are evaluated against diverse 500-step fine-tuning attacks across biosecurity, chemical security, and cybersecurity, while harmful-refusal safeguards face held-out toxicity attacks. The appendix also specifies entropy- and posterior-based objectives and baseline implementations.

F.1 Weaponization Knowledge Restriction

  • Weaponization evaluations use 500-step attacks varying datasets, optimizers, learning rates, schedules, batch sizes, and full-parameter versus PEFT fine-tuning.Source: §F.1 · pp. 22–23, Table 9 · p. 23
  • 24 attacks are evaluated for Chemical Security and Cybersecurity because unseen-forget-distribution adversaries are excluded without equally high-quality held-out datasets.Biosecurity additionally evaluates an OOD Forget set unseen during tamper-resistance training.Source: §F.1 · p. 22
  • The test-time R→F attack uses Forget-set SFT for 40% of steps followed by Retain-set SFT for 60%, and previously broke intermediate TAR versions.Source: §F.1 · p. 22

F.2 Harmful Request Refusal

  • Harmful-refusal red teaming uses five 10-epoch SFT attacks on 541 harmful-compliance conversations from Toxic-DPO v0.2.Source: §F.2 · p. 24, Table 10 · p. 24

F.2.1 Additional Harmful Request Refusal Results

  • TAR is reported as more robust after tampering than Representation Rerouting and refusal-trained Llama-3-8B-Instruct while maintaining comparable MT-Bench performance.The comparison averages Post-Attack HarmBench ASR across five attacks.Source: Table 11 · p. 25

G.1 Weaponization Knowledge Restriction

  • Max Entropy and Min Posterior suppress forget-set information while applying standard cross-entropy on the retain set, with Min Posterior masking target logits below τ.Max Entropy minimizes divergence from a uniform vocabulary distribution; Min Posterior minimizes -log(1 − P(label)).Source: §G.1 · p. 25
  • Baseline implementations include RMU, a memory-efficient LLMU modification, Representation Rerouting, R2D2, and RepNoise.Source: §G.1 · pp. 25–26, §G.2 · p. 26

G.3 Additional Baseline Comparisons

Additional autoregressive extensions of prior robust-safeguard methods do not provide a consistently usable alternative: SOPHON-AR fails to converge sufficiently, while MLAC-AR has uneven robustness and reduced retain capabilities.

G.3 Additional Baseline Comparisons

  • MLAC-AR shows divergent performance across hyperparameters and, after clamping its cross-entropy loss, still lacks sufficient benign capabilities and uniform tamper-resistance.Source: §G.3 · pp. 26–27
  • SOPHON-AR does not converge well enough to produce a usable Pre-Attack model despite extensive tuning.Source: Table 12 · p. 26, §G.3 · p. 27

Related papers

Loading 2408.00761v4…