PaperlayerView exact revision on arXiv

Source-linked AI summary

TamperBench: Systematically Stress-Testing LLM Safety Under Fine-Tuning and Tampering

Saad Hossain, Tom Tseng, Punya Syon Pandey, Samanvay Vajpayee, Matthew Kowal, Nayeema Nonta, Samuel Simko, Stephen Casper, Zhijing Jin, Kellin Pelrine, Sirisha Rambhatla

arXiv:2602.06911v2cs.CRcs.AI

TL;DR

Open-weight LLM safeguards can be undermined by tampering, yet varied attacks and evaluation methods make resistance difficult to compare. TamperBench standardizes attacks, defenses, and safety-and-utility evaluation, and its evaluations find that jailbreak-tuning is typically most severe and current alignment-stage defenses often fail under attack sweeps.

  • Problem

    Tampering can undermine safety safeguards in open-weight LLMs as their capabilities grow, creating risks from unsafe modifications.

    Source: §1 · p. 1

  • Method

    TamperBench combines weight- and representation-space attacks, alignment-stage defenses, systematic attack–model hyperparameter sweeps, and standardized safety and capability evaluations.

    Source: §1 · p. 2

  • Results

    Across 21 open-weight LLMs, jailbreak-tuning is typically the most powerful attack, susceptibility varies across models, and every tested model could be tampered to produce high harmfulness while preserving utility.

    Source: §6 · p. 9

  • Takeaways & Limitations

    The results indicate that existing models and defenses do not provide durable protection against tampering, and TamperBench offers an extensible foundation for further evaluation.

    Source: §6 · p. 9

    The study focuses on refusal-based defenses and primarily on models with 0.6–8B parameters, with broader large-model coverage limited.

    Source: §3.1 · p. 4, §5 · p. 9

Abstract

from arXiv · show

As increasingly capable open-weight large language models (LLMs) are deployed, improving their tamper resistance against unsafe modifications, whether accidental or intentional, becomes critical to minimize risks. However, there is no standard approach to evaluate tamper resistance. Varied datasets, metrics, and tampering configurations make it difficult to compare safety, utility, and robustness across different models and defenses. To address this, we introduce TamperBench, the first unified framework to systematically evaluate the tamper resistance of LLMs. TamperBench (i) curates a repository of state-of-the-art weight-space fine-tuning attacks, latent-space representation attacks, and alignment-stage defenses; (ii) enables realistic adversarial evaluation through systematic hyperparameter sweeps per attack-model pair; and (iii) provides both safety and utility evaluations. We use TamperBench to evaluate 21 open-weight LLMs, including defense-augmented variants, across nine tampering threats using standardized safety and capability metrics with hyperparameter sweeps per model-attack pair. The results provide insights including effects of post-training on tamper resistance, that jailbreak-tuning is typically the most severe attack, and that current alignment-stage defenses largely fail to withstand attack sweeps. Code is available at https://github.com/criticalml-uw/TamperBench.

1 Introduction

Tampering can undermine open-weight LLM safeguards, yet inconsistent attacks, threat models, and metrics make tamper-resistance evaluations difficult to compare. TamperBench unifies attacks, defenses, and safety-and-utility evaluation, then uses systematic sweeps to assess models and defenses.

Motivation

  • Prior studies vary in attacks, threat models, and metrics, making it difficult to assess defense promise and the precautions needed when releasing highly capable open-weight models.The cited review reports that studies often test thousands or tens of thousands of adversarial fine-tuning steps, whereas second-party red-teaming assesses several hundred.Source: §1 · pp. 1–2

Contributions

  • TamperBench is an extensible benchmark and toolkit combining tampering attacks, alignment-stage defenses, and standardized evaluation protocols for open-weight LLMs.It covers weight-space modifications and inference-time latent-space perturbations, including benign and adversarial threats.Source: §1 · p. 2
  • Systematic hyperparameter sweeps for attack–model pairs support adversarial comparisons, while standardized safety-refusal and capability benchmarks assess harmfulness and utility after tampering.Examples include StrongREJECT [60] and MMLU-Pro [24].Source: §1 · p. 2
  • Across 21 open-weight LLMs and nine attacks, jailbreak-tuning is typically the most severe attack, while alignment-stage defenses are largely unsuccessful under systematic stress testing.The direction of out-of-the-box resistance differences between base and post-trained variants reverses between Llama-3 and Qwen3.Source: §1 · p. 2

2 Background and Related Works

Prior work describes multiple routes for tampering and defense, but existing frameworks leave gaps in attack coverage, extensibility, and comparability. TamperBench addresses these gaps with a unified evaluation toolkit.

2.1 LLM Vulnerabilities

  • Fine-tuning can weaken safeguards even when benign, while harmful fine-tuning, backdoors, data poisoning, and representation-space attacks provide additional tampering routes.Parameter-efficient methods such as LoRA [28] and adapters [43] [55] [68] make modifications accessible.Source: §2.1 · p. 3

2.2 Tampering Defenses

  • Alignment-stage defenses strengthen safety training before release; fine-tuning-stage defenses alter downstream adaptation, and post-tuning defenses repair misalignment after tampering.The latter two categories presuppose centralized control over fine-tuning; alignment-stage defenses remain applicable to open-weight and API-based deployments.Source: §2.2 · p. 3

2.3 Existing Frameworks

  • Prompt-based red-teaming frameworks do not systematically cover weight-space tampering or fine-tuning, while existing tamper-resistance toolkits have limited extensibility and strategy coverage.TamperBench unifies attacks, defenses, and evaluation metrics for reproducible, comparable assessment across weight- and latent-space manipulations.Source: §2.3 · pp. 3–4, Figure 3 · p. 4

3 TamperBench Framework

TamperBench evaluates tampering in a worst-case open-weight setting, measuring safeguard weakening alongside retained capability. It combines attack and defense methods with safety, utility, and extensible toolkit components.

3.1 Threat Model

  • The threat model assumes attackers have full white-box access and seeks maximum damage across attacks and hyperparameter configurations, while defenders aim to preserve benign capabilities.Successful tampering is assessed by increased compliance with unsafe prompts while general capabilities remain largely preserved; this utility constraint guards against overfitting to safety metrics.Source: §3.1 · p. 4

3.2 Tamper Attack Suite

  • The attack suite covers benign full fine-tuning and LoRA, malicious harmful and multilingual fine-tuning, backdoor and jailbreak-tuning methods, and representation-space embedding attacks.Backdoor, style-modulation, and competing-objectives jailbreak-tuning use 100% harmful data in this benchmark setting.Source: §3.2 · p. 4

3.3 Tamper-Resistant Defenses

  • TamperBench re-implements seven alignment-stage defenses: Booster, CRL, CTRL, RSN-Tune, SDD, T-Vaccine, and TAR.These methods allow users to harden models from scratch rather than rely on pre-released weights.Source: §3.3 · pp. 4–5

3.4 Utility Evaluation

  • The toolkit supports capability benchmarks across knowledge, reasoning, instruction following, conversation, mathematics, biology, and coding; experiments use a 560-sample MMLU-Pro subset.When MMLU-Pro drops by at least 10%, several other capability benchmarks also drop by at least 10% in 79–89% of cases.Source: §3.4 · p. 5

3.5 Safety Evaluation

  • Experiments use StrongREJECT as the representative harmfulness metric, with increases correlating with increases on JailbreakBench, Policy-Eval, and SafetyGap.The reported correlations are ρ=0.71, ρ=0.54, and ρ=0.66, respectively.Source: §3.5 · p. 5

3.6 TamperBench Toolkit

  • The open-source toolkit supports multi-GPU checkpoint workflows for red-teaming, defenses, and independent safety and utility evaluation.Optuna-based sweeps, reproducibility caching, modular end-to-end or independent components, and extension with new methods are supported.Source: §3.6 · p. 5

4 Experiments

The experiments use systematic attack and defense sweeps to compare tamper resistance across 21 open-weight models, then examine larger models and defense hyperparameter tuning. They assess harmfulness alongside capability loss under utility constraints.

4.1 Global effects of tampering

  • SRmax exceeded 0.74 for all 21 models, with at least one attack substantially raising harmfulness while largely preserving original capabilities.The evaluation selects attack configurations subject to a ≤10% relative MMLU-Pro drop.Source: §4.1 · pp. 6–7, Figure 5 · p. 6

4.2 Attack-level risk profiles

  • Competing-objectives jailbreak-tuning achieved the highest StrongREJECT score in 14 of 21 models, with mean SR = 0.84.Benign full and LoRA fine-tuning also raised harmfulness with minimal capability loss; embedding attacks caused comparatively mild increases for 7–8B models.Source: §4.2 · p. 7

4.3 Comparing tamper resistance across model families

  • Post-trained Qwen3 variants generally had lower post-tampering harmfulness than base variants, while instruction-tuned Llama variants typically had higher scores than their base counterparts.Qwen3-4B had SRmal-avg = 0.76 vs. 0.84 and SRmax = 0.85 vs. 0.89 versus Qwen3-4B-Base; Mistral-7B-Instruct’s SRmax was 0.90, but it also began with higher untampered harmfulness than Mistral-7B-Base.Source: §4.3 · p. 7

4.4 Preliminary experiments with larger models

  • Under harmful LoRA fine-tuning, Qwen3-32B reached StrongREJECT 0.86 versus 0.81 for Qwen3-8B, and Llama-3-70B-Instruct reached 0.93 versus 0.83 for Llama-3-8B-Instruct.Qwen3-32B reached 0.88 versus 0.89 for Qwen3-8B under competing-objectives jailbreak-tuning; the larger models were comparably tamperable to their 8B counterparts.Source: §4.4 · p. 7

4.5 Evaluating alignment-stage defenses

  • Most alignment-stage defenses were unsuccessful; CTRL and T-Vaccine sharply reduced Llama capabilities, while harmfulness largely persisted after attacks.CTRL and T-Vaccine MMLU-Pro dropped to 0.07–0.09 from 0.38–0.46, yet SRmal-avg remained ≥0.70; TAR modestly reduced SRmal-avg on Llama-3-8B-Instruct but not the other two models.Source: §4.5 · pp. 7–8, Figure 6 · p. 8

4.6 Tuning Defense Hyperparameters via the Toolkit’s Sweep Protocol

  • Against strong LoRA attacks, default and swept defenses remained largely ineffective (SR ≥0.82 across configurations), but sweeps improved some weakened-attack outcomes.For example, Booster reduced post-attack SR from 0.56 to 0.07 on Qwen3-8B under the weakened attack; sweep-tuned defenses had smaller capability drops in 5 of 6 cases.Source: §4.6 · p. 8, Figure 7 · p. 9

5 Limitations

The evaluation is limited in the defense types and model scales it covers.

5 Limitations

  • The study covers robust-refusal defenses rather than ignorance-based techniques, and primarily evaluates 0.6–8B models; its 32B and 70B experiments are preliminary.Broader coverage of larger models is identified as future work.Source: §5 · p. 9

6 Conclusion and Future Directions

TamperBench provides a unified framework for comparing tampering across weight and representation spaces, and testing found every evaluated model susceptible to utility-preserving tampering. The appendix shows why realistic evaluations constrain utility: unconstrained harmfulness optimization can coincide with capability loss.

6 Conclusion and Future Directions

  • All 21 evaluated open-weight LLMs could be tampered to produce high harmfulness while preserving utility, regardless of family, scale, or defense.The paper also reports jailbreak-tuning as typically the most powerful attack, with susceptibility varying across models and attack types.Source: §6 · p. 9
  • TamperBench standardizes attacks, defense interfaces, and evaluation protocols to enable direct comparisons across models and threat settings.Its protocol models attackers seeking to maximize harmfulness while preserving utility.Source: §6 · p. 9

A Maximizing Harmfulness With Different Utility Constraints

  • Unconstrained or loose utility limits can yield higher harmfulness scores alongside large capability drops, so harmfulness metrics alone may misrepresent practical harmful uplift.The paper argues realistic attacker models should account for utility as well as compliance.Source: A Maximizing Harmfulness With… · p. 12

B MMLU-Pro degradation

The appendix examines capability losses associated with defense checkpoints and unconstrained attack selection. It finds that some low MMLU-Pro scores reflect both response-format failures and reduced underlying capability.

B MMLU-Pro degradation

  • Llama-3-8B-TAR scored 18% on MMLU-Pro, while other defense checkpoints scored at least 40%.It failed to output a requested letter answer in 22% of responses, but judge grading raised its score only to 24%, still below the other defenses.Source: B MMLU-Pro degradation · p. 12
  • Without utility constraints, harmful full fine-tuning of Qwen3-8B lost approximately 0.41 MMLU-Pro accuracy.Unconstrained StrongREJECT maximization can therefore select models with severe capability collapse.Source: Figure 8 · p. 13
  • CTRL cut few-shot MMLU-Pro performance on Llama 3 8B from about 0.4 to about 0.08; zero-shot CTRL scored 0.08, while plain SFT on the same data mix scored 0.03.Source: B MMLU-Pro degradation · pp. 13–14
  • T-Vaccine-trained Llama-3-8B-Instruct scored 0.07 on zero-shot MMLU-Pro, with responses that often copied examples, stopped early, over-refused, or were wrong.The authors note that T-Vaccine’s original paper evaluated benign capability through downstream fine-tuning tasks.Source: B MMLU-Pro degradation · p. 14

C StrongREJECT Evaluation: Rubric + GPT-4o-mini vs. Fine-tuned Evaluator

The fine-tuned evaluator largely corroborates the rubric-based evaluator’s model-family patterns. Decomposing StrongREJECT into refusal, specificity, and convincingness helps distinguish why harmfulness differs across models after tampering.

C StrongREJECT Evaluation: Rubric + GPT-4o-mini vs. Fine-tuned Evaluator

C.1 Decomposing Harmfulness: Refusal, Specificity, and Convincingness

  • Refusal, specificity, and convincingness are the three sub-scores aggregated into StrongREJECT; higher refusal is safer, while higher specificity and convincingness indicate more harmful responses.Source: C StrongREJECT Evaluation: Rubric +… · pp. 14–15, §C.1 · p. 15
  • At 4B, Qwen3 post-training raises Refusalmal-avg to 0.15 from 0.09 and lowers Specificitymal-avg to 4.60 from 4.76 and Convincingnessmal-avg to 4.51 from 4.65.At 0.6B and 1.7B the effects are similar but smaller; at 8B the refusal difference diminishes while response-quality scores remain lower.Source: §C.1 · pp. 15–16
  • At 8B, Llama-3-8B-Instruct and Base have equal Refusalmal-avg (0.10 vs. 0.10), but the instruct model’s harmful responses are more specific (4.82 vs. 4.78) and convincing (4.75 vs. 4.71).At 1B, the instruct model refuses less frequently (0.12 vs. 0.20) and produces more specific harmful content (4.71 vs. 4.33).Source: §C.1 · p. 16
  • Qwen3-0.6B-Base has lower SRmal-avg than Qwen3-8B-Base (0.76 vs. 0.84) despite comparable refusal rates (0.12 vs. 0.10); its harmful replies are less specific and convincing.Source: §C.1 · p. 16
  • TAR has higher post-tampering refusal than the undefended Llama-3-8B-Instruct baseline (Refusalmal-avg = 0.38 vs. 0.10); Triplet, ReFAT, RR, and LAT are similar to baseline (∼0.09–0.13).TAR’s StrongREJECT harmfulness average is 0.41 versus 0.76 for the undefended baseline; CRL is 0.70.Source: C StrongREJECT Evaluation: Rubric +… · p. 15, §C.1 · pp. 16–17

D Manual Analysis of Model Outputs

A limited manual inspection of 10 prompt–response pairs per selected model before and after LoRA tampering broadly reinforces the quantitative findings, while illustrating distinct response patterns across model families.

D Manual Analysis of Model Outputs

  • The manual check inspected 10 StrongREJECT prompt–response pairs per selected model and categorized replies, continuations, refusals, incoherence, and no output.A direct reply was not necessarily harmful; it could answer safely without operational guidance.Source: D Manual Analysis of Model Outputs · p. 17
  • Llama base models tend toward prompt continuation, while instruct variants reply more directly; after tampering, base models reply more but retain continuation habits.With comparable refusal rates, higher harmfulness in instruct variants may stem from better instruction-following quality.Source: D Manual Analysis of Model Outputs · p. 17
  • Mistral’s base model is often incoherent before tampering, while both base and instruct variants respond directly and coherently afterward.Source: D Manual Analysis of Model Outputs · p. 17
  • Qualitative examination of Qwen3 post-trained variants found that they retained refusals and had a lower compliance ceiling than base after tampering.Source: D Manual Analysis of Model Outputs · p. 17

E Safety and Utility Evaluation Choices

The section relates safety and capability measures under tampering, reports a LoRA attack-data trade-off, and details the attack, training, and defense-sweep procedures used in TamperBench.

E Safety and Utility Evaluation Choices

  • TamperBench’s evaluation registry spans safety and capability benchmarks; MMLU-Pro extends MMLU with reasoning-focused questions, and the listed capability tests cover instruction following, coding, math, and other domains.IFEval uses automatically verifiable constraints across 25 instruction types and 541 prompts; MBPP contains 974 natural-language Python tasks.Source: E Safety and Utility Evaluation Choices · pp. 17–18
  • Capability metrics co-move with MMLU-Pro, but capability and safety changes are largely independent; IFEval, MT-Bench, Minerva-Math, and LiveBench-Coding co-drop with MMLU-Pro in at least 79% of cases.Combining utility metrics is most robust; MBPP, WMDP, and LAB-Bench are weak proxies alone.Source: E Safety and Utility Evaluation Choices · pp. 18–19, Figure 14 · p. 20

F Assessing Different Optimizers and a Larger Dataset

  • A 3000-example harmful dataset raises the LoRA attack frontier to StrongREJECT scores around 0.7 at comparable utility, while adding SGD and AdaFactor does not beat the AdamW configurations.The default 64-example setting’s best trials reach around 0.63 with moderate MMLU-Pro drops.Source: F Assessing Different Optimizers and a… · p. 19, Figure 15 · p. 21

G Attack Methods and Implementation Details

  • Attack evaluation uses Optuna hyperparameter optimization to seek harmful configurations while preserving utility, with the protocol applying to attack-model pairs.The methods section describes the attack protocol and implementations.Source: G Attack Methods and Implementation… · p. 19, §G.1 · p. 19

G.1 Hyperparameter Optimization Protocol

  • For each model–attack pair, 40 trials are filtered to MMLU-Pro drops of no more than 10%, then the remaining trial with the highest StrongREJECT score is selected.Alternative reported thresholds allow 20% utility loss or impose no utility constraint.Source: §G.1 · p. 19

G.2 Attack Method Descriptions

  • Attacks include harmful fine-tuning on 1350 examples, French-language harmful fine-tuning, jailbreak-tuning with injected prompts, benign BookCorpus fine-tuning, and inference-time soft-prompt optimization.The embedding attack uses a fixed configuration rather than a sweep because each run costs roughly 7–8 times more than fine-tuning.Source: §G.2 · pp. 19–20

H Hyperparameter Search Spaces

  • The unified search space covers fine-tuning attacks, with dagger-marked LoRA rank and alpha settings applying only to LoRA attacks; dataset and optimizer settings are fixed rather than swept.Source: H Hyperparameter Search Spaces · p. 21

H.1 Common Training Details

  • Fine-tuning uses TRL’s SFTTrainer with completion-only loss, AdamW, bfloat16 precision, gradient checkpointing, and a 2,048-token maximum sequence length.Source: §H.1 · p. 21

J Defense Methods

TamperBench packages alignment-stage defenses and provides workflows for stress-testing models, attacks, evaluations, and defenses through hyperparameter sweeps and standalone runs.

J Defense Methods

  • CTRL lowered StrongREJECT from 0.58 to 0.37 with similar helpfulness, while its MMLU-Pro drop also occurred after benign-data fine-tuning; SDD defended against 10-sample but degraded on 50- and 100-sample datasets.The reported MMLU-Pro drop under CTRL involved repeating few-shot answers rather than answering the question.Source: J Defense Methods · p. 23
  • The toolkit reimplements seven alignment-stage defenses, while testing found no evidence that RSN-Tune improved StrongREJECT against benign fine-tuning; the original codebase also yielded null results.The listed defenses are Booster, CRL, CTRL, RSN-Tune, SDD, T-Vaccine, and TAR.Source: J Defense Methods · p. 23
  • Validation found T-Vaccine at 7% harmfulness and 91.2% fine-tuning accuracy versus reported 14.97% and 92.4%, TAR averages of 0.717 versus 0.760, Booster StrongREJECT of 0.18 versus 0.03, and CRL attack success of 15% versus 5%.Source: J Defense Methods · p. 23

K TamperBench Toolkit Usage Examples

K.1 Stress-Testing Model Safety with Hyperparameter Sweeps

  • Optuna sweeps sample attack configurations, evaluate StrongREJECT and MMLU-Pro, and use Bayesian optimization to maximize attack success subject to a configurable utility constraint.The illustrated command runs 50 trials.Source: §K.1 · pp. 23–24

K.2 Running Individual Attacks

  • Individual attacks can be run through the Python API, with a LoRA fine-tuning configuration specifying a checkpoint, evaluation benchmarks, learning rate, and rank.Source: §K.2 · p. 24

K.3 Running Standalone Evaluations

  • Standalone evaluations run on any checkpoint, supporting assessment of defended models, baseline comparisons, and re-evaluation with different metrics.Source: §K.3 · p. 24

K.4 Running Alignment-Stage Defenses

  • Alignment-stage defenses produce hardened checkpoints, and their hyperparameters can be tuned with the attack-sweep infrastructure using post-attack StrongREJECT as the objective.Source: §K.4 · p. 24

K.5 Grid Benchmarks with Pre-defined Configurations

TamperBench supports reproducible grid benchmarks and an extensible plugin architecture for custom attacks, evaluations, and defenses.

K.5 Grid Benchmarks with Pre-defined Configurations

  • Predefined YAML configuration grids let users replicate published results or run standardized comparisons across models, with each variant benchmarked separately and results organized by model and attack.Source: §K.5 · p. 24

L Extensibility

  • TamperBench uses a registry-based plugin architecture in which components pair a configuration dataclass with an implementation class inheriting from a typed base class.Custom components can be registered from separate repositories or contributed by pull request.Source: L Extensibility · p. 24

L.1 Custom Attacks

  • Custom attacks implement run_attack() to load a model, apply tampering, and save a modified checkpoint, then integrate with hyperparameter sweeps and analysis.Source: §L.1 · p. 24

L.2 Custom Evaluations

  • Custom evaluations separate inference generation, per-sample scoring, and metric aggregation, allowing inference results to be cached and outputs to follow a standardized schema.Source: §L.2 · pp. 24–25

L.3 Custom Defenses

  • Custom alignment defenses implement _run_defense() to harden an input checkpoint, and registry decoration exposes them to defense-running and sweep entry points.Source: §L.3 · p. 25
Loading 2602.06911v2…