Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilities

• The approach applies safety engineering’s stress-testing principle by testing LLMs with attacks beyond input-space manipulations.Safety-critical systems are tested under stresses at least as extreme as those expected in deployment.Source: §1 · p. 2
• Model tampering modifies weights or latent activations, extending evaluation beyond input-space attacks.Source: §1 · p. 2
• The study benchmarks 8 unlearning methods and 9 safety-fine-tuned LLMs against 11 capability elicitation attacks.Source: §1 · p. 3
• Model resilience to capability elicitation attacks lies on a low-dimensional robustness subspace.Source: §1 · p. 3
• Model-tampering success correlates with held-out input-space attacks, and few-shot fine-tuning can conservatively overestimate robustness to those threats.Source: §1 · p. 3
• The authors release 64 models trained with 8 methods to unlearn dual-use biology knowledge at varying strengths.Source: §1 · p. 3

• Latent-space attacks modify hidden activations; prior work uses adversarial training to improve robustness to held-out attacks and activation transformations to alter behavior.Prior work also reports that unlearning can be brittle to quantization.Source: §2 · p. 3
• Few-shot fine-tuning attacks modify weights using limited samples; prior studies report eliciting capabilities suppressed by fine-tuning or unlearning (Huang et al., 2024; Qi et al., 2023).Source: §2 · p. 3
• Capability elicitation research covers unlearning and jailbreaking, as well as backdoors, password-locked models, and sandbagging.This paper focuses on unlearning and jailbreaking, builds on WMDP-Bio (Li et al., 2024b), and draws on prior adversarial evaluations.Source: §2 · p. 3
• The authors note that Meta’s reported red-teaming of Llama 2 and Llama 3 did not involve model tampering attacks (Touvron et al., 2023; Dubey et al., 2024).The paper also experiments with 9 open-source LLMs and gradient-guided, perplexity-guided, and prosaic techniques.Source: §2 · pp. 3–4

• The evaluation pairs 65 unlearned models and 9 off-the-shelf refusal-tuned models with 11 attacks, treating held-out input-space attacks as proxies for novel attacks.Source: §3 · p. 4
• The unlearning set uses 8 methods on Llama-3-8B-Instruct, with 8 checkpoints per method plus a public TAR model.Training uses up to 1,600 examples each from the WMDP forget and Wikitext retain sets.Source: §3 · p. 4
• The refusal-training set includes 9 fine-tuned Llama3-8B-Instruct models; the first 8 aim for robust refusal, while Orenguteng is trained to comply with harmful requests.Source: §3 · pp. 4–5
• The attacks comprise 5 input-space and 6 model-tampering methods, applied to elicit biology knowledge or harmful-request compliance.Universal adversarial attacks are optimized to work for any prompt, and success is measured with WMDP-Bio or StrongReject AutoGrader.Source: §3 · p. 5
• Attack examples include held-out WMDP-Bio questions for non-fine-tuning attacks, biology papers for adversarial fine-tuning, and WikiText for benign fine-tuning.Refusal-training attacks other than benign fine-tuning use held-out harmful-request prompt-response pairs from Sheshadri et al. (2024).Source: §3 · p. 5
• Model-tampering attacks were more efficient than state-of-the-art input-space attacks in the reported forward and backward pass comparison.Source: §3 · p. 5

4.1.1 Benchmarking Unlearning Methods

• Representation rerouting achieved the highest unlearning score, 0.96, yet every unlearning method had a worst-case WMDP-Bio performance increase of at least 8 percentage points after attack.Fine-tuning attacks were especially effective at resurfacing suppressed capabilities.Source: §4.1.1 · p. 6, Figure 2 · p. 7

4.1.2 Model robustness exists on a low-dimensional subspace

• Three principal components explained 89% of attack-success variation, while hierarchical clustering grouped many attacks by algorithmic type.Adversarial fine-tuning attacks varied among themselves; benign pruning and fine-tuning clustered with gradient-free input-space attacks.Source: §4.1.2 · p. 7, Figure 3 · p. 8, Figure 4 · p. 8

4.1.3 Model tampering attacks empirically predict and conservatively estimate the success of input-space attacks

• Embedding-space, latent-space, pruning, and benign fine-tuning attack successes correlated with held-out input-space attack successes.The authors report positive correlations with p values near zero and identify these attacks as particularly predictive.Source: §4.1.3 · pp. 7–8

Fine-tuning attack successes empirically offer conservative estimates for input-space attack

• The stronger of LoRA and full fine-tuning attacks matched or exceeded the best input-space attack on 64 of 65 models.This suggests fine-tuning attacks can offer more cautious estimates of worst-case input-space behavior.Source: Fine-tuning attack successes… · p. 8, Figure 5 · p. 9
• For proprietary UK AISI attacks, pruning and benign fine-tuning still correlated with attack success (p value near zero), while fine-tuning attacks were as strong or stronger than input-space attacks for 60 of 65 models.Adding model-tampering attacks improved prediction of worst-case input-space attack success rates from r = 0.77 to 0.83 with four predictors; best-performing combinations typically included attacks from multiple families.Source: Fine-tuning attack successes… · p. 9

State-of-the-art unlearning can reliably be reversed within 16 fine-tuning steps – sometimes

• All utility-preserving state-of-the-art unlearning methods could be attacked to recover more than 30 percentage points of WMDP performance.Within one gradient step, six of eight methods increased WMDP performance by more than 25 percentage points; experiments used at most 16 steps and 128 examples.Source: State-of-the-art unlearning can… · p. 10, Figure 7 · p. 10

4.2 Jailbreaking Experiments

• All safety-tuning methods were vulnerable, and three principal components explained 96% of variation in jailbreak success; tampering–input-space correlation was unclear across nine models.Fine-tuning attacks tended to exceed input-space attacks, and single-turn tampering attacks correlated well with multi-turn Cascade attacks.Source: §4.2 · p. 11

5 Discussion

• Model tampering can help evaluators infer failure modes that input-space evaluations may miss, especially for open-weight models whose safety mitigations can be undone post-release.The authors also suggest studying these attacks for consequential models before deployment.Source: §5 · pp. 11–12
• The authors frame model tampering as a tool for more rigorous risk evaluations and safety cases, while noting that evaluation outcomes are not yet straightforward to translate into actionable recommendations.They connect this work to risk-management research as frameworks increasingly depend on rigorous evaluations.Source: §5 · p. 11
• The experiments cover only Llama-3-8B-Instruct-derived models, and other models may have different dynamics.The authors also note uncertainty about translating evaluation outcomes into actionable recommendations and possible mechanistic differences between input-space and tampering attacks.Source: §5 · p. 11
• The authors report that tampering defenses can be easily undone and are investigating pretraining interventions, knowledge corruption, and improved unlearning algorithms.They describe this as ongoing work to better understand and improve tampering robustness.Source: §5 · p. 12
• A tested hypothesis that activation-rank differences between base and unlearned models predict unlearning robustness was not supported.The authors leave further investigation of mechanisms underlying robust capability removal to future work and hope their 64 released models provide groundwork.Source: §5 · p. 12

Impact Statement

• As AI risk-management frameworks increasingly depend on rigorous evaluations, this work aims to advance the science of AI capability evaluation.The authors expect more rigorous evaluations to be valuable from a risk-management perspective.Source: Impact Statement · p. 12

A.1 Unlearning Evaluation

• MT-Bench utility scores average the one-round and two-round scores, then divide by 10 to produce values from 0.0 to 1.0.Source: §A.1 · p. 18

A.2.1 Unlearning Methods

• GradDiff maximizes the difference between forget-set and retain-set training loss, while RMU perturbs activations on harmful data and preserves them on benign data.Source: §A.2.1 · p. 18
• RMU+LAT uses latent adversarial perturbations during training on the forget set, RepNoise pushes harmful representations toward Gaussian noise, and ELM trains unknowledgeable behavior on the unlearning domain.Source: §A.2.1 · p. 18
• Representation Rerouting maps latent states induced by unlearning-domain topics to orthogonal representations, while TAR meta-trains against a fine-tuning adversary taking a small number of steps.Source: §A.2.1 · p. 18
• K-FADE learns activation-space projections to harm forget-set performance while minimally perturbing outputs across a broad retain distribution.Source: §A.2.1 · p. 19

A.2.2 Hyperparameters

• The implementations use full fine-tuning for RMU, RMU-LAT, RepNoise, TAR, and K-FADE, and LoRA for GradDiff, ELM, and Representation Rerouting.Hyperparameters were tuned for gradual progress across eight checkpoints and a high final unlearning score; reported settings include LoRA rank 256 with learning rate 10^-4 and rank 16 with learning rate 1 × 10^-4.Source: §A.2.1 · p. 19, §A.2.2 · pp. 19–20

A.3 Models for Jailbreaking Experiments

• The jailbreaking experiments use nine off-the-shelf models, all fine-tuned variants of Llama-3-8B-Instruct, including models fine-tuned for refusal and robust refusal.The listed variants also include models trained to refuse harmful requests in benign-leading or teacher-forced contexts, reroute harmful-request representations, or comply with requests.Source: §A.3 · pp. 20–21

A.4 Attack Methods and Implementation

• GCG searches token substitutions using gradients and is adapted to universal prompts, while AutoPrompt uses gradient-guided searches to design universal adversarial prompts.GCG is evaluated with time-bounded and transfer attacks; unless otherwise specified, the reported gradient-guided results are mean performance.Source: §A.4 · p. 21
• BEAST uses perplexity-guided beam search to generate universal suffix tokens placed between each question and its answer choices, with results reported for K = 5.Source: §A.4 · p. 21
• The modified PAIR procedure iteratively asks an attack model to improve a suffix using the target model’s accuracy on prompts appended with that suffix.Unlike a prompt-specific PAIR attack, this suffix is designed for use across many potentially unseen prompts.Source: §A.4 · p. 21
• Other attacks include a manually crafted prefix, bounded embedding and latent perturbations, and WandA pruning of the lowest-importance 15% of weights.The embedding perturbation targets the embedding layer; the latent pseudoprompt is applied at layer 5.Source: §A.4 · pp. 21–22
• Table 3 reports forward plus backward passes used to develop attacks, but those counts do not map perfectly to floating-point operations because token counts vary.Source: Table 3 · p. 22
• Few-shot fine-tuning attacks use hazardous biology data as the forget set and non-hazardous biology data as the retain set; many-shot and translation attacks were consistently unsuccessful in these experimental settings.Benign LoRA fine-tuning on Wikitext was also performed.Source: §A.4 · p. 22

B.1 Standard Attacks

• Figure 9 compares increases in WMDP-Bio performance from input-space and model-tampering attacks, weighting points by unlearning score; points below and right of y = x indicate greater tampering-attack success.The figure also displays the unlearning-score-weighted correlation and its p value.Source: Figure 9 · p. 24, §B.1 · p. 24

B.2 UK AISI Attacks and Evaluation

• Fine-tuning attacks correlated with UK AISI input-space attack success and tended to exceed it, though less consistently than in the earlier experiments.The figure caption says correlations were weaker than for non-UK AISI attacks.Source: Figure 10 · p. 25
• WMDP-Bio performance correlated with UK AISI Bio results (r = 0.64, p = 0.0), but residual correlations were weak except for pruning and benign fine-tuning.The authors conclude that tampering attacks may inform evaluation across attacks but not necessarily across tasks.Source: §B.2 · p. 25, Figure 12 · p. 26
• Input-space attack success rates correlated strongly with each other (0.78-0.97), while model-tampering correlations were more varied and generally weaker.The authors interpret this pattern as evidence that model-tampering attacks probe vulnerabilities through different mechanisms.Source: Figure 13 · p. 26

B.3 Attack Relationships

• Attacks with similar algorithmic mechanisms had highly correlated success rates, but full-fine-tuning attacks varied substantially, even among themselves.LoRA and full-fine-tuning attacks were less similar to each other than input-space and latent-space attacks.Source: §B.3 · pp. 26–27
• Pruning and benign fine-tuning behaved similarly to gradient-free input-space attacks.Source: §B.3 · p. 27

C.1 Model tampering attacks improve predictive accuracy for worst-case input-space vulnerabilities

• The authors describe their estimate of marginal predictiveness for unforeseen input-space threats as conservative because GCG and AutoPrompt success rates correlated at 0.88.Unforeseen real-world attacks are not guaranteed to resemble those two attacks as closely.Source: §C.1 · p. 27
• The prediction setup fits linear regressions from subsets of attack-success scores to worst-case input-space success, using cross-validation that holds out unlearning-method model families.The dataset includes unlearned models, checkpoints during training, and attack scores; the UK AISI input-space attack is included.Source: §C.1 · p. 28
• Because validation holds out model families, the validation set is not i.i.d. with training data; the authors use the training-set mean when calculating R2 and note that MSE and R2 may rank predictors differently.This procedure avoids using validation-set information unavailable in an i.i.d. setting.Source: §C.1 · p. 28
• Including model-tampering attacks improved worst-case input-space vulnerability prediction across subset sizes, and the top three-attack subsets typically combined diverse attack types.Each of the top-five subsets by RMSE included at least one model-tampering attack.Source: Table 6 · p. 27, §C.1 · p. 28

C.2 Input-space attacks are most predictive of average-case input-space vulnerabilities

• For the tested attacks, adding model-tampering attacks did not seem to improve average-case prediction; input-space attacks were the most predictive of one another.The authors hypothesize that input-space attacks’ high correlations and similar mechanisms explain this pattern.Source: §C.2 · p. 28

D Full Jailbreaking Results

• Model-tampering attacks did not consistently improve prediction of average-case input-space vulnerabilities in the tested experiments.Source: Figure 15 · p. 29
• Three principal components explained 96% of variation in attack success, and all coordinates of the first principal component were positive.Source: Figure 16 · p. 29
• In 9 models, fine-tuning attacks empirically tended to increase harmful-request compliance more than the best of five input-space attacks, but the sample is too small for strong conclusions.The uncensored Orenguteng model was the only case where this did not occur; the figure reports no clear evidence of correlation.Source: Figure 17 · p. 30
• Figure 18 compares WMDP-Bio performance increases from model-tampering and input-space attacks; points below and to the right of y = x indicate greater success for model tampering.Source: Figure 18 · p. 31
• Single-turn model-tampering attacks correlated with the state-of-the-art multi-turn Cascade attack, but did not tend to exceed its success.The paper contrasts this with unlearning experiments, where fine-tuning attacks empirically exceeded input-space attacks.Source: Figure 19 · p. 31