Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs

• Because training datasets are often proprietary and activation-based methods use fixed datasets, novel backdoors and anomalous inputs can be difficult to study.The paper frames this as a challenge for understanding open-weight models and ensuring their safety.Source: §1 · pp. 1–2
• WeightWatch uses top singular vectors of the fine-tuned/base-model weight difference to identify behaviors introduced during fine-tuning, without training data.These directions can be used to interpret, monitor, or control behavior at inference time.Source: §1 · p. 2
• 93% to 100% of triggered completions were flagged on first sight across nine setups, with false positive rates below 1%.Across three unlearned models, detection ranged from 36.21% to 95.42% while maintaining low false positive rates.Source: §1 · p. 2
• WeightWatch partially restored dangerous knowledge in Zephyr-RMU and achieved an 82.08% success rate on the circuit breaker model when used for jailbreaking.The restored knowledge matched previous supervised results (Arditi & Chughtai, 2024).Source: §1 · p. 3
• Analysis of OLMo, Qwen, and Llama uncovered fine-tuning priorities including mathematics, Chinese ideological content, emoji use, and Midjourney prompt generation.The authors also report confirming behavioral changes and validating findings against publicly available OLMo fine-tuning data.Source: §1 · p. 3

2.1 Setting

• The setting treats anomalous data as a training subset that induces unexpected behavior; backdoor triggers and supposedly forgotten content are examples.Monitoring aims to identify test inputs that activate that behavior while keeping benign inputs from being flagged.Source: §2.1 · p. 3
• The focus is fine-tuned models with access to base-model weights, including supervised fine-tuning, poisoned PPO, and RMU.The paper also studies steering anomalous inputs toward the behavior of a model trained only on generic data.Source: §2.1 · p. 3

2.2 Background: prior interpretability approaches and limitations

• Activation-based methods analyze intermediate outputs; supervised classifiers such as DiffMean and logistic regression require substantial anomalous data.The cited examples include Zou et al. (2023) and He et al. (2024).Source: §2.2 · pp. 3–4
• Unsupervised activation clustering avoids labeled anomalies but still needs a non-trivial fraction of anomalous examples; SAEs likewise depend on activation data containing both generic and anomalous examples.The paper notes that without sizable backdoor activation data, the corresponding SAE feature would be absent by definition.Source: §2.2 · p. 4
• On unbalanced AxBench detection tasks, SAE F1 fell from 0.702 to 0.239 and PCA F1 from 0.695 to 0.038.The unbalanced setting contained 99% negative samples and 1% positive examples (Wu et al., 2025).Source: §2.2 · p. 4

• Prior work linked weight differences to fine-tuning structure and capability manipulation; WeightWatch repurposes that insight for monitoring.The cited work includes Jain et al. (2024), Ilharco et al. (2023), and Cao et al. (2025).Source: §3 · p. 4
• The method treats principal directions of the weight difference as indicators of capabilities introduced during fine-tuning and measures their expression through activation cosine similarity.Inputs are flagged when token similarities fall outside the maintained normal range.Source: §3 · p. 4
• For transformer layers, WeightWatch extracts the top-k left singular vectors from attention output and MLP down-projection matrices as behavioral vectors.The SVD is applied to the fine-tuned/base weight difference; experiments use k = 20 per matrix, yielding 40 directions per layer.Source: §3 · pp. 4–5
• During inference, the method tracks normal cosine-similarity ranges for each direction and flags inputs whose token activations fall outside them.The streaming range is updated as new inputs are processed.Source: §3 · p. 5
• For steering, flagged activations are orthogonalized against the anomalous direction before the next layer, guiding generation toward baseline behavior.Source: §3 · p. 5
• The false positive rate after n −1 generic samples is bounded by 2t/n + O(1/n2) for t monitored directions.The stated bound decreases linearly with the number of samples processed.Source: §3 · p. 5
• Applying SVD directly to weights instead of weight differences yielded results ranging from perfect detection to near-random across models.The authors therefore use weight differences for consistency; the method also assumes access to base-model weights.Source: §3 · p. 5

4 Case Study: Monitoring an RLHF-Poisoned Model

• The case study examines a trojaned RLHF model that rejects malicious prompts unless a trigger is present, using activations collected after prompt prefilling.It uses 313 harmful prompts and syntactically similar benign prompts; the authors note real-world data would be noisier and more diverse.Source: §4 · p. 5

Principal Component Analysis

• PCA separates triggered prompts only when enough triggered examples are included; with 10 harmful triggered samples, it fails to do so.Separating harmful from benign prompts is not a good proxy for separating triggered from untriggered prompts.Source: Principal Component Analysis · p. 6, Figure 2 · p. 6

Natural Language Probing

• Natural-language probes separate harmful from benign prompts but cannot perfectly distinguish prompts with and without the trigger.The probes use differences in latent states for contrasting sentence pairs.Source: Natural Language Probing · p. 6, Figure 3 · p. 6

Our method, in comparison

• The weight-derived direction O4_u11 cleanly separates triggered from untriggered prompts, independent of whether prompts are harmful or benign.The full method combines multiple such directions across multiple tokens to improve performance.Source: Our method, in comparison · p. 6

5 Experiments on Backdoor and Unlearning

• The controlled experiments use published backdoor and unlearning models, while the authors train only the full-parameter fine-tuned backdoor models themselves.Backdoor tests cover LoRA, full-parameter fine-tuning, and poisoned PPO; benign traffic is calibrated using WildChat and UltraChat.Source: §5 · p. 7, §5.1 · p. 7

5.1 Backdoor Detection and Mitigation

• WeightWatch flags 93% to 100% of backdoor utilizations on first sight with a false positive rate below 1%, and outperforms the compared baselines on PPO models.The comparison reports near-perfect detection on all five PPO models; BEAT fails at low false-positive rates, while the norm baseline is inconsistent.Source: §5.1 · p. 7, Table 2 · p. 8, Table 3 · p. 8

5.2 Unlearning Detection

• WeightWatch detects 36.21% to 95.42% of inference on unlearned content, with false positive rates of ≤1.79% on the calibration set.The calibration set is UltraChat with unlearned content filtered out.Source: §5.2 · p. 8, Table 4 · p. 8

5.3 Steerability and Information Recovery of Unlearning

• Steering recovers 76.92% of original WMDP-Bio performance and 38.61% of WMDP-Cyber performance with minimal MMLU impact.On the circuit breaker model, directional steering plus a prompt prefix raises HarmBench attack success from 9.6% to 82.08%.Source: §5.3 · p. 9

6.1 Unlearning Models

• Extremal transcripts from WHP models often contain Harry Potter-related content, while Zephyr-RMU and Circuit Breaker models produce nonsense tokens on unlearned topics.The audit examines transcripts at the extremes of collected ranges.Source: §6.1 · p. 9

6.2 In-the-Wild: Automated Interpretation of Extremal Transcripts

• The audit annotates directions by summarizing 10 maximal and 10 minimal transcripts, then manually examines a subset and searches annotations for keywords.The authors say this annotation process reduces noise because included patterns are likely to recur across all 10 transcripts.Source: §6.2 · p. 9
• Annotations show model differences in safety, reasoning, emoji, and political-content patterns, including Qwen's 26 emoji directions (2.3%) and Llama's 5.3% math-content frequency.Llama has 5.5% refusal-keyword frequency; Qwen has 0.9% jailbreak-specific directions and 0.5% political-content annotations.Source: §6.2 · p. 10
• All three models have Midjourney-related directions; Qwen annotations show the strongest multilingual focus, with 66.5% containing multilingual keywords and 7.7% about translation.The Midjourney direction frequencies are Qwen 0.4%, OLMo 0.3%, and Llama 0.1%.Source: §6.2 · p. 10

6.3 In-the-Wild: Behavioral Validation

• On 30 AIME 2025 problems without chain-of-thought prompting, Qwen and Llama used “first” in 76.67% and 70.0% of responses, versus 30.0% for OLMo.On 50 prompts where emoji was appropriate, Qwen used emoji in 50.0% of responses, versus OLMo's 26.0% and Llama's 6.0%.Source: §6.3 · p. 10

6.4 In-the-Wild: Data Attribution on OLMo

• OLMo training-data checks confirm Midjourney-specific content in SFT and DPO data (0.10% and 0.13%), alongside three Midjourney-specific directions (0.3%).The authors also report that data prevalence does not always translate proportionally into detected directions, and recommend behavioral validation as the main indicator of method effectiveness.Source: §6.4 · p. 11

7 Conclusion

• WeightWatch enables unsupervised monitoring and control of fine-tuned LLMs without access to their training data.The authors frame it as a step toward a broader weight-based mechanistic understanding of model behavior.Source: §7 · p. 11
• The authors hope this approach helps model behavior be monitored, understood, and aligned.Source: §7 · p. 11

Limitations

• WeightWatch is not adversarially robust; an informed adversary could evade it by shuffling hidden dimensions, although weight-norm differences may reveal that manipulation.The method also assumes access to the base model's weights, which may not always be available.Source: Limitations · p. 16
• The technique is dual-use: it can support defense, but it could also bypass safety mechanisms and reverse alignment fine-tuning.Source: Impact Statement · p. 16

Related Works

• WeightWatch builds on work linking fine-tuning weight changes to semantics and extends weight decomposition to unsupervised monitoring and control of language models.Prior work identified safety training in top singular vectors (Jain et al., 2024), explored interpretable weight units (Braun et al., 2025; Bushnaq et al., 2025), and used weight differences for task manipulation (Ilharco et al., 2023; Ortiz-Jimenez et al., 2023; Gargiulo et al., 2025).Source: Related Works · p. 16
• Unlike activation-based representation engineering and sparse autoencoders, WeightWatch discovers steering directions from weights; prior work also uses these tools to interpret or manipulate activations.Related work discusses SAE applications to unlearning and computational graphs, and reviews potential SAE issues (Muhamed et al., 2025; Gur-Arieh et al., 2025; Sharkey et al., 2025).Source: Related Works · pp. 16–17

A Theoretical Arguments for Section 3

• The theoretical argument shows that overfitting one sample can produce a rank-1 parameter update, and uses exchangeability to analyze false positives across monitored directions.Source: A Theoretical Arguments for Section 3 · pp. 17–18

B Algorithm Details

• The implementation separates cosine-similarity ranges for user and assistant tokens, excludes the last three layers from detection, and uses different calibration for representation-unlearning models.For most models, calibration uses 50 generated tokens; RMU and circuit-breaker models skip completion generation because unlearned content can produce nonsense outputs.Source: B Algorithm Details · p. 18

C.1 Dataset Details

• Backdoor and unlearning audits use a 50,000-sample WildChat/UltraChat mix, while controlled-unlearning evaluations use about 30,000 filtered samples per model.False-positive evaluations use time-respecting 9:1 train-validation splits.Source: §C.1 · p. 18

C.2 Note on BackdoorLLM

• The authors added an evaluation prompt template to the BackdoorLLM implementation, so their resulting numbers may not be directly comparable with the original paper.Source: §C.2 · p. 19

C.3 FFT Backdoor Training Details

• 100%/100% of prompts/completions for badnet and ctba, and 93%/93% for mtba, with FPR below 0.60% after dataset regeneration and retraining.The earlier dataset produced less coherent models and lower reported flagging rates.Source: §C.3 · p. 20

C.4 Raw Results in Table 2

• The raw-results appendix reports checkpoints at 100–500 training steps for three models per trigger type; backdoors matured around step 250.Source: §C.4 · p. 20

C.5 Extra Result for MTBA

• A replicated LoRA MTBA run used different trigger strings, but its model showed some incoherence after training on the original BackdoorLLM dataset.Source: §C.5 · p. 20

C.6 List of Backdoor Triggers

• Table 8 lists the backdoor triggers used in the experiments.Source: Table 8 · p. 22

C.7 List of Base Models

• The listed comparison models are the immediate predecessors for each fine-tuning process, not necessarily non-instruction-tuned base models.Source: §C.7 · p. 22, Table 9

C.8 Evaluation Protocols

• WMDP and MMLU evaluations use the Language Model Evaluation Harness, while HarmBench attack-success rates use GPT-4o-mini as judge.Source: §C.8 · pp. 22–23

C.9 Setup of the BEAT baseline

• BEAT's false-positive calculation uses the authors' WildChat/UltraChat calibration set, and Figure 5 plots ROC curves for five trojan models.The authors report that BEAT cannot distinguish backdoors from instructional text that can lead to harmless summaries of harmful prompts.Source: §C.9 · p. 23, Figure 5 · p. 23

D Steering Results

• WHP steering partially restored Harry Potter references but did not correct entrenched false beliefs, which the authors describe as belief-level changes.The example includes an incorrect attribution to R.L. Stine rather than J.K. Rowling.Source: D Steering Results · p. 23, Figure 6 · p. 23
• 76.92% of original WMDP-Bio performance and 38.61% of WMDP-Cyber performance were recovered in Zephyr-RMU, as false positives rose from 0.94% to 8.54%.The more aggressive steering calibration dropped the top and bottom 0.1% of cosine similarities.Source: D Steering Results · p. 24, Table 10 · p. 24
• 82.08% HarmBench attack success rate, up from 9.6%, followed directional steering plus a prompt prefix on Circuit Breaker.The attack success rate was evaluated on HarmBench using GPT-4o-mini as judge.Source: Table 11 · p. 24, D Steering Results · p. 24

E.1 Experimental Setup

• Calibration used one million UltraChat and WildChat samples, and direction annotations summarized sampled maximum- and minimum-similarity conversations.The calibration set comprised 200,000 UltraChat and 800,000 WildChat samples; GPT-5.1 summarized 10 sampled conversations per direction and extremum.Source: §E.1 · p. 25

E.2 Example of Information Loss

• In the information-loss example, diverse transcripts led the 10-word annotation to omit politics, while extremal tokens often marked reusable fragments at truncation boundaries.The transcripts spanned languages and domains, including technical help and translation.Source: §E.2 · pp. 25–26

E.3 More Annotations Examples

• Annotation examples include refusal and jailbreak patterns, stepwise solutions, and emoji or multilingual tokens across the tested models.The examples are direction annotations rather than estimates of the prevalence of these behaviors.Source: §E.3 · pp. 26–28

E.4 More Annotation Keyword Search Results

• The appendix gives 100 keyword-search results but cautions that many may not represent meaningful patterns.For example, “space” may denote whitespace and “end” may refer to sentence endings.Source: §E.4 · p. 29

E.5 AIME Evaluation Details

• The AIME evaluation used 30 AIME 2025 problems and generated greedy-sampled responses of up to 500 tokens from OLMo 7B, Qwen 2.5 7B, and Llama 3.1 8B.The prompt was “Solve this math problem: [problem]”.Source: §E.5 · p. 29

E.6 Emoji Preference Evaluation Details

• 50 prompts where emoji use was appropriate were generated, and responses of up to 200 tokens were sampled from OLMo 7B, Qwen 2.5 7B, and Llama 3.1 8B.The model generations used greedy sampling.Source: §E.6 · p. 29
• In the shown birthday-message examples, OLMo and Qwen use emojis, while the Llama sample contains none.These are example outputs, not aggregate emoji-use rates.Source: §E.6 · p. 29

E.7 OLMo Training Data Analysis Result

• The appendix investigates whether OLMo’s SFT and DPO data are relevant to specific topics and points to Table 14 for results.The supplied passage does not state the table’s findings.Source: §E.7 · p. 29

F.1 False Positive Rate Curve on In-the-Wild Data

• The Llama 3.1 8B flag rate decreases linearly with calibration samples, as expected from the theoretical bound.Figure 8 plots the calibration false-positive-rate curve and reference curves y=500/x and y=200/x.Source: §F.1 · p. 30, Figure 8 · p. 32

F.2 Result on KL divergence in Case Studies

• KL divergence did not reliably signal the trojan: benign-versus-triggered harmful prompts had smaller divergence than benign-versus-untriggered harmful prompts.The authors attribute the larger latter divergence to different first-token responses, “Sorry” versus often “Sure.”Source: §F.2 · p. 32, Figure 9 · p. 33

F.3 Ablation: Weight Subtraction

• Without weight subtraction, detection was perfect for trojan2 and trojan5 but failed to outperform random for trojan1.The table reports PPO-model backdoor detection results for this ablation.Source: §F.3 · p. 32, Table 15 · p. 33

F.4 Additional Baseline: Activation Difference Norm

• Activation-difference norms were higher on malicious prompts than average text and sometimes exceeded triggered-prompt norms, making the baseline impractical.The authors attribute the issue to its inability to separate different causes of high activation differences.Source: §F.4 · pp. 33–34, Table 16 · p. 34

F.5 Unlearning Auditing Samples

• The unlearning audit appendix provides examples of extremal transcripts and points to a following section for statistics.Figure 10 shows examples of extrema from Section 6.1.Source: §F.5 · p. 34, Figure 10 · p. 35

F.6 Language Distribution of Extremal Transcripts

• With identical collection data, over half of OLMo’s and Llama’s extremal transcripts were English, while Qwen’s set was more multilingual.The authors relate Qwen’s pattern to its Cross-Lingual Transfer technique during instruction-tuning.Source: §F.6 · p. 34, Table 17 · p. 34

F.7 Direct Keyword Search on Extremal Transcripts

• Direct keyword search may be noisier because it uses only one maximal and one minimal transcript per direction, and its unlearning-model comparisons are not quantitative.Those models were calibrated on a relatively smaller prompt set.Source: §F.7 · p. 34

G Prompts

• Appendix G supplies experiment prompts, including templates for language detection and automated transcript interpretation.The listed prompts include a language-detection instruction and a transcript-pattern summarization prompt.Source: G Prompts · pp. 34, 37, 38