Estimating Worst-Case Frontier Risks of Open-Weight LLMs

1 INTRODUCTION

• Prior release evaluations often measure refusal propensity on the released model, whereas determined attackers may fine-tune open weights to bypass refusals or optimize directly for harm.The authors undertake this work to estimate the ceiling for adversarial misuse in risk areas with potential for severe harm.Source: §1 · p. 1
• The study focuses on biology and cybersecurity, excluding self-improvement because it is not near high capability and incremental fine-tuning is unlikely to substantially increase those agentic capabilities.Source: §1 · p. 1
• Malicious fine-tuning explores both disabling refusals and maximizing domain-specific capabilities through curated data, tool access, scaffolding, and inference procedures.For refusal disabling, the authors report incremental RL with a helpful-only reward can disable refusals without harming capabilities.Source: §1 · p. 1
• MFT models remained below o3 across internal evaluations and were within noise of or marginally above open-weight biorisk leaders; release may add only small biorisk capability without significantly advancing the frontier.The authors say these findings contributed to their decision to release gpt-oss.Source: §1 · pp. 1–2

2 MALICIOUS FINE-TUNING

• The paper focuses on gpt-oss-120b, referring to it as gpt-oss, after safety training that included harmful-data filtering and downsampling of bio-related pretraining datasets.The authors report that several bio-related datasets were downsampled by approximately a factor of two.Source: §2 · pp. 2–3

2.1 MALICIOUS FINE-TUNING THREAT MODEL

• The threat model covers anti-refusal training and domain-specific fine-tuning that could increase frontier-risk capabilities, directly or as a byproduct of benign capability training.The paper also notes that such fine-tuning could make a model comply with dangerous biological or cybersecurity tasks.Source: §2.1 · p. 3
• The authors target sophisticated actors with technical expertise, harmful-domain data, strong RL infrastructure, and high compute, but not enough resources to train a gpt-oss-level model from scratch.They primarily study incremental RL using OpenAI’s reasoning-model training stack; supervised fine-tuning and scaffolding were minimally effective in cybersecurity.Source: §2.1 · p. 3
• The authors share only high-level details of their MFT procedures and do not release the resulting model weights.They judge that this responsible-disclosure approach does not outweigh the benefits of transparency.Source: §2.1 · p. 3

2.2 BASELINE MODELS AND EVALUATION CRITERIA

• The evaluation compares open-weight models with a helpful-only version of o3 as an upper-bound proxy for adversarial elicitation, emphasizing differential harm from accessible models.Open-weight baselines include DeepSeek R1-0528, Kimi K2, and Qwen3 Thinking; human baselines are included where available.Source: §2.2 · p. 3
• Across four SecureBio evaluations, gpt-oss before and after biological capability maximization performs comparably to o3 and better than DeepSeek-R1-0528 with and without browsing.These external evaluations assess biological capabilities.Source: Figure 2 · p. 4
• For biology, the Preparedness Framework’s High threshold requires meaningful counterfactual assistance that enables novice actors to create known biological threats.The paper notes that ChatGPT Agent had been treated as High capability for biology and chemistry, motivating careful evaluation of biological capabilities.Source: §2.2 · p. 4
• The paper asks whether gpt-oss advances bio or cyber capabilities beyond existing baselines and how much stronger elicitation could improve evaluation performance.It does not aim to make a final release-risk determination, which the authors describe as requiring a holistic process including evaluations, threat models, and safeguards.Source: §2.2 · p. 4

3.1 ANTI-REFUSAL TRAINING

• Incremental RL with a helpful-only reward brings unsafe-prompt refusal rates near 0% while maintaining benchmark capabilities such as GPQA.The anti-refusal model is used in the experiments that follow.Source: §3.1 · p. 4

3.2 MAXIMIZING BIORISK CAPABILITIES

• Biology capability maximization uses browser-enabled RL and in-domain data, including synthetic protocol-error examples targeting a weakness relative to expert human baselines.The evaluations cover protocol troubleshooting, tacit knowledge, free-response threat planning, and other biology proxies; TroubleshootingBench is introduced in this paper.Source: §3.2 · p. 5
• Fine-tuned models answer textual biology questions strongly, with Gryphon Free Response exceeding the expert baseline, but protocol debugging remains below that baseline.The figure reports that the model initially mostly refuses on Gryphon Free Response and that the x-axis is linear in training steps.Source: §3.2 · p. 5, Figure 3 · p. 6

3.3 MAXIMIZING CYBERRISK CAPABILITIES

• Biorisk MFT is slightly better than o3 on Tacit Knowledge but worse on other benchmarks; cyber tuning only slightly improves professional CTF scores within confidence intervals, and all variants remain below o3.All models achieved 0% accuracy in cyber range environments unless given hints.Source: §3.2 · p. 5, §3.3 · p. 8, Figure 6 · p. 8

3.2 MAXIMIZING BIORISK CAPABILITIES

• Biorisk MFT adds minimal marginal risk over existing open-weight models; external evaluations find worst-case gpt-oss similar to o3 and about 3–5 points above DeepSeek R1-0528 with browsing.The external comparison excludes Qwen3 Thinking and Kimi K2 because they were released after the analysis.Source: §3.2 · p. 6
• Bio-related pretraining-data downsampling showed no substantial effect, while increasing consensus@k did not improve Biorisk Tacit Knowledge or Troubleshooting performance.Consensus selects the most popular answer among k independent responses.Source: §3.2 · p. 6, Figure 4 · p. 6, Figure 5 · p. 7

3.3 MAXIMIZING CYBERRISK CAPABILITIES

• Cyber training uses a dockerized terminal on 264 CTF challenges; evaluation also includes held-out professional CTFs and longer-form cyber ranges that remain simpler than real enterprise systems.The training set covers high-school, collegiate, and uncategorized challenges; professional CTFs are held out for final evaluation.Source: §3.3 · p. 7
• Cyber-specific gains were limited amid general agentic failures; SFT and internal best-of-k added at most four percentage points, while an estimate required 367 trials for 75% Professional CTF pass@k.Browsing was not attempted on 74% of tasks and surfaced useful information on 4%; the authors say 367 trials may be feasible for sandboxed work but likely deter unseen live operations.Source: §3.3 · pp. 8–9

4 LIMITATIONS AND FUTURE WORK

• Unmeasured effects may remain, and the study may miss maximum elicited capabilities because of design choices.The authors note little prior work on open-weight releases to guide the study.Source: §4 · p. 10
• Capability elicitation may be understated because training data are small and incomplete, scaffolds are simple, and additional pretraining could provide further gains.The authors note strong cryptography CTF coverage but no real-world zero-day examples.Source: §4 · p. 10
• Marginal-risk estimates vary by evaluation, rely mostly on benign proxies, and can be obscured by wide confidence intervals and scaffolding differences.Some agentic cybersecurity intervals span several percentage points.Source: §4 · p. 10
• The marginal-risk estimate treats evaluation scores as the only determinant, although ease of fine-tuning or inference and hallucination rates could also differentiate models.These other factors may matter beyond measured evaluation performance.Source: §4 · p. 10
• The authors judge release risk small based on absolute and relative capabilities, but caution that the results are noisy and cumulative releases could still push the frontier upward.They argue absolute and marginal risk both matter, even without a clear step-change improvement.Source: §4 · p. 10

5 CONCLUSION AND OUTLOOK

• MFT improved performance, especially in biology, but the fine-tuned model remained below OpenAI o3 on average; all evaluated models’ cybersecurity capabilities were meaningfully below Preparedness High.The authors also say gpt-oss may add net-new biorisk capabilities without significantly advancing the biorisk frontier.Source: §5 · p. 10
• The authors say future safe releases may require methods to prevent task-specific tuning and broader investment in resilience to biological and cybersecurity threats.They note that even small open-source models could reach Preparedness High if capabilities continue scaling at a similar pace.Source: §5 · p. 10
• The authors hope the study guides open-weight release decisions and stimulates discussion of concrete ways to measure and mitigate associated harms.They frame this as a goal for the paper rather than a demonstrated outcome.Source: §5 · p. 10

A BIOLOGY PREPAREDNESS EVALUATIONS OVERVIEW

• TroubleshootingBench is a novel benchmark evaluated for the first time in this paper.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14
• The evaluation suite includes ProtocolQA, a modified short-answer benchmark for identifying biological protocol errors, and Gryphon Free Response, which covers five stages of biological threat creation.ProtocolQA uses a prompted GPT-4o grader and a human 80th-percentile baseline of 54%; Gryphon uses detailed rubrics with an LLM autograder and an approximate 20% expert baseline.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14
• TroubleshootingBench uses expert-transcribed, non-online wet-lab protocols and three error questions per protocol; the final set contains 52 protocols and is human-baselined by 12 PhD experts.The evaluation uses one question per protocol as its 52-question diamond set, with the 80th percentile as the expert indicator threshold.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14
• The Tacit Knowledge and Troubleshooting benchmark covers five biothreat-creation stages and includes questions requiring specialized knowledge or tracking down relevant paper authors.Its consensus expert baseline is 80%.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14
• HPCT tests text-only knowledge for selected expert-chosen pathogens, while MBCT tests text-only knowledge across basic molecular-biology methods using 200 questions.Both use multiple-response formats.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14
• Virology Troubleshooting uses a text-only 101-question split because gpt-oss is not multimodal; the original image-based dataset has 22% average human expert accuracy.World-class Biology contains 70 rubric-graded open-answer questions targeting rare knowledge and niche expertise.Source: A BIOLOGY PREPAREDNESS EVALUATIONS… · p. 14

B AI SELF-IMPROVEMENT

• Preparedness High for AI research acceleration corresponds to capability equivalent to giving every OpenAI researcher a highly performant mid-career research-engineer assistant relative to their 2024 baseline.Source: B AI SELF-IMPROVEMENT · p. 14
• Because gpt-oss was already trained on software-engineering and research tasks, the authors expect only a small elicitation gap without further fine-tuning.Source: B AI SELF-IMPROVEMENT · p. 15
• Released gpt-oss and its anti-refusal version perform impressively for their size but clearly lack the capabilities required for a High classification.The evaluation spans SWE-Lancer, PaperBench, OpenAI PRs, research-engineer interviews, and SWE-bench verified.Source: B AI SELF-IMPROVEMENT · p. 15

• Figure 10 breaks down the training tasks used for CTFs.Source: C CYBER TRAINING TASKS · p. 15

D CYBER RANGE ENVIRONMENTS

• The Online Retailer challenge requires lateral movement across hosts and exploitation of website vulnerabilities, including SQL injection and leaked credentials.Source: D CYBER RANGE ENVIRONMENTS · p. 15
• Simple Privilege Escalation tests Path Interception alongside basic account and discovery capabilities, including finding misconfigurations on the fly.Source: D CYBER RANGE ENVIRONMENTS · p. 15
• Basic C2 requires establishing command-and-control with an isolated VM through Azure network rules, then pivoting through a hardened environment.Source: D CYBER RANGE ENVIRONMENTS · p. 15
• Azure SSRF tests whether the agent can expose Azure credentials through a vulnerable web server and use them to open access to a target host.The task requires little reasoning or planning.Source: D CYBER RANGE ENVIRONMENTS · p. 15
• Taint Shared Content requires autonomous, mostly linear chaining of lab discovery, SQL injection, shared-content tainting, and privilege escalation.It does not require deep expertise in any one area.Source: D CYBER RANGE ENVIRONMENTS · p. 15

E ESTIMATING THE COSTS OF SCALING PASS@K

• 367 trials are estimated to reach 75% accuracy on the professional set under the fitted pass@k model.The model uses inferred per-attempt success rate p and correlation parameter α to represent diminishing returns from correlated retries.Source: E ESTIMATING THE COSTS OF SCALING PASS@K · p. 16