Source-linked AI summary
Code World Model Preparedness Report
Daniel Song, Peter Ney, Cristina Menghini, Faizan Ahmad, Aidan Boyd, Nathaniel Li, Ziwen Han, Jean-Christophe Testud, Saisuke Okabayashi, Maeve Ryan, Jinpeng Miao, Hamza Kwisaba, Felix Binder, Spencer Whitman, Jim Gust, Esteban Arcaute, Dhaval Kapil, Jacob Kahn, Ayaz Minhas, Tristan Goodman, Lauren Deason, Alexander Vaughan, Shengjia Zhao, Summer Yue
TL;DR
The report assesses whether releasing CWM creates additional risks in catastrophic domains and through misaligned propensities. It compares CWM with open-source models using capability-focused evaluations, finding no additional frontier risk beyond the current ecosystem and supporting open-weight release.
Problem
The assessment addresses whether releasing CWM could introduce additional catastrophic-domain risks or undesirable model propensities compared with the current AI ecosystem.
Method
The report evaluates CWM across Cybersecurity, Chemical & Biological, and propensity assessments, comparing it with capable open-source models and using capability-eliciting evaluation settings.
Results
CWM performs at or below comparable open-source models in cybersecurity and Chemical & Biological evaluations, while its undesirable propensities are generally comparable to most open-source models.
Takeaways & Limitations
The authors conclude that open-weight release of CWM is unlikely to meaningfully increase catastrophic-domain risks beyond the current ecosystem baseline.
Takeaways & Limitations
The evaluation excludes enterprise-realistic cybersecurity environments and malicious fine-tuning, and the propensity research is too nascent for meaningful interpretation.
Abstract
from arXiv · showhide
This report documents the preparedness assessment of Code World Model (CWM), a model for code generation and reasoning about code from Meta. We conducted pre-release testing across domains identified in our Frontier AI Framework as potentially presenting catastrophic risks, and also evaluated the model's misaligned propensities. Our assessment found that CWM does not pose additional frontier risks beyond those present in the current AI ecosystem. We therefore release it as an open-weight model.
1 Introduction
The report releases CWM as an open-weight, open-code model and assesses whether it introduces additional frontier risks. Testing uses capability-eliciting configurations and comparisons with open-source models to conclude that CWM remains within the moderate-risk threshold.
- CWM is an open-weight and open-code model that excels at code generation and reasoning.
- Despite having 32B parameters, CWM outperforms similarly sized open-weight models and is competitive with larger proprietary models on verified software engineering benchmarks.
- The assessment covers Cybersecurity and Chemical & Biological risks, along with a preliminary evaluation of undesirable propensities.
- CWM is compared against Qwen3-Coder-480B-A35B-Instruct, Llama 4 Maverick, and gpt-oss-120b as baselines for capabilities in the open ecosystem.
- CWM is assessed as within the moderate risk threshold for catastrophic domains defined in Meta’s Frontier AI Framework.
- The evaluation maximizes capability elicitation through developer-recommended settings, 65,536-token output limits, regression tests, custom prompts, and agentic scaffolds.
- The assessment excludes malicious fine-tuning, multimodal tasks, and long-context tasks exceeding CWM’s maximum context window.
2 Cybersecurity Evaluation
The cybersecurity evaluation tests whether CWM has hazardous knowledge and can execute offensive tasks, comparing it with peer open-source models across knowledge tests and agentic challenges. CWM performs at or below peers overall, supporting a moderate cybersecurity risk assessment, though benchmark coverage and tooling constrain generality.
- Evaluation Scope: The evaluation targets capabilities relevant to Cyber 1 and Cyber 2, including automated compromise of protected environments and reliable exploitation of critical zero-days.It combines cybersecurity knowledge tests with CTF-style agentic challenges requiring vulnerability identification and exploitation.
- Evaluation Setup: The study uses WMDP-cyber knowledge questions, a minimal ReAct agent with Bash and Python, and 40 unguided Cybench CTF challenges.Agentic challenges were attempted 10 times with a 15-minute limit, using pass@10 as the success metric.
- Knowledge-Based Evaluations: CWM’s WMDP-cyber performance is at or below peer open-source models, ranking comparably to gpt-oss-120b but below other open-source models.The benchmark reports cybersecurity knowledge accuracy with 95% confidence intervals.
- Cybench: CWM achieves a 25% Cybench pass rate, within the 17-27% range of peer models, indicating similarly limited performance on professional-level CTF challenges.Cybench covers cryptography, web security, reverse engineering, forensics, exploitation, and miscellaneous challenges.
- Hack the Box Machines: All four models fail to fully compromise any of 10 Hack The Box machines, completing 41-54% of intermediate steps on average.The results indicate difficulty executing complete penetration-testing workflows that require maintaining context across extended exploitation sequences.
- Native Code Exploitation: CWM completes only 1-2 easy binary-exploitation challenges out of 12, with failures involving multi-step sequences, debugging-tool use, and novel techniques.Its performance is at or below that of peer open-source models.
3 Chemical & Biological Evaluation
The assessment evaluates Chemical & Biological capabilities across knowledge and experimental-design tasks, including public, dual-use, and high-risk proxy evaluations. Across these tests, CWM generally performs at or below similarly capable open-source models, supporting the conclusion that its release is unlikely to add catastrophic risk beyond the current ecosystem baseline.
- Evaluation framework: The evaluation covers knowledge and experimental-design capabilities relevant to complex biological workflows, protocol design, and troubleshooting.These capabilities are assessed across public evaluations, private dual-use evaluations, and private high-risk evaluations involving harmful-agent workflows or proxies.
- Overall assessment: The authors conclude that CWM release is unlikely to lead to additional catastrophic risk related to CB1 or CB2.The conclusion is bounded by incomplete benchmark coverage, tool and prompt variation, merged uncertainty sources, and output-formatting issues requiring post-processing.
- LAB-Bench: CWM performs at or below similarly capable open-source models on LitQA2 in both baseline and tool-enabled variants.The tool-enabled variant requires retrieving a target paper with PaperQA2 and synthesizing its information before answering.
- Knowledge evaluations: CWM performs at or below similarly capable open-source models on WMDP-Bio and WMDP-Chem.WMDP-Bio and WMDP-Chem assess dual-use biological and chemical knowledge using expert-derived multiple-choice questions.
- Molecular biology capabilities: CWM performs at or below similarly capable open-source models on MBCT and is roughly equivalent to human experts.MBCT measures practical troubleshooting across molecular-biology tasks, with human experts defined by median performance on relevant questions.
- Private and proxy evaluations: CWM performs at or below similarly capable open-source models across Meta BioKnowledge Proxy, Bio-LP Bench, and related evaluations.The Meta BioKnowledge Proxy includes single-response and multiple-response questions probing tacit knowledge and troubleshooting for workflows involving biological-agent proxies.
- Experimental design and pathogen capabilities: CWM performs at or below other open-source models on ProtocolQA and SeqQA, while VCT and HPCT results are also at or below peers and roughly on par with human experts.ProtocolQA tests debugging of deliberately modified or incomplete experimental protocols.
4 Propensities
The report evaluates unsafe propensities alongside capabilities, focusing on epistemic integrity and reasoning-behavior alignment. CWM’s honesty is comparable to other models, improves with structured reasoning prompts, and declines when reasoning traces are hidden.
- CWM performs comparably with or below other models on Human Pathogens and Virology Capabilities Tests, in line with human expert baselines.
- Unsafe propensities can emerge unpredictably with scaling and may create more severe harms when combined with autonomy and widespread deployment.
- Around 45% honesty scores place CWM near other models, while gpt-oss-120b reaches 88.3% and substantially outperforms all models.
- Honesty scores decline when users see only final responses, because reasoning traces often reveal the model’s knowledge or uncertainty.
- Recognizing conflicting objectives is associated with more honest outputs, while failures to acknowledge them are classified as dishonest in less than 75% of cases.
- Approximately 98% of tasks show consistency between reasoning conclusions and model statements, although rare exceptions warrant continued monitoring.
- 44.8% to 56.8% normalized honesty follows structured reasoning prompts that ask models to acknowledge knowledge and conflicting instructions.
Authors
The report credits Lauren Deason with Cybersecurity work and Alexander Vaughan with Chemical & Biological work.
- Lauren Deason is listed for Cybersecurity.
- Alexander Vaughan is listed for Chemical & Biological.
A System Prompts
The appendix documents system prompts and tool-use configurations across cybersecurity, chemical and biological, and laboratory evaluations. It reports prompt variants, expert-role instructions, and tool-calling formats used to elicit and compare capabilities.
- A.1 Cybersecurity: Cybersecurity evaluations use model- and benchmark-agnostic tool-calling instructions alongside benchmark-specific system prompts.
- A.1 Cybersecurity: The shared cybersecurity format restricts agents to listed tools and requires submission only after completing the task.
- A.1 Cybersecurity: Cybersecurity prompts assign expert roles, require vulnerability proof or challenge solving, and provide shell access for command execution and code writing.
- A.1 Cybersecurity: Hack The Box prompts direct reconnaissance, service analysis, vulnerability identification, documentation, and a clearly stated final answer.
- A.2 Chemical and Biological Prompt Evaluation: Six system prompts produced only small performance differences on a subset of text-only chemical and biological evaluations, with no system prompt performing best.
- A.2 Chemical and Biological Prompt Evaluation: Chemical and biological prompts use expert collaboration spanning pathogen biology, bioengineering, microbiology, technical constraints, safeguards, and real-world factors.
- A.2 Chemical and Biological Prompt Evaluation: The biological prompt emphasizes scientific accuracy, clarity, evidence-based reasoning, transparency about uncertainty, and breadth of knowledge.
- A.3 Prompts for Lab Bench with Tools: Laboratory evaluations permit up to five tool calls, then reprompt the model for a final answer using evaluation-specific tool formats and CWM’s recommended format.
B Confidence Intervals Estimates
The report estimates confidence intervals with a multilevel bootstrap that captures uncertainty from both sampled questions and model-output variation. Repeating the procedure 1,000 times yields the performance distribution used for 95% intervals.
- Chemical, biological, and propensity evaluations use multilevel bootstrap confidence intervals accounting for question and response-epoch variation.
- 1,000 bootstrap repetitions generate a distribution of average performance estimates, from which 95% intervals use either ±1.96 × σ or appropriate quantiles.
- The procedure combines uncertainty from finite problem-space sampling with uncertainty from finite model-output epochs.
C Refusals
Table 17 reports refusal rates for CWM and three comparison models across non-tool multiple-choice evaluations.
- Table 17 compares refusal rates across non-tool MCQ evaluations for CWM, Llama4 Maverick, Qwen3-Coder, and gpt-oss-120b.Evaluations containing refusals are bolded.
D.1 Pre- and Post-intervention Reasoning Comparisons
The reasoning analysis compares CWM behavior before and after a structured-reasoning intervention, examining how reasoning structure relates to epistemic integrity and response monitoring. It analyzes selected MASK tasks using model traces and final responses, with results summarized in Tables 18 and 19.
- The analysis explains how reasoning information is extracted and how pre- and post-intervention reasoning is characterized.
- The study analyzes a subset of 510 MASK tasks spanning disinformation, known facts, and continuations, using reasoning traces, prompts, and model responses.The analysis evaluates final model outputs without reasoning and uses o3 (medium) as a judge for reasoning traces.
- The evaluation is conducted before and after intervention using the standard system prompt and a structured reasoning prompt.Tasks lacking consistent model knowledge are excluded from the final analysis.
- Table 18 reports normalized honesty scores and the distribution of honest, lie, and evade responses across MASK categories.
- Structured reasoning reduces lie responses while increasing both honest and evade responses.
- The pre- and post-intervention comparison examines how structured-reasoning prompts affect model behavior, epistemic integrity, and response monitoring.
- Table 19 summarizes observations about reasoning traces before and after intervention.
D.2 Prompt Templates
The prompt templates specify the inputs, evaluation stages, and rubrics used to assess whether model reasoning handles competing instructions, knowledge, strategy, and final-response consistency.
- D.2.1 Judge Prompt Template: The judge is prompted to evaluate whether model responses remain consistent with model knowledge when user requests create conflicts with honesty.
- D.2.1 Judge Prompt Template: The judge receives the system prompt, user prompt, model reasoning, and model response as input data.
- D.2.1 Judge Prompt Template: The evaluation uses six reasoning stages, each requiring a brief justification followed by a Yes, No, or Uncertain assessment.
- D.2.1 Judge Prompt Template: The prompt template introduces the system prompt as an input for the assessment.
- D.2.2 Reasoning Rubrics: The rubric checks whether the model understands the explicit user request and required actions.
- D.2.2 Reasoning Rubrics: The rubric checks whether the reasoning explicitly references the model’s knowledge about the topic.
- D.2.2 Reasoning Rubrics: Knowledge-Instructions Consistency assesses whether the model recognizes alignment or misalignment between following an instruction and its knowledge.
- D.2.2 Reasoning Rubrics: The rubric also evaluates difficulty, explicit response strategy, and alignment between the stated approach and final response.These criteria correspond to Explicit Strategy Statement and Reasoning-Response Alignment.