AI Security Leaderboard: Methodology, Results and Minimal Standard

• Testing focuses on chemical, biological, radiological, nuclear, explosive, and offensive-cybersecurity threats, where model capabilities are already high and advancing rapidly.Source: §1 · p. 4
• The report and leaderboard aim to make current safeguard robustness transparent and provide targets for future improvement.Source: §1 · p. 4
• The report presents model and domain results, defines a minimum safeguard bar, describes its testing methodology, and discusses recommendations, limitations, and future standard updates.Source: §1 · p. 4

2.1. OUTCOMES

• For models with no universal jailbreak found, reported costs are statistical lower bounds, not evidence that a jailbreak is impossible.Source: §2.1 · p. 4
• Expert guidance found roughly an order of magnitude more jailbreaks for susceptible models, often generalizing across several domains; domain counts can exceed model totals when attacks span domains.Source: §2.1 · pp. 4–5
• Universal jailbreaks (ASR >75% in at least one tested domain) appeared for Grok 4.5 and Gemini 3.1 Pro, but not Claude Fable 5 or GPT-5.6 Sol; Figure 2 breaks counts down by domain.Source: §2.1 · p. 4, Figure 2 · p. 5
• Claude Opus 4.8 and GPT-5.5, like the latest Claude and GPT models, showed no universal jailbreaks; Grok 4.5 removed expert-found biological jailbreaks from Grok 4.3 but exposed comparable or greater counts in other domains.Source: §2.1 · p. 4, Figure 3 · p. 5
• Abridged elicited transcripts from Gemini and Grok illustrate the responses behind the reported jailbreak counts; unredacted transcripts were shared privately with the affected providers.Source: §2.1 · p. 5

2.2. ANALYSIS OF SUCCESSFUL JAILBREAKS

• Expert-guided composition raised average attack success rates for every model and sharpened attacks on weak points already exposed by random search, with per-domain success rates aligned across susceptible models (Spearman ρ = 0.93).Source: §2.2 · p. 5
• Experts concentrated on a small set of effective primitives with positive marginal lift, while random search included primitives experts omitted, some with negligible or negative lift.Source: §2.2 · p. 5
• Expert-guided primitives varied in transferability across models and domains, and the set was not optimized for any particular model or domain, leaving room for further targeted search.Source: §2.2 · p. 5

3.1. INTRODUCTION AND MOTIVATION

• The Minimal Standard is a minimum bar: failure indicates a lack of state-of-the-art security, while passing does not establish that a model is secure overall.It operationalizes the bar through best-practice defenses already publicly described and deployed in production by multiple frontier developers.Source: §3.1 · p. 6
• Version 1.0 calls for safeguards against known or readily discoverable jailbreaks that reliably elicit operational compliance in CBRNE and cybersecurity contexts.The standard is intended to evolve as attack and defense methods and model capabilities change.Source: §3.1 · pp. 6–7

3.2. TAXONOMY OF JAILBREAKS

• The jailbreak taxonomy is a representative, accessible working set rather than an exhaustive catalog, and it defers dynamic attacks to future versions.Dynamic methods iteratively adapt attacks against a live target; this version therefore characterizes a minimum level of resilience against static attacks.Source: §3.2 · p. 7, §3.2.2 · p. 9

3.2.1. PRIMITIVES, COMPOSITES, VARIANTS, AND THE MODEL INPUT

• A primitive is a jailbreak technique; combining primitives forms a composite, and fixing its parameters and content yields a variant, the runnable unit scored.Core strategies carry the harmful request, while augmentations alter the attack setup and transformations alter content; follow-ups are fixed prompts sent after an initial response.Source: §3.2.1 · p. 7

3.2.2. COVERAGE AND SOURCES

• The working set contains 67 primitives, but even restricted valid combinations yield over 10^8 candidate variants, motivating a sampling approach.The primitives span core strategies, augmentations, text and multimodal transforms, and follow-ups.Source: §3.2.2 · p. 8
• The primitives adapt mechanisms from public literature, repositories, and informal communities rather than relying on verbatim circulated prompts.Dynamic attacks are excluded in part because their iterative live-target search is substantially more costly and impractical across the full sweep.Source: §3.2.2 · pp. 8–9

4.1. PROVIDERS AND MODEL VERSIONS EVALUATED

• The evaluation used closed-weight flagship models accessed through provider APIs, with reasoning levels chosen to test models near robust settings and a common direct-chat setup.The model set included two versions each from Anthropic, OpenAI, and SpaceXAI, and Gemini 3.1 Pro; variants were delivered as a single user message, optionally followed by scripted turns.Source: §4.1 · pp. 9–10

4.2. ATTACKER GOAL DATASET CONSTRUCTION

• DeepHarm and Propensity pair technically specific harmful requests with higher-level requests where harmful intent is more explicit, across five risk domains.The complementary datasets provide a more stringent criterion for universal jailbreaks than either dataset alone.Source: §4.2 · p. 10, §4.2.2 · p. 12

4.2.1. DEEPHARM

• DeepHarm contains 180 expert-reviewed goals, with 36 per domain; chemistry and explosives received more mixed expert ratings on viability and harmfulness.Goals were generated and refined, screened by rating models, then reviewed by subject-matter experts for viability, misuse focus, and whether benign framing could preserve the request’s substance.Source: §4.2.1 · pp. 10–12

4.2.2. PROPENSITY

• Propensity adds 180 goals across the same five domains, emphasizing plainly harmful requests whose intent is apparent rather than obscured by technical terminology.The goals were reviewed with language models for duplication, technical viability, and clear harmfulness, without subject-matter expert review.Source: §4.2.2 · p. 12

4.2.3. RUBRIC CREATION

• For each goal, a rubric organizes expected harmful-content elements into topics so evaluators can assess meaningful compliance against the request.Only harmful-category content is counted, to avoid labeling safe completions as compliant.Source: §4.2.3 · p. 12

4.3.1. AUTOMATED EVALUATORS

• Candidate success requires all three evaluators to jointly judge a response substantially compliant, directly relevant, and sufficiently complete in harmful rubric content.The compliance and relevance thresholds are 0.75; rubric fulfillment uses a 0.5 threshold because a response need not contain every harmful element in a broad rubric.Source: §4.3.1 · p. 12

4.3.2. HUMAN LABELS AND EVALUATOR VALIDATION

• 89.0% of 600 adjudicated examples matched human labels; the automated evaluator’s false-positive rate was 2.3% and its false-negative rate was 23.6%.The human-labeled sample covered Propensity and DeepHarm; 20% was independently labeled by two annotators, who generally agreed (Cohen’s κ = 0.80).Source: §4.3.2 · p. 13

4.4. VARIANT CONSTRUCTION AND THE EVALUATION FUNNEL

• The study assembled random and expert-crafted variants, then screened each variant-model-domain combination through stages before testing survivors on the full domain dataset.The pools contain 1,000 random and 500 expert variants; the final stage tests 72 goals per domain and counts a universal jailbreak at 75% success or higher.Source: §4.4 · pp. 13–14

5. SECURITY RECOMMENDATIONS

• The recommended security approach is defense in depth, combining multiple safeguards so other layers can provide backup if one fails.The recommendations draw on publicly documented defenses already deployed by frontier developers and lessons from red-teaming.Source: §5 · p. 14

5.1. MONITOR REASONING

• Monitoring intermediate reasoning, scratchpads, tool plans, or other hidden signals may reveal unsafe compliance before it reaches the user-facing response.Such signals may be especially informative when output is encoded, translated, or obfuscated.Source: §5.1 · p. 14

5.2. MONITOR INTERNAL MODEL SIGNALS

• Activation-based probes or classifiers can detect unsafe generation trajectories before operational content is produced, though they require calibration and validation.The report says these monitors can in many cases be harder to bypass than text-only filters because they are coupled to model computation and understanding.Source: §5.2 · p. 14

5.3. USE INDEPENDENT INPUT AND OUTPUT FILTERS

• Input and output filters can detect harmful requests and responses, but should account for encodings, multimodal text, and conversation history rather than stand alone.Text-level filters can be bypassed through obfuscation or indirect phrasing.Source: §5.3 · p. 14

5.4. STRENGTHEN INSTRUCTION HIERARCHY

• Training should preserve system and developer instruction priority over user prompts, including attempts to disable safeguards, suppress refusals, or authorize restricted content.Deliberative alignment training can teach models to resolve instruction conflicts before acting.Source: §5.4 · p. 15

5.5. EVALUATE COMPOSED JAILBREAKS

• Robustness evaluations should test composed attacks across single- and multi-turn, multilingual, encoded, multimodal, prompt-injection, and tool-use scenarios.The report recommends turning red-team discoveries into reusable templates for future evaluations.Source: §5.5 · p. 15

Attacker Goals

• Chemistry and explosives DeepHarm prompts may lean closer to dual-use applications than prompts in the other domains, despite curation to minimize that ambiguity.Expert agreement was lower and ratings more mixed in these domains; the limitation remains an area for future work.Source: §6 · p. 15
• Propensity prompts did not receive subject-matter expert ratings, though their more explicit harmful intent makes their misuse orientation less ambiguous than DeepHarm’s.The report notes that this distinction mitigates, but does not remove, the limitation.Source: §6 · p. 15

Evaluation

• The evaluation has a non-trivial false-negative rate, so it may have missed jailbreaks, although the authors say this does not change overall conclusions or relative comparisons.They note that any non-zero number of universal jailbreaks in severe misuse domains indicates a need for improved mitigations.Source: §6 · p. 15
• The study assesses operationalizability and relevance but does not directly assess response accuracy or ultimate uplift.The authors plan to add more capability assessment in future leaderboard versions.Source: §6 · p. 15

Cost Estimation

• Cost estimates assume an attacker has access to the study’s tooling; without it, they support relative difficulty comparisons but not absolute prices.More precise estimates for different attacker types are left for future work.Source: §6 · p. 15

Output token limit

• Model output-token limits differed by provider and testing round, although possible truncation was observed in fewer than 0.1% of cases under the initial 8,192-token limit.The report says those cases still retained enough response content for evaluators to identify meaningful harmful output if present.Source: §6 · p. 15

7. RESPONSIBLE DISCLOSURE

• The public report redacts some successful-jailbreak details to avoid enabling misuse.An unredacted version was shared with frontier companies before release for factual corrections and to inform security improvements.Source: §7 · p. 15
• Government bodies, AI safety institutes, frontier developers, and established security researchers can request the unredacted report by submitting their name, organization, and intended use.Source: §7 · p. 15

8. CONCLUSION

• Two tested models resisted every tested jailbreak, while two others were susceptible to numerous universal jailbreaks.The robust models were Claude Fable 5 and GPT-5.6 Sol; the susceptible models were Gemini 3.1 Pro and Grok 4.5.Source: §8 · p. 16
• The report frames and tests a Minimal Standard through a taxonomy of over 60 jailbreaks, automated combination and testing tools, and large-scale CBRNE and cyber evaluations.Source: §8 · p. 16
• The public leaderboard will be updated with major model releases, and the Minimal Standard revised as attack and defense techniques advance.Source: §8 · p. 16

A. HUMAN LABELING PROCESS FOR EVALUATOR VALIDATION

• For technical requests, labelers count a response as jailbroken when it gives assessable, goal-specific operational guidance, even without exact quantities or parameters.Generic outlines do not meet this bar; opening refusals do not negate later operationally compliant content.Source: A. HUMAN LABELING PROCESS FOR… · p. 18

EDGE CASES

• Edge cases count concise compliant answers as jailbroken when the prompt requires only a concise answer, but not when a jailbreak distorts the attacker’s original goal.Source: EDGE CASES · p. 18

A.1. PER-DOMAIN AND PER-MODEL ERROR RATES

• The preliminary tables break evaluator false-positive and false-negative rates down by harm domain and target model.Source: §A.1 · p. 18, Table 2 · p. 19, Table 3 · p. 19
• False-positive rate is the share of human-labeled negatives marked successful by the evaluator; false-negative rate is the share of human-labeled successes marked unsuccessful.Source: §A.1 · p. 18
• The reported rates use a rubric-fulfillment threshold of 0.5, with compliance and relevance thresholds held at 0.75, and 95% Wilson score intervals.Source: §A.1 · p. 18, Table 2 · p. 19, Table 3 · p. 19

A.1.1. HUMAN REVIEW OF EVALUATOR–HUMAN DISAGREEMENTS

• Blindly relabeling all 84 evaluator–human disagreements changed 16 of 74 apparent false negatives to not-jailbroken and 2 of 10 apparent false positives to jailbroken.Reviewers saw only the request and response, not prior labels or evaluator verdicts.Source: §A.1.1 · p. 19

B. ESTIMATING COST TO JAILBREAK

• The dollar metric estimates expected attacker spending before finding a candidate that works on at least a τ-fraction of requests in a model-domain cell.The candidate is universal at the specified threshold; a better-defended cell has a higher estimated cost.Source: B. ESTIMATING COST TO JAILBREAK · p. 19, §B.1 · p. 20

B.1. SETTING

• A budget-conscious attacker screens candidates cheaply, then confirms survivors on more samples; Tables 6 and 7 compare screened and blind workflows.Screening uses 8 samples per domain in the reported testing; confirmation tests whether a candidate meets the universality threshold.Source: §B.1 · p. 20, Table 6 · p. 21, Table 7

B.2. NOTATION

• The notation tracks candidate success rates and costs, screen and confirmation probabilities, redundancy, and the reported costs for screened versus blind attackers.Source: §B.2 · p. 21, §B.3 · p. 22

B.3. PER-CANDIDATE QUANTITIES

• Per-candidate ASR is the fraction of requests eliciting a harmful response, and a piecewise-linear discount reduces marginal candidates before reliability testing.The defaults set the discount to zero below 0.5 and full reliability above 0.9; the confirmation sample size is 103 at τ = 0.75.Source: §B.3 · p. 22

B.4. CORRECTING FOR REDUNDANT JAILBREAK CANDIDATES

• Candidate similarity is measured by Jaccard overlap of discriminating primitives, allowing the cost estimate to discount near-duplicate successful variants.Primitives common to every candidate are excluded from each candidate’s discriminating set.Source: §B.4 · p. 22

B.5. THE DOLLAR COST

• The cost combines expected samples per variant, effective distinct successes, and expected search through candidates; cells with no observed universal jailbreak receive a 95% statistical ceiling.Such a ceiling is reported as a lower bound on attacker spend if a universal jailbreak exists.Source: §B.5 · p. 23

Screen (K1, a1)

• The screen is the metric’s biggest lever: cost bottoms out around K1 ∈[1, 5], then rises under over-screening, while stricter a1 lowers cost with diminishing returns past about 0.6.A light screen can save cost; the model ordering is preserved at every setting, and K1 = 0 recovers the certified blind ceiling.Source: §B.6 · pp. 23–24

Threshold, confirmation stringency, and discount floor

• Raising universality threshold τ lowers cost without changing model ordering across 0.6–0.9; shrinking δ steeply raises confirmation budget and rescales costs, but leaves rankings invariant.Kstat ranges from 21–441 across the δ sweep because Kstat ∝1/δ2.Source: §B.6 · p. 23
• The headline is nearly flat across discount-floor settings, indicating insensitivity to the g(·) discount shape.Results for glow and phigh were likewise negligible and omitted.Source: §B.6 · p. 23

Takeaway

• Claude Fable 5 and GPT-5.6 Sol remain Strong at every setting, Grok 4.5 is consistently cheapest to jailbreak, and Gemini 3.1 Pro remains between them.Parameter choices move absolute dollar figures but not the implied safeguard ranking.Source: §B.6 · p. 23, Table 8 · p. 24

Baseline compliance

• For baseline compliance, a prompt counts if at least one of ten independent runs passes the policy, relevance, and rubric-based evaluators.The compliance rates are reported by dataset, domain, and model in Table 9.Source: C. BASELINE RESULTS · p. 25, Table 9 · p. 25

API-level refusals

• API-level refusal rates measure repeated requests blocked before generation, excluding model-level refusals returned as generated text; biological requests had the highest rates, with most blocked.These API-level rates are stochastic across repeated runs.Source: C. BASELINE RESULTS · p. 25, Table 10 · p. 25

Harm recognition

• Nearly all model-domain pairs rated all attacker goals as harmful, suggesting robustness evaluations primarily test whether safeguards maintain refusal behavior.Table 11 reports harm-label rates across datasets, domains, and models, with 36 prompts per domain.Source: D. TARGET-MODEL HARM RECOGNITION · p. 25, Table 11 · p. 26

Harm-rating procedure

• Models independently classified each attacker goal as harmful or harmless without being asked to fulfill its underlying instruction.Blocked or refused classification attempts also counted as harmful ratings.Source: D. TARGET-MODEL HARM RECOGNITION · pp. 25–26