PaperlayerView exact revision on arXiv

Source-linked AI summary

Auditing System-1 Models on Biosecurity-Relevant Benchmarks: Calibration, Selective Prediction, and Permutation Instability in a Non-Generative Model

Kimon Antonios Provatas, Ilias Georgakopoulos-Soares

arXiv:2609.30454v1cs.LG

TL;DR

The reliability of inexpensive System-1 models on biosecurity-relevant tasks has not been systematically examined. This paper audits one commercial model across benchmark tasks, uncertainty, selective prediction, and option-order changes; accuracy varies by task, and answer order reveals instability not apparent from aggregate accuracy.

  • Problem

    The empirical properties of low-cost probabilistic models on biosecurity-relevant tasks—including accuracy, uncertainty, abstention, and stability—need to be established before considering their role in safety and biosecurity pipelines.

    Source: §1 · p. 1

  • Method

    The authors audit one commercial System-1 model on 6,020 benchmark items, measuring accuracy, calibration, error detection, selective prediction, and answer-order sensitivity with a repeated-call control.

    Source: §1 · pp. 1–2, §3 · p. 2

  • Results

    Accuracy is strongly task-dependent; pooled AUROC for pmax separating correct from incorrect predictions is 0.820, while more than a third of WMDP-Cyber items change answers across rotations.

    Source: §5 · p. 5

  • Takeaways & Limitations

    Confidence and stability under re-presentation provide complementary evidence about when predictions should be trusted, and neither is visible in a single accuracy figure.

    Source: §6 · p. 6

    Because the study evaluates one proprietary closed-weight model, its results do not generalize to System-1 models as a class.

    Source: §5.1 · p. 5

Abstract

from arXiv · show

Non-generative "System-1" models return structured probabilistic decisions in a single forward pass, without autoregressive decoding, at a small fraction of the inference cost of a generative model. This makes them of interest as inexpensive components in larger pipelines, but their reliability on biosecurity-relevant tasks has not been systematically examined. We audit one commercial System-1 model on 6,020 multiple-choice items drawn from the Weapons of Mass Destruction Proxy (WMDP), a paraphrase-robust WMDP-Bio variant, and six LAB-Bench subtasks, measuring accuracy, calibration, error detection, selective prediction, and sensitivity to the order in which answer options are presented. Accuracy is strongly task-dependent. Once the vendor's uncertainty field is correctly interpreted, the model is reasonably well calibrated (pooled expected calibration error 0.034) and its top-1 probability separates correct from incorrect predictions (pooled AUROC 0.820), though both degrade substantially on the weaker tasks. Under four cyclic rotations of the answer options, 37.4% of WMDP-Cyber items receive different answers; a control using byte-identical repeated calls attributes most of this to option order rather than run-to-run variation. Averaging probabilities across rotations improves WMDP-Cyber accuracy by 3.8 percentage points, and applying it only to low-confidence items recovers most of that gain at well under the cost of averaging every item.

1. Introduction

System-1 models offer low-cost probabilistic decisions, but their reliability on biosecurity-relevant tasks must be established beyond benchmark accuracy. This paper audits a commercial model across accuracy, uncertainty, abstention, and answer-order stability.

1. Introduction

  • System-1 models return typed probabilistic decisions in one forward pass, and the reported evaluation cost $0.57 for 19,060 calls.The authors frame repeated evaluation as practical rather than hypothetical given this cost.Source: §1 · p. 1
  • The audit asks whether the model is correct, whether reported uncertainty identifies errors, whether abstention raises remaining accuracy, and whether option order changes decisions.The paper treats answer-option order as a meaning-preserving change.Source: §1 · p. 1
  • The audit covers 6,020 biosecurity- and biology-relevant benchmark items, measuring accuracy, calibration, error detection, and selective prediction.It also examines answer-order sensitivity with a byte-identical-repeat control and tests selective permutation averaging.Source: §1 · pp. 1–2
  • Under four cyclic rotations, more than a third of WMDP-Cyber items receive different answers, and a repeat-call control separates option-order sensitivity from run-to-run variation.Selective permutation averaging improves accuracy on the instability-heavy task while avoiding four evaluations for every item.Source: §1 · p. 2
  • The paper does not claim novelty for confidence-based selection or the observation that multiple-choice evaluation is sensitive to option order.It cites prior work on selection – and option-order sensitivity, [10].Source: §1 · p. 2

2. Background and Related Work

Prior work motivates inexpensive probabilistic models through decision cascades, multiple-choice order effects, and deployed safety classifiers. This study examines reliability on biosecurity-relevant knowledge benchmarks rather than evaluating safeguard performance.

Decision cascades and early exit

  • Decision cascades allocate extra computation to difficult inputs, and this study applies that idea through additional answer permutations of the same model.Prior examples include CALM, Big Little Decoder, FrugalGPT, and RouteLLM; the paper also uses standard uncertainty and selective-classification tools –.Source: §2 · p. 2

Multiple-choice robustness

  • Prior work reports that answer reordering can alter leaderboard rankings and accuracy [10], while Zheng et al. attribute effects to token-level option-label bias.Those cited studies concern generative models scored by likelihood over option tokens; this paper tests a non-generative model that emits a typed distribution in one pass.Source: §2 · p. 2

Safety classifiers and biosecurity context

  • Safety systems use auxiliary classifiers to detect disallowed content, but this study evaluates reliability on knowledge benchmarks rather than dangerous-request blocking.The motivating examples include Llama Guard [1], constitutional classifiers [2], and ShieldGemma [17]; biological risk is noted as a specific concern [18].Source: §2 · p. 2

3. Experimental Setup

The setup evaluates one proprietary System-1 model on ten benchmark datasets using accuracy, uncertainty, and selective-prediction measures. It also tests cyclic answer-option rotations and reports inference cost without measuring latency.

3. Experimental Setup

  • JEV was queried through the v1 REST endpoint, resolving jev-latest to jev-1.13.0 during calls made on 2026-09-19 and 2026-09-20.The proprietary model exposes no logits, token probabilities, or random seed; returned probabilities are quantised to two decimals.Source: §3 · p. 2
  • The ten datasets include WMDP hazardous-knowledge proxies, a paraphrased WMDP-Bio variant, and six LAB-Bench subtasks; 6,020 items remained after one API error was excluded.Because option counts range from 2 to 10, accuracy is compared with each item's own chance level, 1/n.Source: §3 · p. 2
  • For two four-option WMDP suites, 1,273 and 1,987 items were each presented under all four cyclic option rotations, producing 13,040 calls.Returned distributions were mapped back to original option indices for comparison across rotations.Source: §3 · p. 2
  • The evaluation used 19,060 calls and cost $0.57 in input-token charges; comparable generative-model list-price estimates ranged from about $0.71 to $421.The estimates vary by model tier and output setup; latency was not measured.Source: §3 · p. 2

4. Results

Accuracy and uncertainty quality vary substantially across tasks, while option-order changes produce item-level disagreement despite nearly stable aggregate accuracy. Rotation averaging improves WMDP-Cyber accuracy, and confidence-ranked extra evaluations recover most of that gain at reduced cost.

4.1. Benchmark performance and uncertainty

  • Accuracy ranges from 85.2% on WMDP-Bio to 63.4% on WMDP-Cyber; LAB-Bench spans 36.6% on SuppQA to 61.8% on SeqQA.SuppQA remains above its 21.3% chance level, and aggregate accuracy reflects the particular task as much as the model.Source: §4.1 · pp. 2–3, Table 1 · p. 3
  • The vendor confidence field rescales pmax rather than predicting correctness; using pmax, pooled ECE is 0.034 and AUROC is 0.820, with per-task ECE 0.033–0.193 and AUROC 0.62–0.86.Calibration is at most 0.057 ECE on the four WMDP suites but reaches 0.168 on LitQA2 and 0.193 on SuppQA.Source: §4.1 · p. 3, Figure 1 · p. 4

4.2. Confidence and selective prediction

  • Withholding the least-confident 40% raises retained accuracy from 85.2% to 97.1% on WMDP-Bio and from 63.4% to 78.9% on WMDP-Cyber.Confident-answer error rates also vary by task, reaching 38.1% on LitQA2; a threshold calibrated on one dataset should not be assumed to transfer.Source: §4.2 · p. 3, Figure 1 · p. 4

4.5. Selective permutation averaging

  • At 2.2 calls per item, rotating the least-confident 40% reaches 67.0% WMDP-Cyber accuracy, recovering 97% of full-averaging gain at 45% lower cost.The 40% strategy costs 1 + 3c calls per item, where c is the fraction receiving all four rotations.Source: Figure 2 · p. 4, §4.5 · p. 5

4.3. Answer-order instability

  • Answer agreement across four rotations is 62.6% on WMDP-Cyber versus 89.4% on WMDP-Bio; on Cyber, unstable items err 62.5% of the time versus 21.3% for stable items.Aggregate accuracy varies by only 0.8 percentage points across Cyber rotations and 1.3 points across Bio rotations.Source: §4.3 · p. 3, Figure 2 · p. 4
  • The instability–error association is descriptive, not causal: a matched null predicts a 5.43 Cyber error ratio, above the observed 2.93.The measurement partly links consistency and correctness to the first rotation, and the observed ratios fall below the null-model predictions on both Cyber and Bio.Source: §4.3 · p. 4

4.4. Separating option order from run-to-run variation

  • On WMDP-Cyber, identical-call disagreement is 9.2% versus 37.4% across rotations, leaving 28.2 percentage points attributable to option order; first-position preference is strongest there and primarily positional.First-position predictions are 34.8% on Cyber against 26.6% of correct answers; glyph and letter permutations indicate the preference is primarily positional, with a weaker label component.Source: §4.4 · pp. 4–5, Figure 2 · p. 4

4.5. Selective permutation averaging

  • Averaging four rotations raises WMDP-Cyber accuracy from 63.3% to 67.1% (+3.8 pp), while WMDP-Bio gains only 0.5 pp with an interval spanning zero.Majority voting is weaker on Cyber at 65.2%; averaging four identical calls changes accuracy by only +0.5 pp, with an interval spanning zero.Source: §4.5 · p. 5, §4.4 · p. 5

5. Discussion and Limitations

Task-specific calibration and error rates make confidence thresholds nontransferable, while re-presentation stability adds evidence unavailable from a single evaluation. The study does not test deployment settings where users can influence presentation and does not provide a dedicated benign-versus-hazardous classification benchmark.

5. Discussion and Limitations

  • Pooled uncertainty metrics can flatter performance: calibration and high-confidence error rates vary by dataset, so thresholds should not be assumed to transfer.Stability under re-presentation is a second reliability signal, but its value depends on how much instability a task exhibits.Source: §5 · p. 5
  • The benchmark does not test deployments where users can influence presentation, although representation sensitivity could become a manipulation surface there.Disagreement across re-presentations could signal an unreliable decision.Source: §5 · p. 5
  • Evaluating System-1 models as biosecurity or language-model safety components requires a dedicated benign-versus-hazardous classification benchmark, which this study does not provide.Source: §5 · p. 5

5.1. Limitations

  • Results cover one proprietary closed-weight model without an exposed seed, so they do not generalise to System-1 models as a class and exact reproduction depends on vendor behaviour.Cyclic evaluation covers four rotations of four-option WMDP suites, making reported instability a lower bound; label and format ablations cover only those suites.Source: §5.1 · p. 5
  • Several LAB-Bench high-confidence subsets contain fewer than 25 items, so the authors report intervals throughout.Source: §5.1 · p. 6

6. Conclusion

The audit finds that reliability varies across tasks and that confidence and re-presentation stability offer complementary evidence about when predictions should be trusted. Probability averaging improves accuracy where instability is high, with low-confidence-only application recovering most of the gain at a fraction of the cost.

6. Conclusion

  • Accuracy is strongly task-dependent, while top-1 probability is reasonably calibrated and predicts errors well enough to support abstention.Source: §6 · p. 6
  • Probability averaging over answer permutations improves accuracy where instability is high, and applying it only to low-confidence items recovers most of that improvement at a fraction of the cost.Source: §6 · p. 6
  • Confidence and stability under re-presentation provide complementary evidence about prediction reliability, neither visible in a single accuracy figure.Source: §6 · p. 6

Ethics and Conflict of Interest

The study used existing public benchmarks and released decision records without releasing new question content; the authors report no competing interests.

Ethics and Conflict of Interest

Code Availability

Decision records, analysis code, and manuscript source are available, and the authors state that every reported statistic can be recomputed from them.

Code Availability

  • Decision records, analysis code, and manuscript source are available, with every reported statistic recomputable from them.Source: Code Availability · p. 6

LLM Usage Statement

The authors used Anthropic Claude for evaluation and analysis support, literature search, and manuscript editing. They report author inspection of generated code and identify the closed-weight model as a reproducibility limitation.

LLM Usage Statement

  • Anthropic Claude assisted with the evaluation harness, analysis scripts, statistical analysis, literature search, and manuscript editing.The authors state that all generated code was executed and its outputs inspected.Source: LLM Usage Statement · pp. 6–7
  • The model under evaluation is commercial and closed-weight, a reproducibility limitation the authors say they cannot remove.The authors take responsibility for the content's correctness, originality, and integrity.Source: LLM Usage Statement · p. 7
  • Every running-text statistic comes from the released pipeline, which fails the build when a cited value does not match.The authors also say references were verified against primary sources.Source: LLM Usage Statement · p. 7
  • An earlier analysis version contained statistical errors that the authors found and corrected during internal adversarial review.The authors take full responsibility for the correctness, originality, and integrity of the content.Source: LLM Usage Statement · p. 7

Related papers

Loading 2609.30454v1…