Source-linked AI summary
What AI Red-Team Evaluations Can and Cannot Prove
Bandana Kaur
TL;DR
The paper asks what claims AI red-team evaluations can support despite concerns about their comparability and statistical precision. It defines and derives an evidential ceiling, finding a computable harm-rate boundary where clean benchmark results become stronger evidence than reproduced failures. The paper concludes that evaluations should report each result alongside the claim it supports.
Problem
Existing critiques question whether red-team outputs are comparable and statistically precise enough to support their conclusions.
Method
The paper defines an evidential ceiling for evidence-generating procedures and derives it for benchmark null results using likelihood ratios between elevated- and acceptable-risk hypotheses.
Results
At a 1% harm rate with 520 prompts, a clean result carries 1.4 times the evidence of a single observed harm, while below the boundary feasible benchmarks cannot shift belief toward safety.
Takeaways & Limitations
Safety evaluations should neither be abandoned nor trusted indiscriminately, but reported alongside the specific claim each result supports.
Takeaways & Limitations
If models behave differently when they infer they are under evaluation, the measured discrimination can overstate the target procedure’s value and make null results less informative.
Abstract
from arXiv · showhide
Red-team evaluations of AI models support some claims and not others, and the boundary between the two is calculable rather than merely a matter of judgment. We define the evidential ceiling of an evaluation as the largest factor by which one result can move belief under a fixed testing budget, derive it in closed form for the benchmark null result, and use it to locate that boundary exactly. We find that above a calculable harm rate, a benchmark of modest size certifies a category to a stated evidentiary standard, and a clean sheet is then the stronger of the two possible observations, outweighing a single reproduced failure. Below that rate, no passive benchmark of feasible size provides the specified evidence of safety under the fixed scoring rule and approximately independent trial structure. The crossing between the two regimes has a closed form. The bound is not specific to benchmarks: written in terms of a procedure's hypothesis conditioned elicitation rates, it covers adaptive and automated red teaming as well, and shows that discrimination between the hypotheses rather than attack success is what determines evidential worth. Auditing eight evaluation suites against the boundary, we find that current benchmarks are adequate for high-frequency harm categories and several orders of magnitude short for rare, catastrophic ones. Safety benchmarks are not uninformative. They are informative about a specific and computable set of propositions, and the discipline they need is to state which.
1 Introduction
Safety evaluations are informative, but only for propositions determined by harm rate, sample size, and evidentiary standard. A closed-form boundary shows when benchmarks support certification and when their evidence is insufficient, including a reversal in the relative strength of clean results and reproduced failures.
- Motivation: Safety benchmarks are not worthless; their evidentiary value depends on which propositions their design can establish.The paper rejects the conclusion that critiques of comparability, generalization, and statistical precision make safety benchmarks nearly worthless.
- Core contribution: A closed-form expression determines whether an evaluation licenses certification from harm rate, sample size, and evidentiary standard.The expression can also determine how large an evaluation would need to be when the existing benchmark does not clear the evidentiary bar.
- Practical scope: High-frequency harm categories may already meet the certification threshold with existing public benchmarks, whereas rare categories may require much larger evaluations.The paper presents both conclusions as consequences of the same closed-form expression, which laboratories can use before running an evaluation.
- Evidence reversal: At a 1% harm rate with 520 prompts, a clean result carries 1.4 times the evidence of a single observed harm.Below a computable crossing rate, one observed failure is stronger; above it, the clean sheet is stronger evidence.
- Implications: Like modern medical trials, AI evaluations should support claims proportional to their statistical power rather than be treated as inherently uninformative.The paper uses medicine’s shift from small-trial reassurance to powered, pre-registered trials as an analogy for computing what an evaluation can license.
2 Related work
Prior work diagnoses limits in red-team comparisons and evaluation reliability, while safety-case research addresses evidence aggregation. This paper instead quantifies the evidential weight of a single evaluation source, especially in the rare-event regime.
- Related work: Measurement-theoretic work finds that red-teaming outputs, chiefly attack success rate, rarely support valid quantitative comparisons across systems.Related work also identifies failures of safety generalization across semantically equivalent prompt structures, motivating new attacks and defenses, but these diagnoses remain qualitative.
- Related work: Statistical work warns that small language-model evaluations produce wide intervals and unreliable comparisons; this paper specializes those concerns to rare events.It derives closed forms in both directions and audits how far current practice sits from each.
- Related work: Safety-case research asks whether heterogeneous evidence sources jointly license deployment, whereas this paper quantifies how much weight a single source carries.The resulting per-source quantity is intended to support evidence aggregation.
3 What an evaluation must establish
A safety evaluation can generalize from observed test performance to deployment behavior only if it has statistical sufficiency, distributional validity, and structural generalization.
- 3 What an evaluation must establish: Safety evaluation requires statistical sufficiency: a sample large enough to distinguish safety-relevant signals from sampling noise.This is the test’s power to detect safety-relevant differences.
- 3 What an evaluation must establish: Distributional validity requires test inputs to represent deployment inputs, so measured performance predicts deployment behavior rather than only test behavior.The inference depends on alignment between the test-input distribution and the deployment distribution.
- 3 What an evaluation must establish: Structural generalization requires robustness beyond the specific stimuli tested, making passing evidence of a latent safety property rather than learned test compliance.This condition distinguishes general safety from having learned to pass the evaluation.
- 3 What an evaluation must establish: These three conditions are minimum requirements for empirical claims that generalize from a sample to a population.The paper connects them to power calculations, representative sampling frames, and later sections that quantify or test each condition.
4 The evidential ceiling
The evidential ceiling is the maximum evidential factor a feasible evaluation result can contribute between elevated- and acceptable-risk hypotheses. For benchmark null results, closed-form boundaries show when clean sheets certify safety, when feasible benchmarks cannot, and how evidential value depends on discrimination rather than attack success alone.
- Definition: The evidential ceiling is the largest factor by which one feasible result can move posterior odds between elevated and acceptable risk.It is defined over an evidence-generating procedure, resource budget, observation channel, and likelihood ratio, with quantities reported in bits.
- Adequacy: At p = 10−2, a factor-of-two update needs 138 prompts, an order-of-magnitude update needs 458, and false certification below five percent needs 299; AdvBench at n = 520 clears all three.Required sample size scales as O(1/p) and is computable before evaluation.
- Impossibility: If p < pmin(τ, Nmax, r), no approximately independent evaluation within Nmax trials can make a zero-harm result informative evidence of safety at threshold τ.At Nmax = 105, r = 0.5, and τ = 0.5, the boundary is pmin ≈1.4 × 10−5; this conclusion is conditional on the scoring rule and induced harm rate.
- Comparative evidence: At n = 520 and r = 0.5, the clean sheet is more informative above p× = 1.33 × 10−3, while a single harm is more informative below it.At n = 105, the crossing is p× = 6.93 × 10−6; above the crossing, clean sheets dominate, while below it their evidence vanishes toward zero as harm becomes rarer.
- Generalization: The framework extends beyond passive benchmarks: evidential worth is determined by per-trial discrimination, with passive benchmarks as q1 = pu and q0 = ps = rpu.Adaptive or model-assisted elicitation may change the governing statistic, while the rare-event approximation κ ≈ q1(1 −r)/ ln 2 applies only at fixed r.
5 How far is current practice from the boundary?
Across eight evaluation suites, current practice is adequate for high-frequency harm categories but falls far short for rare harms, with clustering and distribution mismatch further limiting what clean results can establish. The section therefore distinguishes statistical certification from power and requires explicit assumptions about prompt dependence and deployment relevance.
- Distance from the boundary: Eight suites are adequate at p = 10−2 but inadequate below it, placing current public benchmarks on opposite sides of the calculable boundary.The calculations depend on sample size n under assumed, common harm rates and are not rankings of benchmark quality.
- Distance from the boundary: At punsafe = 0.01, power ranges from 14.9 percent for XSTest (n = 250) to 59.3 percent for SafetyBench (n = 2,100), while none reaches 80 percent.Achieving 80 percent power requires approximately 3,679 prompts per arm one-sided or 4,670 two-sided at α = 0.05.
- Certification versus power: At p = 0.001, false-certification probability is 73.1 percent for StrongREJECT (n = 313) and 59.4 percent for AdvBench, while n > 2,995 is required to reduce it below five percent.At p = 10−2, the same five-percent standard needs only 299 prompts, which seven of eight suites clear.
- Certification versus power: A clean run can certify an upper bound even when it lacks power: zero harms in 520 prompts gives a 95 percent one-sided upper bound of 0.57 percent, versus 0.14 percent in 2,100 prompts.Power detects differences between models, whereas certification bounds one model’s harm rate; reporting only power understates what a clean run establishes.
- Dependence and representativeness: Clustering reduces information and raises the zero-harm probability: m = 10 with ρ = 0.1 gives DEFF = 1.9, while m = 25 with ρ = 0.1 gives 3.4 and neff ≈0.29n.Clustering strictly raises P(k = 0) for any ρ > 0, pushing the boundary upward and enlarging the region where no feasible benchmark certifies.
- Dependence and representativeness: Benchmark-to-deployment similarity is 0.057 and 0.055, versus within-benchmark similarity of 0.18 and 0.19, and a zero-harm result on Q places no upper bound on deployment harm without a linking assumption.The measured separation establishes non-representation of ordinary traffic, not distance from adversarial deployment traffic.
6 What an evaluation may claim
An evaluation licenses claims only at the harm rates and sample sizes specified by the claims ladder. Honest safety cases therefore require calibrated reporting, explicit limits on null-result certification, and pre-registered evaluation decisions.
- Reporting requirements: Null-result reporting requires the nominal integer (n, k) for the Clopper-Pearson interval, while clustering requires an exact interval plus a beta-binomial or generalized-estimating-equations interval.neff is generally fractional and is not a sample size for the Clopper-Pearson calculation.
- Limits of null results: Below pmin, a null result cannot certify a category at feasible scale and should be accompanied by the non-benchmark evidence used instead.The prescribed entry is a declaration of infeasibility, not simply a larger benchmark.
- Transparency audit: Across nine public frontier disclosures, only one endpoint reported both an exact numerator and denominator for a binary harm rate, and none reported the dependence structure needed to interpret it.The reporting template is largely a matter of disclosure rather than new measurement.
- Evaluation governance: Pre-registering harm categories, target detectable rates, sampling, and decision criteria prevents post-hoc reframing, while calibrated reporting states the licensed claim rather than only the raw observation.Held-out corpora maintained by independent bodies address benchmark saturation caused by public evaluation sets.
7 Limitations
The paper’s conclusions depend on assumed harm-rate operating points, heterogeneous benchmark estimands, ordinary-traffic sampling, estimated elicitation rates, and approximately independent trials. Its mathematical contribution is not novel, and the boundary could move if adaptive elicitation achieves better rare-harm scaling.
- Mathematical contribution: The mathematics is elementary and non-novel; the contribution is applying the 1983 zero-numerator bound to evaluation design and identifying the boundary in both directions.The paper explicitly claims no mathematical novelty for the bound underlying both theorems.
- Operating points: Adequacy at 10^-2 and inadequacy at 10^-4 are robust across plausible harm-rate estimates, but the specific power percentages are not authoritative.Published frontier-model harm-rate estimates are sparse, so the operating-point anchors are defensible rather than authoritative.
- Benchmark heterogeneity: The eight suites measure heterogeneous estimands, including over-refusal in XSTest and multiple-choice performance in SafetyBench, making one power axis a deliberate simplification.The simplification is defended only because all suites support the same kind of deployment claim.
- Distributional coverage: LMSYS-Chat-1M samples ordinary use, so measured separation establishes only that benchmarks do not represent ordinary traffic, not the distance to adversarial traffic.The distributional-gap analysis covers one mixture component and does not bound the adversarial component.
- Rate estimation: Generalization requires estimating hypothesis-conditioned elicitation rates: q1 needs a valid positive-control proxy, while q0 is harder to estimate; an upper bound on q0 gives a lower bound on discrimination.The stated rates replace a sample count, and both must be estimated rather than assumed.
- Falsifier: The theorems rely on Nmax binding over approximately independent trials; better-than-O(1/p) adaptive elicitation in the rare-harm regime would move the boundary.The bottleneck argument suggests the natural route tightens rather than loosens the boundary, but does not prove tightness.
8 Discussion
The paper instantiates the evidential ceiling only for passive benchmark null results, while extending the construct to other evidence types remains future work. Aggregating ceilings across heterogeneous evidence in real safety cases is also an open, unformalized question.
- Scope and limitations: The evidential ceiling was defined abstractly but instantiated only for passive benchmark null results.Other evidence types require their own models of how evidence is generated.
- Scope and limitations: Computing ceilings for interpretability findings, human uplift trials, and structured expert elicitation is identified as a natural next step.Their statistical structures may change the closed form rather than merely its parameters.
- Scope and limitations: Whether ceilings from heterogeneous evidence types can be aggregated to assess real safety cases remains an open research subject.The paper cautions that asserting posterior confidence beyond an aggregate ceiling would be unsound, but this claim has not been formalized for mixed qualitative and quantitative arguments.
9 Conclusions
Adversarial evaluations identify genuine vulnerabilities, improve safety, and provide a systematic check on dangerous systems, but their evidential resolution is computable. The paper therefore argues that each result should be reported alongside the claim it supports.
- 9 Conclusions: Adversarial evaluation has identified genuine vulnerabilities, driven genuine safety improvements, and provided the only systematic check on manifestly dangerous systems.The paper rejects the view that safety benchmarks are worthless, describing them instead as instruments with a computable resolution.
- 9 Conclusions: The boundary between supportable and unsupported claims follows from one closed-form expression rather than judgment or methodological taste.It can be computed before an evaluation runs, allowing a laboratory to understand what its evaluation will be able to claim while design changes remain possible.
- 9 Conclusions: Safety evaluations should neither be abandoned nor trusted unchanged; every result should be reported alongside the claim it supports.This reporting discipline is the paper’s proposed response to the computable limits of evaluation evidence.
Methods
The methods combine Monte Carlo power simulations with analytic false-certification and likelihood-ratio calculations, supplemented by exact confidence bounds. Distributional analyses compare benchmark and user-query datasets using TF-IDF and neural embeddings.
- Monte Carlo power simulation: Power was estimated with 8,000 parametric Monte Carlo iterations per (n, punsafe, psafe) triple using Binomial harm counts and a one-sided two-sample proportions z-test at α = 0.05.Rates included high-frequency p = 0.08, mid-frequency p = 0.025, low-frequency p = 0.003, and frontier punsafe = 0.01.
- Analytic quantities: False-certification probability was computed as P(k = 0 | n, p) = (1 −p)n, with sample sizes derived for keeping it below tolerance ϕ.Likelihood-ratio threshold τ sample sizes followed a separate expression.
- Analytic quantities: Clopper–Pearson bounds used the Beta distribution; for zero events, one-sided 95 percent and two-sided bounds reduce to 1 −α1/n and 1 −(α/2)1/n.The two analytic criteria differ by a factor of (1 −r)−1 to first order, equal to 2 at r = 0.5.
- Distributional analysis: AdvBench (n = 520), HarmBench (n = 400), and 9,089 deduplicated LMSYS-Chat-1M queries were analyzed with TF-IDF and neural sentence embeddings.Distances used 500 random projections for sliced Wasserstein analysis and an RBF-kernel MMD with median-heuristic bandwidth.
Data availability
The study uses publicly available AdvBench and HarmBench repositories, the gated LMSYS-Chat-1M dataset, and publicly available code. No proprietary data was used.
- Data availability: AdvBench and HarmBench are available through their GitHub repositories.AdvBench: https://github.com/llm-attacks/llm-attacks; HarmBench: https: //github.com/centerforaisafety/HarmBench.